ELECTRONIC APPARATUS AND HANDWRITTEN DOCUMENT PROCESSING METHOD

According to one embodiment, an electronic apparatus includes a display processor and a selector. The display processor displays an area stroke on a screen when a first document including a plurality of strokes input by handwriting and one or more images is being displayed on the screen, the area stroke designating a first area. The selector selects first stroke data and first image data based on the area stroke, the first stroke data corresponding to a first stroke among the plurality of strokes, the first image data corresponding to a first part in the one or more images.

DETAILED DESCRIPTION

In general, according to one embodiment, an electronic apparatus includes a display processor and a selector. The display processor is configured to display an area stroke on a screen when a first document including a plurality of strokes input by handwriting and one or more images is being displayed on the screen, the area stroke designating a first area. The selector is configured to select first stroke data and first image data based on the area stroke, the first stroke data corresponding to a first stroke among the plurality of strokes, the first image data corresponding to a first part in the one or more images.

FIG. 1is a perspective view illustrating an external appearance of an electronic apparatus according to an embodiment. The electronic apparatus is, for instance, a pen-based portable electronic apparatus which can execute a handwriting input by a pen or a finger. This electronic apparatus may be realized as a tablet computer, a notebook-type personal computer, a smartphone, a PDA, etc. In the description below, the case is assumed that this electronic apparatus is realized as a tablet computer10. The tablet computer10is a portable electronic apparatus which is also called “tablet” or “slate computer”. As shown inFIG. 1, the tablet computer10includes a main body11and a touch-screen display17. The touch-screen display17is attached such that the touch-screen display17is laid over the top surface of the main body11.

The main body11has a thin box-shaped housing. In the touch-screen display17, a flat-panel display and a sensor, which is configured to detect a touch position of a pen or a finger on the screen of the flat-panel display, are assembled. The flat-panel display may be, for instance, a liquid crystal display (LCD). As the sensor, for example, use may be made of an electrostatic capacitance-type touch panel, or an electromagnetic induction-type digitizer. In the description below, the case is assumed that two kinds of sensors, namely a digitizer and a touch panel, are both assembled in the touch-screen display17.

Each of the digitizer and the touch panel is provided in a manner to cover the screen of the flat-panel display. The touch-screen display17can detect not only a touch operation on the screen with use of a finger, but also a touch operation on the screen with use of a pen100. The pen100may be, for instance, an electromagnetic-induction pen.

The user can execute a handwritten document input operation of inputting a plurality of strokes by handwriting, on the touch-screen display17by using an external object (pen100or finger). During the handwritten document input operation, loci (paths) of movement of the external object (pen100or finger) on the screen, that is, loci (writing traces) of strokes that are handwritten by the handwritten document input operation are drawn in real time. Thereby, the locus of each stroke is displayed on the screen. A locus of movement of the external object during a time in which the external object is in contact with the screen corresponds to one stroke. A handwritten document is composed of a set of many strokes corresponding to handwritten characters or graphics, that is, a set of many loci (writing traces).

In the present embodiment, this handwritten document is stored in a storage medium not as image data but as handwritten document data including time-series information indicative of coordinate series of the loci of strokes and the order relation between the strokes. The details of this time-series information will be described later with reference toFIG. 3. In general, this time-series information means a set of time-series stroke data corresponding to a plurality of strokes. Each stroke data may be of any kind if it can express one stroke input by handwriting, and each stroke data includes coordinate data series (time-series coordinates) corresponding to points on the locus of this stroke. The order of arrangement of these stroke data corresponds to an order in which strokes were handwritten, that is, an order of strokes.

The tablet computer10can read arbitrary existing handwritten document data from the storage medium, and can display on the screen a handwritten document corresponding to this handwritten document data, that is, a handwritten document on which the loci corresponding to a plurality of strokes indicated by time-series information are drawn.

Furthermore, the user can execute an area input operation of inputting an area stroke for designating a first area, on the touch-screen display17by using the external object (pen100or finger). In this area input operation, an arbitrary area on a displayed handwritten document is designated. During the area input operation, like the handwritten document input operation, a locus of movement of the external object (pen100or finger) on the screen, that is, a locus (writing trace) of a stroke that is handwritten by the area input operation, is drawn in real time. Thereby, the locus of one stroke is displayed on the screen. A locus of movement of the external object during a time in which the external object is in contact with the screen corresponds to one stroke. In accordance with a range (first area) on the screen, which is designated by the handwritten single stroke (single-stroke writing), a target part (first part) on the handwritten document is selected (derived). Data of strokes or an object corresponding to the target part is clipped, and can be used in a document which is being processed or in another document.

The user can effect switching between a first mode of performing the handwritten document input operation and a second mode of performing the area input operation, for example, by a predetermined operation using an “area designation” tool or a button, etc. on the pen100, attribute information associated with the pen, etc. Specifically, an input operation on the touch-screen display17is detected as a handwritten document input operation when the tablet computer10is in the first mode, and is detected as an area input operation when the tablet computer10is in the second mode. The attribute information associated with the pen includes, for example, information indicative of the type of pen. More specifically, the attribute information includes information indicative of the setting of “ball-point pen” or “marker pen” (e.g. the thickness, shape, color, transparency, etc. of the pen) or information indicative of the setting of the pen for inputting an area. The “ball-point pen” or “marker pen” is the type of pen, which is named so that the user may easily have an image of a stroke that is to be drawn. By this attribute information, the above-described first mode and second mode can also be discriminated.

In addition, the locus of an area stroke based on the area input operation may be drawn by a line of a kind which is different from the kind of the locus of a stroke based on the handwritten document input operation. For example, the locus of a stroke based on the handwritten document input operation is drawn by a solid line, and the locus of an area stroke based on the area input operation is drawn by a broken line.

Next, referring toFIGS. 2 and 3, a description is given of a relationship between strokes (characters, marks, graphics, tables, etc.), which are handwritten by the user, and time-series information.FIG. 2shows an example of a handwritten document (handwritten character string) which is handwritten on the touch-screen display17by using the pen100or the like.

In many cases, on a handwritten document, other characters or graphics are handwritten over already handwritten characters or graphics. InFIG. 2, the case is assumed that a handwritten character string “ABC” was handwritten in the order of “A”, “B” and “C”, and thereafter a handwritten arrow was handwritten near the handwritten character “A”.

The handwritten character “A” is expressed by two strokes (a locus of “Λ” shape and a locus of “-” shape) which are handwritten by using the pen100or the like, that is, by two loci. The locus of the pen100of the first handwritten “Λ” shape is sampled in real time, for example, at regular time intervals, and thereby time-series coordinates SD11, SD12, . . . , SD1nof the stroke of the “Λ” shape are obtained. Similarly, the locus of the pen100of the next handwritten “-” shape is sampled, and thereby time-series coordinates SD21, SD22, . . . , SD2nof the stroke of the “-” shape are obtained.

The handwritten character “B” is expressed by two strokes which are handwritten by using the pen100or the like, that is, by two loci. The handwritten character “C” is expressed by one stroke which is handwritten by using the pen100or the like, that is, by one locus. The handwritten arrow is expressed by two strokes which are handwritten by using the pen100or the like, that is, by two loci.

FIG. 3illustrates time-series information200corresponding to the handwritten document ofFIG. 2. The time-series information200includes a plurality of stroke data SD1, SD2, . . . , SD7. In the time-series information200, the stroke data SD1, SD2, . . . , SD7are arranged in time series in the order of strokes, that is, in the order in which plural strokes were handwritten.

In the time-series information200, the first two stroke data SD1and SD2are indicative of two strokes of the handwritten character “A”. The third and fourth stroke data SD3and SD4are indicative of two strokes of the handwritten character “B”. The fifth stroke data SD5is indicative of one stroke of the handwritten character “C”. The sixth and seventh stroke data SD6and SD7are indicative of two strokes of the handwritten arrow.

Each stroke data includes coordinate data series (time-series coordinates) corresponding to one stroke, that is, a plurality of coordinates corresponding to a plurality of points on the locus of one stroke. In each stroke data, the plural coordinates are arranged in time series in the order in which the stroke is written. For example, as regards handwritten character “A”, the stroke data SD1includes coordinate data series (time-series coordinates) corresponding to the points on the locus of the stroke of the handwritten “Λ” shape of the handwritten character “A”, that is, an n-number of coordinate data SD11, SD12, . . . , SD1n. The stroke data SD2includes coordinate data series corresponding to the points on the locus of the stroke of the handwritten “-” shape of the handwritten character “A”, that is, an n-number of coordinate data SD21, SD22, . . . , SD2n. Incidentally, the number of coordinate data may differ between respective stroke data.

Each coordinate data is indicative of an X coordinate and a Y coordinate, which correspond to one point in the associated locus. For example, the coordinate data SD11is indicative of an X coordinate (X11) and a Y coordinate (Y11) of the starting point of the stroke of the “Λ” shape. The coordinate data SD1nis indicative of an X coordinate (X1n) and a Y coordinate (Y1n) of the end point of the stroke of the “Λ” shape.

Further, each coordinate data may include time stamp information T corresponding to a time point at which a point corresponding to this coordinate data was handwritten. The time point at which the point was handwritten may be either an absolute time (e.g. year/month/day/hour/minute/second) or a relative time with reference to a certain time point. For example, an absolute time (e.g. year/month/day/hour/minute/second) at which a stroke began to be handwritten may be added as time stamp information to each stroke data, and furthermore a relative time indicative of a difference from the absolute time may be added as time stamp information T to each coordinate data in the stroke data.

In this manner, by using the time-series information in which the time stamp information T is added to each coordinate data, the temporal relationship between strokes can be more precisely expressed.

Moreover, information (Z) indicative of a pen stroke pressure may be added to each coordinate data.

Furthermore, in the present embodiment, as described above, a handwritten document is stored not as an image or a result of character recognition, but as the time-series information200which is composed of a set of time-series stroke data. Thus, handwritten characters and graphics can be handled, without depending on languages. Therefore, the structure of the time-series information200of the present embodiment can be commonly used in various countries of the world where different languages are used.

FIG. 4shows a system configuration of the tablet computer10.

As shown inFIG. 4, the tablet computer10includes a CPU101, a system controller102, a main memory103, a graphics controller104, a BIOS-ROM105, a nonvolatile memory106, a wireless communication device107, and an embedded controller (EC)108.

The CPU101is a processor which controls the operations of various modules in the tablet computer10. The CPU101executes various kinds of software, which are loaded from the nonvolatile memory106that is a storage device into the main memory103. The software includes an operating system (OS)201and various application programs. The application programs include a digital notebook application program202. The digital notebook application program202includes a function of creating and displaying the above-described handwritten document, and a clipping function of clipping an arbitrary area or an arbitrary stroke from a handwritten document.

In addition, the CPU101executes a basic input/output system (BIOS) stored in the BIOS-ROM105. The BIOS is a program for hardware control.

The system controller102is a device which connects a local bus of the CPU101and various components. The system controller102includes a memory controller which access-controls the main memory103. In addition, the system controller102includes a function of communicating with the graphics controller104via, e.g. a PCI EXPRESS serial bus.

The graphics controller104is a display controller which controls an LCD17A that is used as a display monitor of the tablet computer10. A display signal, which is generated by the graphics controller104, is sent to the LCD17A. The LCD17A displays a screen image based on the display signal. A touch panel17B and a digitizer17C are disposed on the LCD17A. The touch panel17B is an electrostatic capacitance-type pointing device for executing an input on the screen of the LCD17A. The touch panel17B detects a contact position on the screen, which is touched by a finger, and a movement of the contact position. The digitizer17C is an electromagnetic induction-type pointing device for executing an input on the screen of the LCD17A. The digitizer17C detects a contact position on the screen, which is touched by the pen100, and a movement of the contact position.

The wireless communication device107is a device configured to execute wireless communication such as wireless LAN or 3G mobile communication. The EC108is a one-chip microcomputer including an embedded controller for power management. The EC108includes a function of powering on or powering off the tablet computer10in accordance with an operation of a power button by the user.

Next, referring toFIG. 5, a description is given of a functional configuration of the digital notebook application program202. The digital notebook application program202executes creation, display and edit of a handwritten document, by using stroke data input by a handwritten document input operation using the touch-screen display17. In addition, in accordance with an area input operation of designating an area in a handwritten document, the digital notebook application program202acquires image data of an image corresponding or relating to the designated area, and/or stroke data of a stroke corresponding or relating to the designated area.

The digital notebook application program202includes, for example, a locus display processor301, a time-series information generator302, a candidate area calculator304, an area display processor305, a selector306, a data storage processor307, a data acquisition processor308, a document display processor309, an object reader310, and an object display processor311.

The touch-screen display17is configured to detect the occurrence of events such as “touch”, “move (slide)” and “release”. The “touch” is an event indicating that an external object has come in contact with the screen. The “move (slide)” is an event indicating that the position of contact of the external object has been moved while the external object is in contact with the screen. The “release” is an event indicating that the external object has been released from the screen.

The locus display processor301and time-series information generator302receive an event “touch” or “move (slide)” which is generated by the touch-screen display17, thereby detecting a handwritten document input operation (or an area input operation). The “touch” event includes coordinates of a contact position. The “move (slide)” event includes coordinates of a contact position at a destination of movement. Thus, the locus display processor301and time-series information generator302can receive coordinate series, which correspond to the locus of movement of the contact position, from the touch-screen display17.

The locus display processor301receives coordinate series from the touch-screen display17. The locus display processor301then displays, based on the coordinate series, the locus of each stroke, which is handwritten by a handwritten document input operation (or an area input operation) with use of the pen100or the like, on the screen of the LCD17A in the touch-screen display17. By the locus display processor301, the locus of the pen100during a time in which the pen100is in contact with the screen, that is, the locus of each stroke, is drawn on the screen of the LCD17A.

The time-series information generator302receives the above-described coordinate series output from the touch-screen display17, and then generates, based on the coordinate series, the above-described time-series information (stroke data) having the structure as described in detail with reference toFIG. 3. In this case, the time-series information, that is, the coordinates and time stamp information corresponding to the respective points of each stroke, may be temporarily stored in a working memory401.

The data storage processor307stores the generated time-series information (the time-series information temporarily stored in the working memory401) as handwritten document data in a storage medium402. The storage medium402is, for example, a storage device in the tablet computer10.

The data acquisition processor308reads from the storage medium402arbitrary handwritten document data which is already stored in the storage medium402. The read handwritten document data is sent to the document display processor309. The document display processor309analyzes the handwritten document data and then displays, based on the analysis result, the locus of each stroke indicated by the time-series information on the screen as a handwritten document (handwritten page).

As has been described above, in the time-series information which corresponds to a handwritten document, a plurality of stroke data, which correspond to a plurality of strokes handwritten on the handwritten document, are arranged in time series. Each stroke data includes coordinate data series corresponding to one stroke, that is, a plurality of coordinates corresponding to a plurality of time-series points on the locus of one stroke. By using such time-series information, the strokes handwritten on the handwritten document can be divided into groups based on time points at which the strokes were handwritten.

As illustrated inFIG. 6, in a handwritten document50, a plurality of strokes are divided into seven stroke groups511to517by detecting intervals of input time points between the strokes, based on time points at which the strokes were handwritten. For example, using time-series information, the time-series information generator302determines that an Nth stroke and an (N+1)th stroke belong to different stroke groups, if an elapsed time from a time point of completion of input of the Nth stroke (i.e. a time point associated with last coordinates SD8nof the Nth stroke) to a time point of start of input of the (N+1)th stroke which follows the Nth stroke (i.e. a time point associated with first coordinates SD91of the (N+1)th stroke) is a threshold time or more.

By repeating the above-described determination, the time-series information generator302divides the strokes on the handwritten document50into seven stroke groups511to517. Accordingly, these seven stroke groups511to517are time-series groups. Information indicative of the stroke groups may be temporarily stored in the working memory401.

In addition, by an area input operation using the “area designation” tool, the user can designate an arbitrary part of the handwritten document50(time-series information) which is being displayed. In accordance with an area designated by the area input operation, the selector306selects strokes of a process target.

As illustrated inFIG. 7, for example, the user designates a first area in the handwritten document50by executing an area input operation of handwriting one stroke (area stroke)5A in the handwritten document50.

As described above, the locus display processor301receives, coordinate series from the touch-screen display17. The locus display processor301then displays, based on the coordinate series, the locus of the area stroke5A, which is handwritten by the area input operation using the pen100or the like, on the screen of the LCD17A in the touch-screen display17.

In addition, the time-series information generator302receives the above-described coordinate series from the touch-screen display17, and then generates, based on the coordinate series, time-series information (stroke data) having the structure as described in detail with reference toFIG. 3. Specifically, the time-series information generator302generates stroke data corresponding to the area stroke5A based on the area input operation. In this case, the stroke data, namely the coordinates and time stamp information corresponding to each point of the stroke, may be temporarily stored in the working memory401.

Then, the candidate area calculator304determines a first candidate area corresponding to the area stroke5A, by using the generated stroke data. The area stroke5A constitutes, for example, a closed loop. In this case, the candidate area calculator304determines an area corresponding to this closed loop to be the first candidate area. Incidentally, the area stroke5A may not constitute a closed loop. In this case, the candidate area calculator304estimates a closed loop based on the area stroke5A by linearly or non-linearly interpolating a stroke portion between the beginning and end of the area stroke5A, and determines an area corresponding to the estimated closed loop to be the first candidate area.

The candidate area calculator304further calculates candidate areas, based on the area stroke5A (or the first candidate area).FIG. 8illustrates examples of the first candidate area51and candidate areas52,53and54calculated by the candidate area calculator304.

The candidate area calculator304calculates, for example, a rectangle including the area stroke5A (first candidate area51) (e.g. a rectangle circumscribing the first candidate area51) as a second candidate area52. This second candidate area52is, for example, a rectangular area which is composed of two sides which are parallel to the horizontal direction of the handwritten document50and two sides which are parallel to the vertical direction of the handwritten document50.

In addition, the candidate area calculator304calculates an area including an offset using time information of time points at which a plurality of strokes in the handwritten document50were handwritten, so that a semantic relation between the strokes in the handwritten document may be complemented. The candidate area calculator304detects, for example, from the plural strokes in the handwritten document50, strokes which are at least partly included in the calculated rectangular area (second candidate area)52, and calculates a period in which the detected strokes were handwritten. Then, the candidate area calculator304detects, from the plural strokes in the handwritten document50, strokes which were handwritten during the determined period, and calculates a third candidate area53including these strokes. In the example illustrated inFIG. 8, a period from a time point at which a stroke SD10was handwritten to a time point at which a stroke SD1N was handwritten is calculated, and strokes handwritten during this period are further detected. Specifically, strokes corresponding to “sample a” and strokes corresponding to “>sample b” are further detected. Then, the candidate area calculator304determines a third candidate area53including the detected strokes.

Furthermore, the candidate area calculator304detects, for example, from the plural strokes in the handwritten document50, strokes which are at least partly included in the calculated rectangular area (second candidate area)52, and calculates a fourth candidate area54which further includes strokes belonging to the same stroke group as each of the detected strokes. In the example illustrated inFIG. 8, strokes of “sample a”, strokes of “>sample b” and strokes of “>sample c” are detected as strokes belonging to the same stroke groups515,516and517as the strokes which are at least partly included in the rectangular area (second candidate area)52. Then, the fourth candidate area54which further includes these strokes is determined. The stroke groups are as have been described with reference toFIG. 6.

The area display processor305displays the calculated candidate areas51,52,53and54on the display17A. The user executes an area select operation of selecting, for example, an area corresponding to strokes which are to be selected, from among the displayed candidate areas51,52,53and54.

The selector306determines the area which has been selected by the user from among the candidate areas51,52,53and54, in accordance with the area select operation using the touch-screen display17. Then, based on the area selected by the user, the selector306selects strokes (hereinafter referred to also as “target block”) among the plural strokes on the handwritten document50, based on the area selected by the user. Besides, when the area has been selected by the area select operation, the area display processor305may erase the candidate areas51,52,53and54from the screen.

FIG. 9illustrates examples of strokes (target block) which are selected by the selector306in accordance with the area select operation.

A target block61includes a plurality of strokes85which are included in the first candidate area51corresponding to the area stroke5A. A target block62includes a plurality of first strokes87which are at least partly included in the second candidate area (the rectangular area including the area stroke5A)52.

A target block63includes strokes which are included in the third candidate area53. Specifically, the target block63includes a plurality of first strokes87which are at least partly included in the rectangular area52including the area stroke5A, and a plurality of second strokes88(i.e. “sample a” and “>sample b”) which were handwritten during the period in which the plural first strokes87were handwritten. That is, the selector306sets an offset of an area by using information of time points at which the strokes were handwritten. The second strokes88are strokes which were handwritten at time points between that one of the first strokes, which was handwritten at the earliest time point, and that one of the first strokes, which was handwritten at the last time point.

A target block64includes strokes which are included in the fourth candidate area54. Specifically, the target block64includes a plurality of first strokes87which are at least partly included in the rectangular area52including the area stroke5A, and a plurality of third strokes89(i.e. “sample a”, “>sample b” and “>sample c”) which belong to the same groups as the plural first strokes87. The third strokes89are strokes which were handwritten continuous with the plural first strokes87.

The selector306selects one or more stroke data corresponding to a selected target block (i.e. selected one or more strokes) from a plurality of stroke data (time-series information) corresponding to a plurality of strokes handwritten on the handwritten document50. The selected one or more stroke data are, for example, a copy of a part of the plural stroke data (time-series information). The selected stroke data is temporarily stored, for example, in the working memory401. In the meantime, the selector306may generate data of an image (clipping image) in which the selected stroke is drawn and store the generated data.

The selector306further reads the temporarily stored stroke data in accordance with an area paste operation using the touch-screen display17. The read stroke data is sent to the area display processor305. The area display processor305analyzes the stroke data and then displays (draws) in a target document the locus of each stroke indicated by the stroke data, based on the analysis result. This target document is a document which is set in an active state when the area paste operation is executed, and is, for example, the handwritten document (first document)50or a handwritten document (second document) which is different from the handwritten document50.

In the meantime, when a plurality of candidate areas are displayed on the handwritten document50, the area display processor305may effect such display that strokes, which are obtained (cut out) when each of the plural candidate areas is selected, can be discriminated by the user.

By the above-described configuration, the user can easily select not only the area51corresponding to the area stroke5A, which was input by the area input operation, but also the areas52,53and54relating to the area stroke5A. In addition, by pasting the strokes corresponding to the selected area on the handwritten document50or another handwritten document, the user can easily execute edit of handwritten documents, such as reconstruction of the handwritten document50or integration of a plurality of handwritten documents. The areas52,53and54relating to the area stroke5A may present to the user an area including strokes having a relativity of which the user is not aware. Thus, the user can confirm the areas52,53and54relating to the area stroke5A as annotations or suggestions at a time of editing handwritten documents, and can efficiently execute edit of the handwritten documents.

In the meantime, the area stroke5A can be altered after the input of the area stroke5A was completed.

FIG. 10illustrates an example in which the area stroke5A is corrected. In this example, it is assumed that the user has executed an operation of dragging a point561on the area stroke5A to a point564by using the touch-screen display17.

The candidate area calculator304and area display processor305display on the screen a corrected area stroke56by linearly or non-linearly interpolating a stroke portion between a point562on the area stroke5A and the point564, and a stroke portion between the point564and a point563on the area stroke5A. The candidate area calculator304calculates a candidate area, based on the corrected area stroke56, and the area display processor305displays the calculated candidate area on the screen.

Thereby, for example, when an input area stroke is not a stroke intended by the user, this input area stroke can easily be corrected.

FIG. 11illustrates another example of strokes selected based on the area stroke. On a handwritten document71, strokes included in an area, which has been calculated based on an area stroke72, are selected from among a plurality of strokes on the handwritten document71. An area73includes a plurality of first strokes which are at least partly included in the area stroke72(the area corresponding to the area stroke72), and strokes which were handwritten continuously before or after these plural first strokes.

By this selection of strokes, the user can easily select the area73corresponding to a group of strokes, such as a paragraph, by simply inputting the area stroke72which designates a rough area.

On the handwritten document50, images or various objects may further be arranged.

The object reader310and object display processor311dispose on the handwritten document50an object such as an image, a graphic, a file path, an icon indicative of a link to a file, a URL, a formula, or a graph, in accordance with an object input operation using the touch-screen display17. The user executes an object input operation of selecting an object, which is to be disposed on the handwritten document50, for example, from a list of various objects, and designating a position on the handwritten document50, at which the selected object is to be disposed.

The object reader310reads the object, which has been selected by the object input operation, from a storage such as the storage medium402. The object display processor311displays the read object at the position designated by the object input operation. Incidentally, the object reader310may temporarily store object information indicative of the read object in the working memory401. The data storage processor307stores the generated time-series information (the time-series information temporarily stored in the working memory401) and the object information (the object information temporarily stored in the working memory401) as handwritten document data in the storage medium402.

The object can be disposed at an arbitrary position in the handwritten document50. As illustrated inFIG. 12, an image58can also be disposed as a background image of the handwritten document50. In addition, after the object (image) is disposed, characters or graphics may be handwritten by a handwritten document input operation.

Even in the handwritten document50in which the object (image)58and strokes are mixed, the user can designate an arbitrary part of the handwritten document50by an area input operation using the “area designation” tool. In accordance with an area designated by the area input operation, strokes and an object, which are a process target, are selected by the selector306. When a handwritten document (first document) including strokes and one or more images (objects)58is being displayed on the screen, the selector306selects first stroke data corresponding to a first stroke of the strokes and first image data corresponding to a first part in the images, based on an area stroke for designating a first area.

In the example illustrated inFIG. 12, the handwritten document50, in which the image58is disposed on the background and strokes handwritten on the image58, is displayed. In this example, an area in the handwritten document50, which includes a part of the image58, is designated in accordance with an area input operation of handwriting one stroke (area stroke)5A.

When the handwritten document (first document) including plural strokes and one or more images corresponding to the handwritten document input operation is being displayed, the locus display processor301displays the locus of the area stroke5A, which is handwritten in accordance with the area input operation, on the screen of the LCD17A in the touch-screen display17. In addition, the time-series information generator302generates stroke data (coordinate data series) corresponding to the area stroke5A.

The selector306selects, based on the area stroke5A, a first stroke among the plural strokes and a first part in one or more images in the handwritten document. Then, the selector306selects first stroke data corresponding to the selected stroke from plural stroke data corresponding to the plural strokes, and selects first image data corresponding to the selected first part from one or more image data corresponding to the one or more images.

To be more specific, as has been described in detail with reference toFIG. 8, the candidate area calculator304and area display processor305calculate a plurality of candidate areas, based on the area stroke5A, and display the plural candidate areas on the screen. In response to selection of one area from among the displayed plural candidate areas, the selector306selects a first stroke corresponding to the one area from the plural strokes in the handwritten document50and selects a first part corresponding to the one area in the image58in the handwritten document50.

FIGS. 13 and 14illustrate examples of strokes and an image part (first part), which are selected by the selector306in accordance with an area select operation.

A target block81illustrated inFIG. 13includes a plurality of strokes85and an image part (first part)86, which are included in the first area in the handwritten document50corresponding to the area stroke5A. A target block82includes a plurality of first strokes87which are at least partly included in the rectangular area52including the area stroke5A (e.g. the rectangular area circumscribing the area stroke5A), and the image part (first part)86included in the first area corresponding to the area stroke5A.

A target block83includes a plurality of first strokes87which are at least partly included in the rectangular area52including the area stroke5A, a plurality of second strokes88(i.e. “sample a” and “>sample b”) which were handwritten during the period in which the plural first strokes87were handwritten. The target block83further includes an image part (first part)86included in the first area corresponding to the area stroke5A. Specifically, the selector306sets an offset of an area using time information of time points at which the strokes were handwritten. The second strokes88are strokes which were handwritten at time points between that one of the first strokes87, which was handwritten at the earliest time point, and that one of the first strokes87, which was handwritten at the last time point.

A target block84includes a plurality of first strokes87which are at least partly included in the rectangular area52including the area stroke5A, a plurality of third strokes89(i.e. “sample a”, “>sample b” and “>sample c”) which belong to the same groups as the plural first strokes87. The target block84further includes an image part (first part)86included in the first area corresponding to the area stroke5A. The third strokes89are strokes which were handwritten continuous with the plural first strokes87.

A target block91illustrated inFIG. 14includes a plurality of strokes87which are at least partly included in the rectangular area52including the area stroke5A, and an image part (first part)95included in this rectangular area.

A target block92includes a plurality of first strokes87which are at least partly included in the rectangular area52including the area stroke5A, and a plurality of second strokes88(i.e. “sample a” and “>sample b”) which were handwritten during the period in which the plural first strokes87were handwritten. The target block92further includes an image part (first part)96corresponding to a rectangular area including the first strokes87and second strokes88.

A target block93includes a plurality of first strokes87which are at least partly included in the rectangular area52including the area stroke5A, and a plurality of third strokes89(i.e. “sample a”, “>sample b” and “>sample c”) which belong to the same groups as the plural first strokes87. The target block93further includes an image part (first part)97corresponding to a rectangular area including the first strokes87and third strokes89.

The selector306selects one or more stroke data and image data corresponding to a selected target block (i.e. selected one or more strokes and image part) from a plurality of stroke data (time-series information) corresponding to a plurality of strokes in the handwritten document50, and image data corresponding to one or more images. The selected one or more stroke data and image data are, for example, a copy of a part of the plural stroke data (time-series information) and a copy of a part of the image data corresponding to the one or more images. The selected stroke data and image data are temporarily stored, for example, in the working memory401. In the meantime, the selector306may generate data of an image (clipping image) in which the selected stroke and image part (first part) are drawn and store the generated data. In addition, the data storage processor307may store in the storage medium402the stroke data and image data which are temporarily stored in the working memory401.

The selector306further reads the temporarily stored stroke data and image data in accordance with an area paste operation using the touch-screen display17. The read stroke data and image data are sent to the area display processor305. The area display processor305analyzes the stroke data and displays (draws) in a target document the locus of each stroke indicated by the stroke data and an image indicated by the image data, based on the analysis result. This target document is a document which is set in an active state when the area paste operation is executed, and is, for example, the handwritten document (first document)50or a handwritten document (second document) which is different from the handwritten document50.

In the meantime, when a plurality of candidate areas are displayed on the handwritten document50, the area display processor305may effect such display that the strokes and image part (first part), which are obtained (cut out) when each of the plural candidate areas was selected, can be discriminated by the user.

By the above-described configuration, the user can easily select not only the area corresponding to the area stroke5A, which is input by the area input operation, but also the areas relating to the area stroke5A. In addition, by pasting the strokes and image, which correspond to the selected area, on the handwritten document50or another handwritten document, the user can easily execute edit of handwritten documents, such as reconstruction of the handwritten document50or integration of a plurality of handwritten documents.

The above description has been given of the example in which the object is an image. However, another object disposed in the handwritten document50can also be selected in accordance with the area input operation. For example, when one or more objects are included in the handwritten document50, the selector306selects an object of the one or more objects based on the area stroke5A. Then, the selector306selects object data corresponding to the selected object, from among one or more object data corresponding to the one or more objects, and temporarily stores the selected object data in the storage medium402. Incidentally, in the case of an object of a character string such as a file path or URL, if a part of the characters is lost, access to a file or a link is disabled. Thus, in the case of the handwritten document50including the object of the character string, the selector306may select, for example, the entirety of the object which is at least partly included in the first area corresponding to the area stroke5A. In addition, when a first object, which is not included in the first area corresponding to the area stroke5A, is associated with a stroke or an image (object) in the first area, the selector306may select this first object (object data corresponding to the first object).

In addition, the candidate area calculator304can also calculate an area including an offset using information of positions (coordinates) at which a plurality of strokes in the handwritten document50were handwritten, so that a semantic relation between the strokes may be complemented.

As illustrated inFIG. 15, the candidate area calculator304calculates, for example, an area55which is obtained by enlarging the rectangular area (second candidate area)52including the area stroke5A by an offset based on a character size. Alternatively, the candidate area calculator304calculates an area which is obtained by reducing the rectangular area (second candidate area)52by this offset. For example, the candidate area calculator304calculates a maximum value (e.g. pixel unit) of the height and width of each stroke in the handwritten document50, or a mean value of the height and width of each stroke, as a character size (a size of one side of the rectangle circumscribing a character) intended by the user. Incidentally, as this character size, it is possible to set an arbitrary value in a range from the minimum value to the maximum value of the height and width of plural strokes in the handwritten document50.

Thereby, as illustrated inFIG. 16, the area selector306selects, for example, strokes included in the area55which is enlarged by an offset corresponding to one character, and then selects stroke data corresponding to the selected strokes from among plural stroke data corresponding to the plural strokes in the handwritten document50. In the meantime, the offset used for enlargement or reduction is not limited to the offset corresponding to one character, and may be an offset corresponding to a plurality of characters.

Next, referring toFIG. 17, a description is given of an example of the procedure of a handwriting input process executed by the digital notebook application202. In this example, it is assumed that the digital notebook application202is set in a first mode for inputting a handwritten document (handwritten character or graphic).

To start with, the locus display processor301displays on the display17A the locus (stroke) of movement of the pen100or the like by a document input operation (block B11). In addition, the time-series information generator302generates the above-described time-series information (a plurality of stroke data arranged in time series) based on coordinate series corresponding to the locus by the document input operation (block B12). The time-series information generator302may temporarily store the time-series information in the working memory401.

Subsequently, the object reader310determines whether an object input operation for inserting various objects (an image, a graphic object, a file path, an icon, a URL, a formula, a graph, etc.) in the handwritten document has been detected or not (block B13). When the object input operation is not detected (NO in block B13), the process returns to block B11, and the input of a handwritten document corresponding to a handwritten document input operation is continued.

When the object input operation has been detected (YES in block B13), the object reader310reads the object, which has been designated by the object input operation, from the storage such as the storage medium402(block B14). This object may be read from storage of a server over a network. The object display processor311displays the read object on the handwritten document on the display17A (block B15). In this object input operation, the position at which the object is displayed, or the size of the object, may be designated. Incidentally, the object reader310may temporarily store object information indicative of the inserted object (e.g. a file path of an image, identification information of a graphic object, a position and size of an object, etc.) in the working memory401.

Further, the data storage processor307stores the time-series information generated by the time-series information generator302(the time-series information temporarily stored in the working memory401) and the information read by the object reader310(the object information temporarily stored in the working memory401) as handwritten document data in the storage medium402.

FIG. 18illustrates an example of the procedure of an area select process executed by the digital notebook application202. In this example, it is assumed that the digital notebook application202is set in a second mode for selecting an area in a handwritten document.

To start with, the locus display processor301displays on the display17A the locus (area stroke) of movement of the pen100or the like by an area input operation (block B21). In addition, the time-series information generator302generates the above-described time-series information (one stroke data) based on coordinate series corresponding to the locus by the area input operation (block B22). The time-series information generator302may temporarily store the time-series information in the working memory401.

Subsequently, the candidate area calculator304calculates candidate areas on the handwritten document by using the generated time-series information (area stroke) (block B23). The area display processor305displays the calculated candidate areas on the display17A (block B24).

The selector306determines whether one area has been selected from among the candidate areas based on the area input operation (block B25). If no area has been selected (NO in block B25), the process returns to block B25, and it is determined once again whether an area has been selected or not.

When an area has been selected (YES in block B25), the selector306determines one or more strokes (and image part) in the handwritten document, which correspond to the selected area (block B26). The selector306acquires, for example, stroke data corresponding to the determined one or more strokes (and image data corresponding to the selected image part). Thereby, the user can instruct the digital notebook application to cut out arbitrary strokes (and image part) from the handwritten document, and to paste the cut-out strokes (and image part) on another area in this handwritten document or on another document.

As has been described above, according to the present embodiment, a handwritten document can easily be handled. The locus display processor301displays on the screen an area stroke for designating a first area, when a first document including a plurality of strokes corresponding to a handwritten document input operation and one or more images is displayed on the screen. The selector306selects first stroke data corresponding to a first stroke of the plural strokes and first image data corresponding to a first part in the one or more images based on the area stroke.

The user inputs an area stroke (freehand object) which designates an arbitrary area on the screen on which the handwritten document is displayed, by an operation using a pointing device such as a touch operation, a stylus operation or a mouse operation. In accordance with the input, strokes and an image (object) in the handwritten document, which correspond or relate to this area stroke, are acquired. Then, the user can use the acquired strokes and image by pasting them on another area in this handwritten document or on another document.

All the procedures in the present embodiment, which have been described with reference to flowcharts ofFIGS. 17 and 18, can be executed by software. Thus, the same advantageous effects as with the present embodiment can easily be obtained simply by installing a computer program, which executes the process procedures, into an ordinary computer through a computer-readable storage medium which stores the computer program, and by executing the computer program.