Abstract:
Systems and methods for detecting annotation digital ink strokes and further associating annotation digital ink strokes with word digital ink strokes are presented. Ink strokes are captured on a writing surface and then classified as words or annotations. Annotations are then anchored to corresponding words. When words are relocated or edited on the writing surface, the anchored annotations are also relocated and may even be reshaped according to the changes in the anchored words.

Description:
CROSS REFERENCE TO RELATED APPLICATIONS 
     This application is a continuation of U.S. patent application Ser. No. 11/275,398, filed on Dec. 29, 2005, and entitled “Annotation Detection and Anchoring on Ink Notes”, the entirety of which is incorporated herein by reference. 
    
    
     BACKGROUND 
     Computer users are accustomed to using a mouse and keyboard as a way of interacting with a personal computer. While personal computers provide a number of advantages over printed documents, users often continue to perform certain functions using printed paper. Some of these functions include reading handwritten documents. In the case of annotations, the printed document assumes a greater significance because of the annotations placed on it by the user. One of the difficulties, however, with having a hand-written document is a later need to have the content entered back into the electronic form of the document. This requires the original user or another user to wade through the handwritten content and enter them into a personal computer. In some cases, a user will scan in the handwritten content and the original text, thereby creating a new document. These multiple steps make the interaction between the printed document and the electronic version of the document difficult to handle on a repeated basis. Further, scanned-in images are frequently non-modifiable. There may be no way to separate the handwritten content from the original text. This makes using the handwritten content difficult. Accordingly, an improved way of handling handwritten content is needed. 
     Tablet PC&#39;s are increasingly being used to capture and display handwritten content. In the context of taking notes during a meeting, for example, a Tablet PC presents less of a barrier between individuals than does a laptop computer. Tablet PC&#39;s also allow for capturing of handwritten notes so that Tablet PC users are able to take notes in their own handwriting, in an unobtrusive manner similar to taking handwritten notes with a pen and paper. 
     Tablet PC&#39;s capture and display digital ink. The term “digital ink” refers to one or more strokes that are recorded from a pointing device, such as a mouse, a stylus/pen on a digitizer tablet, or a stylus/pen on a display screen integrated with a digitizer tablet (e.g., a touch-sensitive display screen). As used herein, the term “ink” is shorthand for digital ink. 
     Documents displayed on writing surfaces of tablet PCs may be annotated by a user. Correspondingly, annotations such as underline and blobs are often incorporated within the document. For example, a user may underline a few words in a document to emphasize the words underlined. Thus, extraction of handwritten annotations plays an important role in document processing. 
     SUMMARY 
     An input device is used to enter a plurality of digital ink strokes on a writing surface, such as a Tablet PC screen. Digital ink strokes are captured and may then be classified. The digital ink strokes may be classified as word ink strokes and annotation ink strokes. Annotation ink strokes are anchored to corresponding word ink strokes. When word ink strokes are relocated on the writing surface, the anchored annotation ink strokes are also relocated. 
     This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the clamed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter. 
    
    
     
       BRIEF DESCRIPTION OF DRAWINGS 
       The foregoing summary of the invention, as well as the following detailed description of preferred embodiments, is better understood when read in conjunction with the accompanying drawings, which are included by way of example, and not by way of limitation with regard to the claimed invention. 
         FIG. 1  is a functional block diagram of an example of a conventional general-purpose digital computing environment that can be used to implement various aspects of the present invention. 
         FIG. 2  shows an exemplary document that may be displayed on a computer device writing surface, in accordance with an embodiment of the invention. 
         FIG. 3  illustrates a system that maybe used to classify and anchoring annotations. Digital ink stroke data may be delivered to one or more classification and anchoring engines, in accordance with an embodiment of the invention. 
         FIG. 4  shows a document having modified annotation positions, in accordance with an embodiment of the invention. 
         FIG. 5  illustrates a method of associating an annotation with one or more words represented by a plurality of digital ink strokes, in accordance with an embodiment of the invention. 
     
    
    
     DETAILED DESCRIPTION 
     Aspects of the present invention relate to classifying annotation digital ink strokes and anchoring the annotation digital ink strokes with word digital ink strokes.  FIG. 1  is a functional block diagram of an example of a conventional general-purpose Tablet PC digital computing environment that can be used to process a writing input from a Pen (e.g., stylus  166 ). As used herein, the term Tablet PC refers to, by way of example, without limitation, a full-function “MICROSOFT” “WINDOWS” operating system-based personal computer incorporating various convenient and intuitive aspects of pencil and paper into a user&#39;s interaction with a PC. In  FIG. 1 , a computer  100  includes a processing unit  110 , a system memory  120 , and a system bus  130  that couples various system components including the system memory to the processing unit  110 . The system bus  130  may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. The system memory  120  includes read only memory (ROM)  140  and random access memory (RAM)  150 . 
     A basic input/output system  160  (BIOS), containing the basic routines that help to transfer information between elements within the computer  100 , such as during start-up, is stored in the ROM  140 . The computer  100  also includes a hard disk drive  170  for reading from and writing to a hard disk (not shown), a magnetic disk drive  180  for reading from or writing to a removable magnetic disk  190 , and an optical disk drive  191  for reading from or writing to a removable optical disk  192  such as a CD ROM or other optical media. The hard disk drive  170 , magnetic disk drive  180 , and optical disk drive  191  are connected to the system bus  130  by a hard disk drive interface  192 , a magnetic disk drive interface  193 , and an optical disk drive interface  194 , respectively. The drives and their associated computer-readable media provide nonvolatile storage of computer readable instructions, data structures, program modules and other data for the personal computer  100 . It will be appreciated by those skilled in the art that other types of computer readable media that can store data that is accessible by a computer, such as magnetic cassettes, flash memory cards, digital video disks, Bernoulli cartridges, random access memories (RAMs), read only memories (ROMs), and the like, may also be used in the example operating environment. 
     A number of program modules can be stored on the hard disk drive  170 , magnetic disk  190 , optical disk  192 , ROM  140  or RAM  150 , including an operating system  195 , one or more application programs  196 , other program modules  197 , and program data  198 . A user can enter commands and information into the computer  100  through input devices such as a keyboard  101  and pointing device  102 . Other input devices (not shown) may include a microphone, joystick, game pad, satellite dish, scanner or the like. These and other input devices are often connected to the processing unit  110  through a serial port interface  106  that is coupled to the system bus, but may be connected by other interfaces, such as a parallel port, game port or a universal serial bus (USB). Further still, these devices may be coupled directly to the system bus  130  via an appropriate interface (not shown). A monitor  107  or other type of display device is also connected to the system bus  130  via an interface, such as a video adapter  108 . In addition to the monitor, personal computers typically include other peripheral output devices (not shown), such as speakers and printers. 
     In an embodiment, a writing surface, such as a pen digitizer  165 , and accompanying pen or stylus  166  are provided in order to digitally capture freehand input to provide a writing input (e.g., a handwritten document) for processing unit  110 . Although a direct connection between the pen digitizer  165  and the serial port is shown, in practice, the pen digitizer  165  may be coupled to the processing unit  110  directly, via a parallel port or other interface and the system bus  130  as known in the art. Furthermore, although the digitizer  165  is shown apart from the monitor  107 , it is preferred that the usable input area of the digitizer  165  be co-extensive with the display area of the monitor  107 . Further still, the digitizer  165  may be integrated in the monitor  107 , or may exist as a separate device overlaying or otherwise appended to the monitor  107 . 
     The computer  100  can operate in a networked environment using logical connections to one or more remote computers, such as a remote computer  109 . The remote computer  109  can be a server, a router, a network PC, a peer device or other common network node, and typically includes many or all of the elements described above relative to the computer  100 , although only a memory storage device  111  has been illustrated in  FIG. 1 . The logical connections depicted in  FIG. 1  include a local area network (LAN)  112  and a wide area network (WAN)  113 . Such networking environments are commonplace in offices, enterprise-wide computer networks, intranets and the Internet. 
     When used in a LAN networking environment, the computer  100  is connected to the local network  112  through a network interface or adapter  114 . When used in a WAN networking environment, the personal computer  100  typically includes a modem  115  or other means for establishing a communications over the wide area network  113 , such as the Internet. The modem  115 , which may be internal or external, is connected to the system bus  130  via the serial port interface  106 . In a networked environment, program modules depicted relative to the personal computer  100 , or portions thereof, may be stored in the remote memory storage device. 
     Aspects of the invention relate to annotation detection and anchoring. An annotation may be in the form of an underline, blob or any other annotation, e.g. callout, that a user would add to a document. An underline, for example, refers to a stroke or a series of strokes drawn under lines of a document. An underline annotates the words above the underline. A blob consists of one or more digital ink strokes that form a closed pattern. A blob may encloses words, table, an image or other elements of a document. A blob annotates the words or other elements surrounded by the digital ink strokes(s). 
       FIG. 2  shows an exemplary document  200  that may be displayed on a computer device writing surface, such as a Tablet PC writing surface. In creating document  200 , a user may use a pen to create digital ink strokes. The digital ink strokes are captured by the computer device and then displayed to the user.  FIG. 2  shows that several digital ink strokes form words. For example, a group of digital ink strokes  202  form the word “click.” Other digital ink strokes form annotations. Digital ink strokes  204  form a blob annotation that surrounds digital ink strokes that form the words “inbox”  204   a  and “folder”  204   b.  Digital ink strokes  206  form an underline annotation that is positioned below several word digital ink strokes that form the words “select”  206   a,  “rules”  206   b , “and”  206   c,  “alerts”  206   d ″ and “from”  206   e.    
       FIG. 3  illustrates a system that maybe used to classify and anchor annotations. Digital ink stroke data may be delivered to one or more classification and anchoring engines. Each digital ink stroke may be stored as one or more ink packets, in which each ink packet may contain coordinates (x, y) corresponding to the position of the pointing device. For example, a user may move a pen along a touch-sensitive display screen of a computer system so as to draw a line or curve, and the computer system may sample the coordinates (x, y) along the trajectory of the pen tip position over time (or on any other interval as known in the art) as the user moves the pen. These coordinates represent points along the curve or line and are stored as ink packets. 
     An underline classification and anchoring engine  304  may be configured to receive digital ink stroke data  302  and determine which ink strokes represent underline annotations. Underline classification and anchoring engine  304  may utilize a processor, computer-executable instructions and/or a rules engine configured to compare attributes of the ink strokes to underline annotation criteria  306 . For example, underline classification and anchoring engine  304  may classify one or more ink strokes as representing an underline if the ink strokes are beneath and adjacent to writing lines. One skilled in the art will appreciate that various other methods exist for classifying digital ink strokes as word digital ink strokes. Other criteria may also be used to classify ink strokes as underline ink strokes. For example, underline classification and anchoring engine  304  may be configured to only consider ink strokes that are drawing ink strokes. Drawing ink strokes are those that do not have attributes of word ink strokes and are part of graphical objects. 
     Other criteria may include classifying an ink stroke as an underline ink stroke when the ink stroke has a relatively large width to height ratio or when the ink stroke overlaps a word in a horizontal direction. One skilled in the art will appreciate that additional, alternative and combinations of criteria may be used to classify an ink stroke as an underline ink stroke. Underline may also be composed of multiple strokes. Dynamic programming may be used to identify strokes that are part of the same underline annotation and determine when the search for strokes that are part of the same underline should end. This may include analyzing the features described above. It may then be determined if ink strokes close to one another form a single underline, for example, considering the proximity of the ink strokes. 
     Underline classification and anchoring engine  304  may also be configured to anchor underline annotations to words or other document elements. Anchoring may include associating one or more underline ink strokes with one or more word or other ink strokes. Underline data  308  may include an ink stroke identification that uniquely identifies the ink stroke and anchoring data that uniquely identifies the word ink strokes or other elements to which the annotation ink stroke is anchored. 
     Once annotation ink strokes are properly anchored, the document may be rearranged without loosing the annotations. For example, if a user underlines a segment of text and then moves the text segment to another part of the document, the annotation can be recreated at the new location.  FIG. 4  shows a document  400  that is a modified version of document  200  (shown in  FIG. 2 ). Words  206   a,    206   b,    206   d,    206   e  and  204   a  have been moved from the positions they had in document  200 . The movement may result from the user selecting one or more words  206   a,    206   b,    206   d,    206   e  and  204   a  with a pointing device and dragging words  206   a,    206   b,    206   d,    206   e  and  204   a  to their new locations. Underline annotation  402  was anchored to words  206   a  and  206   b  and underline annotation  404  was anchored to words  206   d  and  206   e.  A single underline annotation may be broken into two or more underline annotations. For example, in  FIG. 2  underline annotation  206  is under words  206   a - 206   e.  When word “and”  206   c  is removed in  FIG. 4  underline annotations  402  and  404  are formed under the remaining words. Similarly, blob annotation  204  is formed around words  204   a  and  204   b  in  FIG. 2 . When the word “folder”  204   b  is removed in  FIG. 4 , blob annotation  406  is formed only around the word “inbox”  204   a.  Of course, the size of a blob annotation may be reduced or increased to fit around a given collection of words. 
     The system shown in  FIG. 3  also includes a blob classification and anchoring engine  310 . Blob classification and anchoring engine  310  may be configured to operate in a manner similar to that of underline classification and anchoring engine  304 . An exemplary criterion used to classify an ink stroke as a blob includes determining whether one or more ink strokes form a closed pattern. Of course, a “closed pattern” as used herein is meant to encompass a substantially enclosed pattern. Another criterion may include whether or not the closed pattern encloses words or other document elements. Blob classification and anchoring engine  308  may produce blob data  310 . 
     One skilled in the art will appreciate that aspects of the invention may be used to classify and anchor other types of annotations, such as vertical brackets, call outs and highlighting. A miscellaneous classification and anchoring engine  314  is included in the system shown in  FIG. 3  to show that the system may be adapted to classify and anchor other types of annotations and may produce annotation data  316 . When miscellaneous classification and anchoring engine  314  is used to classify multiple types of annotations, annotation data  316  may include a field that identifies the type of annotation. Underline data  308  and blob data  312  may also include fields that identify the type of annotation. 
       FIG. 5  illustrates a method of associating an annotation with one or more words represented by a plurality of digital ink strokes. First, in step  502  a plurality of digital ink strokes are received. The digital ink strokes may be filtered so that they consist only of drawing inks strokes and/or are not words, images or other document elements. Next, in step  504  at least one digital ink stroke is classified as an annotation. They classification may use the criteria described above and may result in the digital ink stroke(s) being classified as an underline, blob or other type of annotation. After annotation digital ink strokes are identified, they are anchored to one or more digital ink strokes that represent one or more words in step  506 . Next, it is determined whether a command to reposition word digital ink strokes on a writing surface are received in step  508 . When a command is not received, in step  510  the process waits a predetermined time period before performing step  508  again. When a command is received, in step  512 , the word digital ink strokes and anchored annotation digital ink strokes are repositioned. 
     Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.