Abstract:
A document organizing system extracts annotations made to a document along with the context surrounding each annotation and organizes the annotations based upon the annotation attributes and/or context. The annotations are created by grouping marks based upon their proximity in time and space. The document is segmented to determine a minimum context associated with each annotation. A list of the annotations sorted by the attributes are then displayed to the user. The context provided by the invention for each annotation allows the user to fully understand the annotation.

Description:
BACKGROUND OF THE INVENTION 
     1. Field of Invention 
     This invention is directed to a document organizing system. In particular, this invention is directed to a method and a system for organizing documents based upon the context of annotations made to those documents. 
     2. Description of Related Art 
     When people read paper documents, they often make annotations to highlight interesting or controversial passages and to record their reactions. Common annotations include margin notes, vertical bars, stars, circles, underlines, highlights, etc. Two advantages of annotating directly on the page are its low overhead and convenience. One disadvantage is that the recorded information is hidden and inaccessible until the reader returns to the specific page in the specific document. 
     To avoid this problem, some readers use a separate reading notebook to record their annotations. A reading notebook is useful because it provides a separate summary of what the user has read along with any commentary. The advantage of a reading notebook is that it permits a quick review of the material because it generally has less information to browse and search than the original document. One disadvantage of a reading notebook, however, is that the reader must recreate the context for each note to fully understand the meaning of each note. 
     Readers also use note cards to organize notes. The advantage of a note card system is that the cards can be easily reorganized. However, as with a reading notebook, unless the reader recreates it, there is no context available to permit the user to fully understand the notes. Additionally, each note must be categorized onto the correct card before it can be recorded. 
     Handwritten notes and keywords are used in a system known as “Marquee” to index video. This system is described in “Marquee: A Tool for Real-Time Video Logging”, K. Webber et al.,  Proceedings of CHI &#39; 94, April 1994, pp. 58-64, incorporated herein by reference in its entirety. In “Marquee”, notes are synchronized to a video string with time zones that are created with horizontal line gestures. Keywords are identified by the user by circling the words and notes that the user has selected as keywords. The keywords are assigned to the time zone in which the keyword is created. Keywords also may be assigned directly by the user by typing the keyword in manually. Because the keywords are associated with time the user can view an index of time zones and go directly to the video by selecting a time zone using an index of the previously identified keyword or annotations. Although “Marquee” uses annotations to index a video document, it does not combine the annotations with the document in a visual way. “Marquee” is thus analogous to notetaking in a separate notebook rather than on the document itself. 
     “Dynomite” is a free-form digital “ink” notebook. The digital ink notebook is a pen-based computer that the user controls by writing with a pen directly on the screen of the computer. The computer senses the location and the positions traversed as the pen moves across the display and assigns ink marks that correspond with the positions of the pen. These ink marks are called digital ink because the ink is described by the computer digitally. Dynomite extracts the ink, assigns properties to each ink mark and can present a list of the ink marks sorted by the assigned properties. This list is known as an ink index. This system is described in co-assigned and co-pending U.S. patent application, Ser. No. 08/821,311, filed on Mar. 20, 1997, entitled “System for Capturing and Retrieving Audio Data and Corresponding Handwritten Notes”, and “Dynomite: A Dynamically Organized Ink and Audio Notebook”, by L. Wilcox et al.,  In CHI &#39; 97  Conference Proceedings,  ACM Press, 1997, pp. 186-193, incorporated herein by reference in their entireties. This ink index shows a “type” of the “ink” along with a time stamp and provides links to the original notebook pages. Dynomite&#39;s ink index provides “ink” marks linked to the corresponding full notebook page. However, Dynomite organizes only the ink notes themselves and not the associated information. 
     “ComMentor” is a platform for shared annotations that attaches text-based comments to locations within web documents. This system is described in “Shared Web Annotations as a Platform for Third-Party Value-Added Information Providers: Architecture, Protocols, and Usage Examples”, by M. Roscheisen, et al., Technical Report STAN-CS-TR-97-1582, Stanford Integrated Digital Library Project, Computer Science Department, Stanford University, November 1994, Updated April 1995, incorporated herein by reference in its entirety. Annotations are grouped into sets. A user can filter these sets and tour through documents within a set. A tour window shows a list of annotations, each annotation shown with the document title of the annotated document and a number of annotation attributes. Clicking on the annotation causes the display to jump to the source document at the position of the annotation. ComMentor uses filtered annotations to produce lists of read documents, but does not support paper-like annotations or present lists of annotations in context. 
     Classroom 2000 is a system for capturing a lecture using recorded audio, prepared visual materials and handwritten notes made on a display overlay of viewgraphs. This system is described in “Classroom 2000: Enhancing Classroom Interaction and Review”, by G. Abowd et al.,  In Proceedings of CSCW &#39; 96, March 1996, incorporated herein by reference in its entirety. Searching the text in the viewgraphs retrieves the viewgraphs along with the overlaid notes. 
     The Freestyle system, which was developed at Wang Laboratories, is a mechanism for sketching and writing on screen snapshots or on sheets of electronic paper. Freestyle records cursor movement and audio as well as the handwriting. This system is described in “Rapid Integrated Design of a Multimedia Communication System, and Human-Computer Interface Design”, E. Francik, Marianne Rudisill et al. (editor), Morgan Kaufman Publishers, Inc.,1996, incorporated herein by reference in its entirety. The result is a dynamic multimedia message that can be mailed to others. Freestyle does not provide the ability to organize the handwritten annotations. 
     The PENPOINT operating system for pen-based computers, recognizes pen gestures for editing and allows arbitrary “ink” marks to be placed on top of any document using an “acetate layer”. This system is described in “The Power of PENPOINT”, by R. Carr et al., Addison-Wesley, Inc., 1991, incorporated herein by reference in its entirety. Although both Freestyle and PENPOINT support free-form document annotation, neither provides any way to retrieve documents based upon those annotations. 
     In 1945 Vannevar Bush described a vision of a mesh of trails running through a mechanized private file and library or memex in “As We May Think”, in  Atlantic Monthly , July 1945, pp. 101-108, incorporated herein by reference in its entirety. These trails were produced as part of the reading activity, and provided a way to create and share personal organizations of information. Bush&#39;s visions were seminal in the development of hypermedia systems such as Engelbart&#39;s NLS and the World Wide Web. However, hypermedia systems have focused on sharing, browsing and more explicit authoring of links, not on personal organization and annotation. 
     Thus, an annotation system for electronic documents is needed that combines the advantages of marking directly on a document with quick accessibility and the flexible organization of marking on note cards or in a notebook. 
     SUMMARY OF THE INVENTION 
     This invention provides a system and method for using digital “ink” for annotations in context to organize a reader&#39;s activities. The system and method of this invention extracts the contents surrounding and underlying a reader&#39;s annotations and presents this information to the reader with links to the full context. The annotations in context provided by the system and method of this invention permits flexible low-overhead organization of material without adding to the effort of reading and notetaking. 
     These and other features and advantages of this invention are described in or are apparent from the following detailed description of the preferred embodiments. 
    
    
     BRIEF DESCRIPTION OF THE DRAWINGS 
     The preferred embodiments of this invention will be described in detail, with reference to the following figures, wherein: 
     FIG. 1 is a block diagram of the document organizing system of this invention; 
     FIG. 2 is a flow chart outlining the control routine of one embodiment of this invention; 
     FIG. 3 shows a document annotated according to this invention; 
     FIG. 4 shows the annotated portions of the document of FIG. 3 
     FIG. 5 shows another view of the annotated document of FIG. 3; and 
     FIG. 6 is a flow chart outlining the annotation control routine of one embodiment of this invention. 
    
    
     DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS 
     FIG. 1 is a block diagram of one embodiment of the electronic document organizing system  10  of this invention. The system  10  has a processor  12  communicating with a display  14 , a first storage device  16 , a second storage device  18  and an input/output interface  20 . The first storage device  16  stores a document  22  displayable on the display  14 . The input/output interface  20  communicates with any number of conventional input/output devices  24  such as a mouse  26 , a keyboard  28  and/or a pen-based device  30 . A user manipulates the input/output devices  24  to annotate the document  22  when displayed on the display  14 . The system  10  then stores these annotations  32  in the second storage device  18 . 
     As shown in FIG. 1, the system  10  is preferably implemented using a programmed general purpose computer. However, the system  10  can also be implemented using a special purpose computer, a programmed microprocessor or microcontroller and any necessary peripheral integrated circuit elements, an ASIC or other integrated circuit, a hardwired electronic or logic circuit such as a discrete element circuit, a programmable logic device such as a PLD, PLA, FPGA or PAL, or the like. In general, any device on which a finite state machine capable of implementing the flowchart shown in FIG. 2 can be used to implement the system  10 . 
     Additionally, as shown in FIG. 1, the memories  16  and  18  are preferably implemented using static or dynamic RAM. However, the memories  16  and  18  can also be implemented using a floppy disk and disk drive, a writable optical disk and disk drive, a hard drive, flash memory or the like. Additionally, it should be appreciated that the memories  16  and  18  can be either distinct portions of a single memory or physically distinct memories. 
     Furthermore, it should be appreciated that the link  17  connecting the memory  16  and the processor  10  can be a wired or wireless link to a network (not shown). The network can be a local area network, a wide area network, an intranet, the internet or any other distributed processing and storage network. In this case, the electronic document  22  is pulled from a physically remote memory  16  through the link  17  for processing in the processor  10  according to the method outlined below. In this case, the electronic document  22  can be stored locally in a portion of the memory  18  or some other memory (not shown) of the system  10 . 
     The method of this invention includes three distinct processes. First, the reader makes annotations on a displayed document, and the annotations are extracted along with their context. Second, the system associates a number of attributes with the annotations in order to facilitate retrieval of the annotations and/or the underlying annotated documents. Third, the reader views collections of the annotations in context, where the collections are organized by those attributes. 
     The system  10  records annotations on electronic documents. A preferred interface for entering the annotations is a pen-based computer, where the reader “writes” directly on the electronic document. On a desktop computer without a pen, clicking a mouse in a margin might create a text overlay box to create the annotation. The system  10  may also support a number of different styles of marking. For example, these styles can include swiping with a highlighter pen, underlining text, vertical bars in the margin, circled regions, and margin notes. 
     FIG. 2 is a flow chart outlining a control routine of one embodiment of the invention. The control routine starts at step S 100  and proceeds to step S 110 , where the user marks on the display of the document with digital ink to annotate it. The control routine then proceeds to step S 120 , where the system groups the marks of the digital ink by time and/or space into collections of marks, treated as a single annotation as will be described in more detail below. Next, the control routine proceeds to step S 130 , where the system determines the minimum context for each annotation. The system has a minimum context that determines how much of the document that surrounds the annotation is to be associated with the annotation. The minimum context may be predetermined as a user preference to be a few words, a sentence, a paragraph or any other amount in accordance with the user&#39;s preferences. The minimum context can be displayed to the user as a bounding box around the minimum context. The bounding box encloses the bounding region and the minimum context is defined as the content enclosed within the bounding region of the corresponding annotation. Segmentation procedures are applied to the document to divide it into graphical components, e.g., lines of text, sentences, paragraphs and figures. Given the minimum context, the control routine expands the context to include all of the nearby segments. With this procedure, the context may include a couple of lines, the surrounding sentence, or the entire surrounding paragraph. FIG. 5 shows a bounding box  34  with the context around a circle annotation  33 . 
     The annotation control routine is shown in FIG.  6 . The control routine starts at step S 200  and proceeds to step S 210  where the user selects and opens an electronic document. The user then starts marking on the document at step S 220  and creates digital ink. The system then determines at step S 230  if the new ink is close enough in time and space to be associated with previous ink marks. The system has time and space thresholds that may be predetermined or adjusted in accordance with a user&#39;s preferences. If the system determines at step S 230  that the ink marks are not separate the system proceeds to step S 240  where the user continues to mark. As each mark is entered by the user steps S 230  and S 240  are repeated until the system determines that the new ink is separated enough by time and space to proceed to step S 250 . At step S 250  the ink marks are grouped together as a single annotation and at step S 260  the context for the annotation is determined and the attributes are assigned to the annotation. The control routine then proceeds to step S 270  where the system determines if a new mark has been input. If a new mark has been input the control routine returns to step S 230 . If no new mark is entered then at step S 280  then the annotations are organized and displayed. The control routine then stops at step S 290 . 
     For some special annotation formats such as those shown in FIG. 5, the control routine determines the context slightly differently. For margin bars  36  and other notes in the margin  38 , the system ignores the horizontal distance when finding nearby segments. Thus, all vertically adjacent material is included in the contexts  40  and  42 , respectively. For the line callouts and circle callouts, the control routine determines the minimum contexts and from the underlined or circled text, etc., ignoring the ink in the callout gesture. 
     After the context of each annotation has been determined, the control routine proceeds to step S 140 , where the control routine assigns attributes to the annotations in at least one of three ways: 1) attributes entered by the user; 2) attributes inherited from the document&#39;s attributes; and 3) implicit or explicit attributes derived from the annotations themselves. 
     The user may enter attributes by interacting with a dialog box or by selecting from a marking menu, or by selecting a special pen. Example, attributes derived from the annotations themselves include “agree”, “disagree”, “good idea”, and “follow-up”. In addition, annotation gestures such as “exclamation point” and “question mark” may be interpreted to mean “good idea” and “questionable” by the system as they are entered on the page. Attributes may also be entered implicitly, the most important of which is the date and time that the annotation was made and the page number at the annotation. Another implicit attribute is the form of the annotation, e.g., highlight, circle, marginal note, etc. 
     Attributes may also be inferred from documents. In the system  10 , the electronic documents are already associated with a variety of attributes, such as creation date, author, provenence and title. 
     After the attributes are assigned to each annotation at step S 140 , the control routine proceeds to step S 150 , where the annotations are organized, ordered or ranked by the assigned attributes. Subsequently, the control routine proceeds to step S 160 , where the annotations are displayed for the user. The control routine then proceeds to step S 170 , where the control routine stops. 
     The system  10  visually presents the annotations in context using different list views. Lists are ordered or filtered by the attributes described above. The system  10  allows the reader to navigate between these views and the underlying electronic documents. Examples of ordered lists include: 
     1) Ordered by time. This view is analogous to a reader&#39;s notebook, but also automatically includes the context of each annotation, as shown in FIG. 4, without further effort by the user. 
     2) Filtered by attributes. Passages across a number of documents are listed in one view; 
     3) Filtered by the type of adjacent material. For example, annotations of pictures along with the pictures themselves; and 
     4) Filtered by the content of adjacent material. For example, annotated passages mentioning patent leather shoes are ranked in relatedness using known information retrieval techniques. 
     It is to be understood that the term annotation as used herein is intended to include text, digital ink, audio, video or any other input associated with a document. It is also to be understood that the term document is intended to include text, video, audio and any other media and any combination of media. Further, it is to be understood that the term text is intended to include text, digital ink, audio, video or any other content of a document to include the document&#39;s structure. 
     While this invention has been described with the specific embodiments outlined above, many alternatives, modifications and variations are apparent to those skilled in the art. Accordingly, the preferred embodiments described above are illustrative and not limiting. Various changes may be made without departing from the spirit and scope of the invention as defined in the following claims.