Abstract:
Widgets are annotated with freeform digital ink. Rather than annotating the data displayed by widgets, annotations are made “on” the widgets themselves and stored in relation to the application and/or window they are associated with. These annotations allow users to interact with widgets without having to keep a separate hardcopy set of notes or instructions explaining how to use the widgets.

Description:
BACKGROUND OF THE INVENTION  
         [0001]    1. Field of Invention  
           [0002]    This invention is related to annotating widgets of a graphical user interface. In particular, this invention is directed to a system and method of annotating widgets to facilitate a user&#39;s learning to use a graphical user interface.  
           [0003]    2. Description of Related Art  
           [0004]    Complex graphical user interfaces that contain windows, such as collections of menus, dialog boxes and toolbars found in everyday software applications, can be hard to learn. Users must often invoke long, precise sequences of commands or dialog box interactions to accomplish a desired task. Although on-line help manuals are commonly available, on-line help manuals often fail to clarify the problem at hand, and usually require several interactions before the desired information is obtained. While users can usually determine the correct sequence of actions through trial and error, infrequently-used command sequences are often forgotten and thus must be subsequently relearned every time they are required. Furthermore, beginners often keep long sequences of notes which correlate to and supplement actions to be performed in a given software application. However, because these notes are separate from, and not particularly well-linked to, the software application, these hardcopy notes are frequently extremely burdensome to use.  
           [0005]    The APPLE GUIDE system is a mechanism in the computer&#39;s help system for showing a user how to accomplish specific tasks with widgets. Specifically, to explain some feature of the interface, highlights and ink marks are used to mark widgets. However, these highlights only exist during the help animation. The freeform digital ink marks cannot be created or manipulated by the user, nor are they available outside the help system.  
         SUMMARY OF THE INVENTION  
         [0006]    Widgets are seen throughout today&#39;s software applications. Widgets, being placeholders for all controls in an interface, can range from windows in a graphical user interface, to simple dropdown menus, to buttons in a toolbar, to highly complex multi-level widgets that control complex functions or enable a user to select multiple options for a given application. Tracking a user&#39;s path through the maze of interfaces to achieve a desired result has become cumbersome.  
           [0007]    This invention provides systems and methods for annotating widgets.  
           [0008]    This invention separately provides systems and methods that allow a user to select an annotation mode, and with freeform digital ink, to record comments or notes directly on widgets.  
           [0009]    This invention separately provides systems and methods that allow audio data to be associated with a widget and to be played by a user to, for example, direct the user how a particular function should be performed.  
           [0010]    This invention separately provides systems and methods that allow a user to attach video data to a widget and to play back the video data to further facilitate performing a particular function.  
           [0011]    This invention separately provides systems and methods that allow a user to annotate widgets with ASCII text.  
           [0012]    This invention separately provides systems and methods for attaching annotations to a widget that can then be shared with other users.  
           [0013]    In traditional freeform digital ink interfaces, annotations occur “inside” the widgets, rather than “on” the widgets. The methods and systems of this invention differ because, in the systems and methods of this invention, an annotation, e.g., freeform digital ink, is not treated as user data, e.g., a signature field, or as an annotation on user data, but rather as a method of associating comment, instructions or other annotations with the user interface. While freeform ink has been used to create content and to annotate documents, it has not been used to annotate user interface widgets. In Interactive Sketching for the Early Stages of User Interface Design, James A. Landay, Ph.D. dissertation, Report # CMU-CS-96-201, Computer Science Department, Carnagie Mellon University, Pittsburgh, Pa., December 26, freeform digital ink is used to add widgets to user interface prototypes, rather than to annotate existing widgets. Specifically, the prototype widgets are the added content. In contrast, the systems and methods of this invention allow for annotating of existing widgets themselves. Furthermore, U.S. Patent Application (Attorney Docket No. 100930), herein incorporated by reference in its entirety, provides methods and systems for maintaining freeform ink annotations on changing views.  
           [0014]    According to methods and apparatus of this invention, a user can directly annotate a widget with freeform graphical ink. Specifically, the user&#39;s annotations are recorded directly on widgets, without affecting their functionality.  
           [0015]    These and other features and advantages of this invention are described in or are apparent from the following detailed description of the preferred embodiments. 
       
    
    
     BRIEF DESCRIPTION OF THE DRAWINGS  
       [0016]    The preferred embodiments of this invention will be described in detail, with reference to the following figures, wherein:  
         [0017]    [0017]FIG. 1 shows an exemplary window comprising a number of widgets;  
         [0018]    [0018]FIG. 2 shows an exemplary set of multi-level windows;  
         [0019]    [0019]FIG. 3 shows a widget annotated according to this invention;  
         [0020]    [0020]FIG. 4 is an exemplary widget demonstrating annotations made according to this invention;  
         [0021]    [0021]FIG. 5 is an exemplary set of annotated widgets corresponding to the unannotated widgets shown in FIG. 2 annotated according to this invention;  
         [0022]    [0022]FIG. 6 shows an exemplary dropdown menu annotated according to this invention;  
         [0023]    [0023]FIG. 7 shows an exemplary dialog box;  
         [0024]    [0024]FIG. 8 shows an exemplary annotated dialog box corresponding to the dialog box shown in FIG. 7;  
         [0025]    [0025]FIG. 9 shows a second exemplary dialog box;  
         [0026]    [0026]FIG. 10 shows a second exemplary annotated dialog box corresponding to the dialog box shown in FIG. 9;  
         [0027]    [0027]FIG. 11 shows one exemplary embodiment of a functional block diagram of a system for annotating widgets according to this invention;  
         [0028]    [0028]FIG. 12 is an exemplary structural block diagram of a window;  
         [0029]    [0029]FIG. 13 is an exemplary structural block diagram of a widget annotated according to this invention;  
         [0030]    [0030]FIG. 14 shows an audio annotation added to a widget according to this invention;  
         [0031]    [0031]FIG. 15 shows a video annotation added to a widget according to this invention;  
         [0032]    [0032]FIG. 16 is a third exemplary annotated widget annotated according to this invention;  
         [0033]    [0033]FIG. 17 is a flowchart outlining one exemplary method for annotating a widget according to this invention; and  
         [0034]    [0034]FIG. 18 is a flowchart outlining another exemplary method for displaying an annotated widget according to this invention. 
     
    
     DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS  
       [0035]    To annotate widgets, the systems and methods of this invention associate and store a user&#39;s annotations corresponding to one or more particular widgets. FIG. 1 is an exemplary window  100  of a graphical user interface. Common elements, i.e. widgets, within a window are dropdown menus  120 , checkboxes  130 , and tabs  140 . Widgets within the window, such as the widgets shown in FIG. 1, are used to specify certain criteria of a desired or selected function. Although the window  100  shown in FIG. 1 is a simple example, windows can become highly complicated and contain hundreds of widgets corresponding to control options. FIG. 2 illustrates an exemplary multi-leveled window containing a first level window  210 , a second level window  220  and a third level window  230 . Clearly, with the quantity of information provided to a user in this context, it would be easy for the user to forget to or be unable to recognize the desired function.  
         [0036]    The systems and methods of this invention provide for annotating these widgets, as shown in FIG. 3. Specifically, the annotated window  300  of FIG. 3 is an annotated version of the window  100  illustrated in FIG. 1. By way of example, the user has indicated in an annotation  310  added to in the window  300  that the vertical ruler is “good for column inches,” and in an annotation  320  that certain scrollbars are not needed for the SCROLLPOINT mouse. Other annotations  330 - 360  have been added to the widget  300  to provide additional information about the widget  300  to the user.  
         [0037]    [0037]FIG. 4 is an exemplary window  400  which contains an annotation  410  that indicates a course of action to be taken before a particular function is performed.  
         [0038]    [0038]FIG. 5 is an exemplary cascaded set of multi-level windows  500 , comprising windows  510 - 530  which have been annotated to direct a user through sequential events. Specifically, when a certain feature of the first level window  510  is selected, the subsequent window  520  is displayed. Again the user is directed via annotations added to the window  520  to a particular location within the window  520 . Upon selecting a function within the annotated window  520 , the user can then be directed to the annotated or highlighted function in the window  530 .  
         [0039]    [0039]FIG. 6 illustrates an annotated dropdown menu  600 . Specifically, the annotation  607  illustrates the circled menu item should be selected for insertion of a picture. Annotation  605  highlights that“caption” adds a caption to an inserted picture. Through the above examples, it is clear that the annotations added according to this invention are not limited to any particular type of window or graphical user interface, but rather are applicable to all widgets.  
         [0040]    As previously mentioned, the methods and systems of this invention allow for annotating widgets themselves, rather than for annotating the data the widgets contain. This is illustrated with reference to FIG. 7- 10 . FIG. 7 shows a print dialog box for a particular printer. The displayed data  655  corresponds to the particular selected printer. FIG. 8 shows annotations  660  and  662  that have been made to the print dialog box of FIG. 7.  
         [0041]    [0041]FIG. 9 shows the same print dialog box for another printer. The displayed data  665  corresponds to this second selected printer. FIG. 10 shows the same annotations  670  and  672  that have been made to the print dialog boxes of FIGS. 8 and 9.  
         [0042]    [0042]FIG. 11 is a functional block diagram illustrating one embodiment of a widget annotation system  700  according to this invention. As shown in FIG. 11, the widget annotation system  700  comprises a user input device  710 , such as a mouse, pen or a keyboard, a mode selector  720 , a widget annotation database  730 , an annotation manager  740 , an annotation renderer  745 , a display  750  that displays a widget  760  to be, or that has been, annotated, and a window manager  755 .  
         [0043]    A user, through the user input device  710 , interacts with the system  700  to annotate the widget  760 . The annotation manager  740  monitors annotations made to the widget  760 . Once an annotation is made, the annotation manager  740  saves the annotation information with the associated unique widget identifier in the widget annotation database  730 . Upon a subsequent display of the same widget  760 , the annotation manager  740  accesses the annotations, if any, from the widget annotation database  730 , and the annotation renderer  745  displays the annotations.  
         [0044]    The widget annotation database  730  stores unique widget identifiers with corresponding annotations. The unique widget identifier as used in this application comprises an application identifier, a widget identifier and an optional user identifier. The application identifier identifies a window manager or an application. The widget identifier comprises a static resource identifier and/or an identifier maintained dynamically by an application or window manager. The optional user identifier uniquely identifies a user of a system.  
         [0045]    The mode selector  720  controls the type of annotation mode selected by the user. The display  750  can be a monitor or other comparable display device that is capable of displaying widgets  760 .  
         [0046]    [0046]FIG. 12 is a structural representation of an exemplary window  800 . The window includes a number of widgets  810  that represent all the visual elements forming the window  800 .  
         [0047]    As shown in FIG. 13, annotations  910  to the window  800  appear as annotations or markings on one or more of the individual widgets  810 . These annotations or markings  910  do not alter the widget itself, but rather are “layered” over the assemblage of widgets  810  that form the window  800 .  
         [0048]    [0048]FIGS. 14 and 15 illustrate additional exemplary types of annotations that can be added to a widget according to this invention. For example, the window  800  shown in FIG. 14 contains an icon  1000  representing an audio segment added or attached to a widget  1010  within the window  800 . As with freeform ink annotations, the icon  1000  is displayed in conjunction with the associated annotated widget. However, in addition to the visual content associated with the icon itself, the icon represents a link, such as a hyperlink or pointer, to an audio file. This audio file may be a preexisting audio file, such as a “wav” file. Alternatively, the audio file could be recorded by a user specifically for the current annotation. For example, when a user decides to annotate a widget, the user could be prompted for the type of annotation desired. If the user selects “audio annotation,” for example, the system could record the user&#39;s audio clip or prompt the user for an already existing audio clip file. During playback, the user simply selects the icon  1000 , for example with a mouse click. In response, an audio clip is played to the user relating to the content of the widget  1010  to which the icon  1000  has been associated. FIG. 15 illustrates a video icon  1100  associated with a widget  11   10 . As with audio annotations, video annotations can be recorded or replayed when the user indicates a video or multimedia annotation is desired, or selects the video icon  1100 , respectively.  
         [0049]    [0049]FIG. 16 is another exemplary annotated window  1200 , illustrating that annotations are not limited to one particular widget, but can be strewn throughout the entire window.  
         [0050]    In operation, a user selects the mode in which annotations are to be performed. For example, one mode can be activated by simply inking over the nonactive portions of the window. Alternatively, if annotations are desired within an active element, a user can temporarily enter a manual mode which would disable the active components of the window and allow annotations to be placed anywhere within the window.  
         [0051]    In either of these modes, the annotations are recorded identically in the widget annotation database. For example, in automatic mode, if the annotation, e.g., inking process, event occurs over an active widget, i.e., a radio button, no annotation, e.g., ink mark, is made. If a selection device controlled by the user, e.g., a mouse pointer, passes over an active widget, e.g., a radio button, that active widget performs normally. Alternatively, if no widget responds to the annotation event, an annotation mode can be triggered.  
         [0052]    A second exemplary embodiment of the methods according to this invention includes an annotation button in the application environment. When the button is selected, the widgets are disabled, the cursor optionally changes to indicate the annotation mode, and user-selected events produce annotations.  
         [0053]    Digital ink can be used to embody the user&#39;s annotations. For example, the freeform digital ink can be used to highlight text or to actually change the characteristics of the text, e.g., underlining. Furthermore, annotations is not limited to actual written annotations by the user. The annotations may encompass audio or video data, animated graphics, and/or may be either transparent or opaque, depending on the user&#39;s desire. Furthermore, it should be understood that the annotations to the widget need not be static. Active elements or hyperlinks can also be included to further broaden the usefulness of the annotations.  
         [0054]    All annotations to the widget are stored in the widget annotation database  730 . This widget annotation database  730  need not be integral to one particular user&#39;s application and one particular user&#39;s annotations. By storing the unique widget identifiers, the widget annotation database  730  can be shared with other users so they can use a first user&#39;s annotations. The receiving user can preview and/or ignore the previous annotations, can replace the existing annotations with the new annotations, and/or add new annotations, or merge the two sets of annotations.  
         [0055]    [0055]FIG. 17 is a flowchart outlining one exemplary method for annotating a widget according to this invention. Control begins in step S 100 , and then continues to step S 200 , where a widget is displayed. With the widget displayed, a user has the option of annotating the widget. Thus, in step S 300 , a user determines whether automatic or manual annotation is desired. If automatic operation is desired, control continues to step S 400 . Otherwise, if manual annotation is desired, control jumps to step S 700 .  
         [0056]    In step S 400 , the particular annotation type desired is selected. This can range, for example, from font attributes, to text attributes, to line thickness and/or color, to insertion of audio or video elements, plain text, highlighting or other type of freeform digital ink. Then, in step S 500 , the actual annotations are created and added by the user relative to the widget. Next, in step S 600 , the added annotations are associated with the widget and saved. Control then continues to step S 700 .  
         [0057]    In step S 700 , the user is queried whether another annotation is desired. If another annotation is desired, control jumps back to step S 300 . If no further annotations are desired, control continues to step S 1   300 , where the control sequence ends.  
         [0058]    In step S 800 , the manual annotation mode is activated. This manual annotation mode allows user complete control over the type and placement of the annotations. Next, in step S 900 , as with step S 400 , the user selects the annotation type. Then, in step S  1000 , the user creates and adds the annotation relative to the widget. Control then continues to step S 1100 .  
         [0059]    In step S 1100 , the annotations are associated with the corresponding widgets and are saved. Then, in step S 1200 , the user determines if additional annotations are desired. If so, control jumps back to step S 900 . Otherwise, control continues to step S 1300  where the control routine ends.  
         [0060]    [0060]FIG. 18 is a flowchart outlining one exemplary embodiment of a method for displaying an annotated widget according to this invention. Control begins in step S 3000 , and then continues to step S 3100 , where a widget is displayed. Then, in step S 3200 , a determination is made as to whether the displayed widget is annotated. If the displayed widget is annotated, control jumps to step S 3210  Otherwise, control continues to step S 3300 . In step S 3300 , the control sequence ends.  
         [0061]    In step S 3210 , the unique widget identifier is determined. Next, in step S 3220 , the corresponding annotation is loaded. Then, in step S 3230 , the annotation is displayed. Control then passes to step S 3300 , where the control sequence ends.  
         [0062]    As shown in FIG. 11, the widget annotation system is preferably implemented on one or more programmed general purpose computers. However, the widget annotation system can also be implemented on a special purpose computer, a programmed microprocessor or microcontroller and peripheral integrated circuit elements, an ASIC or other integrated circuit, a digital single processor, a hardwired electronic or logic circuit such as a discrete circuit, a programmable logic device such as a PLD, PLA, PGA, FPGA or PAL, or the like. In general, any device, capable of implementing a finite state machine that is in turn capable of implementing the flowcharts shown in FIGS.  17 - 19 , can be used to implement the widget annotation system according to this invention.  
         [0063]    It is, therefore, apparent that there have been provided in accordance with the present invention, methods and systems for annotating widgets. While this invention has been described in conjunction with preferred embodiments thereof, it is evident that many alternatives, modifications, and variations be apparent to those skilled in the art. Accordingly, applicants intend to embrace all such alternatives, modifications and variations that fall within the spirit and scope of this invention.