Abstract:
Systems and methods that incorporate various techniques for teaching gestures to a user of a multi-touch sensitive device are disclosed herein. Such techniques can include presenting visible feedback of gestures, such as animated motion trails and/or hand motions, along with affirmative feedback for correctly performed gestures and negative feedback for incorrectly performed gestures. Such techniques can be expanded to provide training exercises that present tasks requiring a particular gesture or sequence of gestures to be performed. These training exercises can take the form of games or other engaging activities to encourage use.

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS 
     This claims priority to U.S. patent application Ser. No. 11/619,571, titled “Multi-Touch Gesture Dictionary,” filed Jan. 3, 2007, and U.S. patent application Ser. No. 11/619,553, titled “Multi-Touch Gesture Dictionary,” filed Jan. 3, 2007, both of which are hereby incorporated by reference in their entirety. 
     This is related to the following U.S. patents and patent applications, each of which is also hereby incorporated by reference in its entirety:
         U.S. Pat. No. 6,323,846, titled “Method and Apparatus for Integrating Manual Input,” issued Nov. 27, 2001;   U.S. patent application Ser. No. 10/840,862, titled “Multipoint Touchscreen,” filed May 6, 2004;   U.S. patent application Ser. No. 10/903,964, titled “Gestures for Touch Sensitive Input Devices,” filed Jul. 30, 2004;   U.S. patent application Ser. No. 10/038,590, titled “Mode-Based Graphical User Interfaces for Touch Sensitive Input Devices,” filed Jan. 18, 2005;   U.S. patent application Ser. No. 11/367,749, titled “Multi-Functional Hand-Held Device,” filed Mar. 3, 2006;   U.S. patent application Ser. No. 11/619,505, titled “Far-Field Input Identification,” filed Jan. 3, 2007;   U.S. patent application Ser. No. 11/649,998, titled “Proximity and Multi-Touch Sensor Detection and Demodulation,” filed Jan. 3, 2007;   U.S. Pat. No. 6,337,678, titled “Force Feedback Computer Input and Output Device With Coordinated Haptic Elements,” issued Jan. 8, 2002.       

     BACKGROUND 
     Many attempts have been made over the years to improve the way users interact with computers. In the beginning, cards or tapes with punched holes were used for user input. Punch cards gave way to terminals with alphanumeric keyboards and text displays, which evolved into the modern keyboard, mouse, and graphical-display based graphical user interfaces. Many expect that the use of multi-finger, touch-sensitive user interfaces (“multi-touch interfaces”, such as those described in the references incorporated above, will become widely adopted for interacting with computers and other electronic devices, allowing computer input to become even more straightforward and intuitive. 
     Users of these multi-touch interfaces may make use of hand and finger gestures to interact with their computers in ways that a conventional mouse and keyboard cannot easily achieve. A multi-touch gesture can be as simple as using one or two fingers to trace out a particular trajectory or pattern, or as intricate as using all the fingers of both hands in a complex sequence of movements reminiscent of American Sign Language. Each motion of hands and fingers, whether complex or not, conveys a specific meaning or action that is acted upon by the computer or electronic device at the behest of the user. The number of multi-touch gestures can be quite large because of the wide range of possible motions by fingers and hands. It is conceivable that an entirely new gesture language might evolve that would allow users to convey complex meaning and commands to computers and electronic devices by moving their hands and fingers in particular patterns. 
     Techniques for teaching these gestures and gesture languages to new users are needed. Techniques that have been proposed include playback of motion trails indicating the gesture, animated hands performing the gesture, and various graphical depictions of gestures and their meanings. Each of these techniques suffers from one or more deficiencies. 
     For example, a motion trail  100  corresponding to a thumb and two-finger rotate gesture is illustrated in  FIG. 1 . The paths  101 ,  102 , and  103  can correspond to the thumb, index, and middle finger motions, with the arrowheads indicating the direction of motion. The paths may be color-coded or have other indication of what finger(s) are used. For example, hand icon  104  can be provided and can include dots to indicate what fingers are used. Even if the motion trails are animated, the abstraction level of this teaching technique can still lead to difficulty in comprehension for a user. Additionally, this technique lacks interactivity and feedback. Therefore, this technique does not lend itself to teaching and practice of a particular gesture or group of gestures by a user. 
     An animated hand gesture (again illustrating a thumb and two finger clockwise rotation gesture) is illustrated in  FIG. 2 . Animated hand gestures may range in complexity from relatively simple line drawings to complex three-dimensional renderings of the hand. These animated hand gestures can address the abstraction problem to a degree by providing a more tangible representation of the users hand  200 . However, this technique may not clearly indicate the path taken by the various hand parts. For example, paths may be obscured by the hand representation. Additionally, this technique also lacks interactivity and feedback that may be beneficial for teaching and practice. 
     The gesture dictionaries disclosed in U.S. patent application Ser. Nos. 11/619,571 and 11/619,553, each titled “Multi-Touch Gesture Dictionary,” and each filed Jan. 3, 2007 (referenced above) can be adapted as described therein to provide a degree of interactivity. For example, a graphical depiction of a gesture performed may be highlighted or otherwise indicated. Additionally, a simple form of feedback arises from the fact that a gesture that is incorrectly executed will cause the wrong command to be displayed. However, this relatively simple level of interactivity and feedback may be better suited for teaching additional gesture “vocabulary” to users that are already somewhat acquainted with some gestures. Gesture learning from ground zero may be enhanced by techniques that incorporate more robust interactivity and feedback. 
     SUMMARY 
     The present invention can relate, for example, to a method for teaching gestures. The method can include presenting a display having two display areas, one of which can be a multi-touch monitor window. The multi-touch monitor window can be used to display interactive feedback to a user indicating what gesture the user is performing. The multi-touch monitor window can be overlaid on the first display area or can be a separate window, such as a side-by-side arrangement. If the multi-touch monitor window is overlaid, it can incorporate transparency or translucency that allows the display area behind it to also be perceived. 
     The interactive feedback can take a variety of forms. For example, the interactive feedback can comprise an animated hand together with one or more motion indicators. The animated hand can be a line drawings, a three-dimensional rendering, a translucent shadow of a hand, or other representation. The motion indicators can include motion trails or other representations, which can also be color-coded. The interactive feedback may also be superimposed with an animated display of a correctly performed gesture so that the user can see the difference, if any, between the gesture he performs and the idealized gesture. 
     The first display area can be used in conjunction with a variety of application programs. For example, the first display area can be used with a utility, entertainment, or communication application, in which case the feedback provided by the multi-touch monitor window serves as reinforcing feedback. As another example, the first display area can be used in conjunction with a gesture learning application. 
     The gesture learning application can take a variety of forms. In some embodiments, the gesture learning application can be a game. In other embodiments, it can be an application that presents gestures to be performed to the user in the form of various repetitive drills. The gesture learning application can detect the gesture or gestures performed by the user in response to the presented gesture or gestures and provide feedback indicating whether the gesture or gestures were performed correctly. 
     The present invention can also relate, for example, to a graphical user interface for a gesture learning application. The graphical user interface can include a main window including an indication of a gesture or sequence of gestures to be performed using a multi-touch interface and a multi-touch monitor window including an interactive feedback mechanism that indicates a gesture or gestures actually performed by the user. To facilitate learning, a sequence of gestures may be presented. These gestures can be arranged according to particular chords, motions, or sequences thereof. 
     The indication of a gesture or gestures to be performed can take a variety of forms, including an animated hand, one or more motion trails, an iconographic representation, a textual description of the gesture to be performed, a textual description of a command corresponding to the gesture to be performed, and combinations thereof. Similarly, the indication of a gesture or gestures actually performed can also take a variety of forms, including an animated hand, one or more motion trails, an iconographic representation, a textual description of the gesture currently being performed, a textual description of a command corresponding to the gesture currently being performed, a positive feedback indicator, a negative feedback indicator, and combinations thereof. The positive and negative feedback indicators could also be in an audible form. 
     In other embodiments of the invention, computer systems including one or more applications or graphical user interfaces as described above are provided. The computer systems can take the form of a desktop computer, notebook computer, tablet computer, handheld computer, personal digital assistant, media player, mobile telephone, or the like. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       The aforementioned and other aspects of the invention may best be understood by reference to the following description taken in conjunction with the accompanying drawings in which: 
         FIG. 1  illustrates a prior art display of gesture motion trails. 
         FIG. 2  illustrates a prior art animated display of a hand performing a gesture. 
         FIG. 3  illustrates a screen display of an interactive gesture learning application according to some embodiments of the present invention. 
         FIG. 4  illustrates an interactive feedback mechanism of an interactive gesture learning application according to some embodiments of the present invention. 
         FIGS. 5A and 5B  illustrate screen displays of an interactive gesture learning application according to some embodiments of the present invention. 
         FIG. 6  illustrates a flow chart for an interactive gesture learning application according to some embodiments of the present invention. 
         FIGS. 7A and 7B  illustrate screen displays of an interactive gesture learning application that permits practice of learned gestures according to some embodiments of the present invention. 
         FIG. 8  illustrates a screen display of an interactive gesture learning application in the form of a game according to some embodiments of the present invention. 
         FIG. 9  illustrates a screen display of an interactive gesture learning application in the form of a game according to some embodiments of the present invention. 
     
    
    
     DETAILED DESCRIPTION 
     To take full advantage of a multi-touch gesture language, users will need to learn and/or remember the meaning of numerous gestures. Multi-touch gestures may be considered to include at least two phases that, taken together in sequence, signal the beginning and completion of a particular gesture. The first phase of a multi-touch gesture can include presenting a specific combination of hand parts, i.e., fingers, thumbs, etc. in a particular configuration. In some embodiments, this may include placing the hand parts down on the multi-touch surface. The second phase of the gesture can include, for example, motion of the specific hand parts. This motion may take the form of lateral motions such as rotation, translation, scaling (expansion and contraction), etc. Again, in some embodiments, this may comprise moving the hand parts around on the multi-touch surface. In such embodiments, the second phase of the gesture may also comprise vertical motions (relative to the multi-touch surface) such as tapping, double-tapping, etc. 
     For convenience, the first phase, e.g., the starting position, number, and configuration of all the hand parts used for a particular gesture, will be referred to herein as a chord. Also for convenience, the hand parts will be referred to as fingers, although this also includes thumbs, palm heels, etc. In some embodiments, other touch devices, such as a stylus, can also be used either alone or in conjunction with hand parts. Modifier buttons and keys (such as Ctrl, Alt, Shift, Option, Command, etc.) keys may also be considered as part of the chord. Therefore, in the examples described herein, a chord can include: a set of fingers or hand parts from either or both hands, touch devices, and/or modifier keys, that are put in motion to form a gesture. In many multi-touch systems the chord may uniquely specify a set of gestures that belong to the combination of fingers and orientations making up the chord. 
     Each of a user&#39;s hands acting alone can execute twenty-five or more chords. For example, five fingers that can be independently raised or lowered give rise to thirty-one combinations. Additional chords may be distinguished by whether only the fingertips are in contact with the surface or whether the length of the finger is flattened against the surface. Further chords may be distinguished based on whether the fingertips are placed on the surface close together or spread apart. As noted above, modifier keys (e.g., the Ctrl, Alt, Shift, and Cmd keys of a keyboard) may be used to distinguish different chords. Modifier keys may also include buttons, touch-sensitive or force-sensitive areas, or other toggles located on the device. However, some chords may be more difficult to execute than others, and various identification and classification problems can arise for the device, particularly in the case of closed versus spread fingertips. 
     Many chords can have at least thirteen different motions associated with them. For example, a two-finger chord (for example, the index and middle fingers) could have specific meaning or action assigned to the lateral motions that include rotation, translation, and scaling. Rotation (clockwise and counter-clockwise) of the two-finger chord gives rise to two unique meanings or actions. Translation (left, right, up, down, and four diagonals) gives rise to at least eight unique meanings or actions. Scaling (contraction or expansion) also gives rise to two meanings or actions. The vertical motion of a chord may comprise lifting the fingers of the chord off the multi-touch surface almost immediately after they had touched down, (e.g., tapping the multi-touch surface with the chord) or multiple taps, etc. 
     With each hand able to execute twenty-five or more chords, and with each chord having thirteen or more motions associated therewith, there may be over three hundred possible gestures for each hand. Many more gestures are possible if both hands are used together. This gives rise to the gesture language referenced above. Learning a multi-touch gesture language may be facilitated by an interactive application that provides some type of demonstration of the expected hand and finger motion as well as feedback indicating whether the gesture was performed correctly. 
     One such interactive application will now be described with respect to  FIG. 3 . A display  300  of a device incorporating a multi-touch input device is displayed. The display may be a touch-screen device that incorporates multi-touch sensing or may be an ordinary display. A computer application may display information is first display area  301 . A second display area  302  can be a multi-touch monitor window. Multi-touch monitor window  302  may be overlaid atop the application window and may optionally incorporate transparency or translucency to permit the “obscured” portion of first display area  301  to be perceived by the user. Alternatively, multi-touch monitor window  302  can be separately displayed, for example using a split screen, etc. Multi-touch monitor window  302  can incorporate an interactive feedback mechanism that provides indication to the user of a gesture being performed. 
     As shown in  FIG. 3 , the interactive feedback mechanism can include an animated hand  303  with motion indications  304   a ,  304   b , and  304   c  that correspond to the motions of hand parts involved in performing the gesture. The animated hand may take various forms, ranging from a simple line drawing to a three-dimensional rendering. One such form, illustrated in  FIG. 4 , may be a translucent “shadow”  401  of a hand. This translucency allows motion trails  402   a ,  402   b , and  402   c  (a form of motion indication) associated with the hand parts to be perceived through shadow  401 . The motion trails may also be color-coded to distinguish one from another. 
     By “interactive feedback mechanism,” it is meant that the hand representation  303  or  404  displays to a user the gesture that is currently being performed. In some embodiments, the parameters necessary to provide the animation may be inferred from the contact points  403  of the gesture. Alternatively, the whole hand may be tracked through some form of proximity sensing. The proximity sensing can, for example, take the form of far-field sensing as described in U.S. patent application Ser. No. 11/619,505, titled “Far-Field Input Identification,” filed Jan. 3, 2007 (referenced above) or infrared sensing as described in U.S. patent application Ser. No. 11/649,998, titled “Proximity and Multi-Touch Sensor Detection and Demodulation,” filed Jan. 3, 2007. In still other embodiments, a camera can be used for hand tracking, as described in U.S. Pat. No. 6,337,678, titled “Force Feedback Computer Input and Output Device With Coordinated Haptic Elements,” issued Jan. 8, 2002 (referenced above). 
     Multi-touch monitor window  302  can be used in connection with various applications being used. For example, display area  301  can display any of a variety of applications, such as utilities, entertainment, communication or other applications, with display area  302  indicating input gestures performed by the user. The feedback provided to the user indicating the gesture performed coupled with the action or command invoked in the application can provide assistance in learning to control application programs using gestures. In some embodiments, the feedback information displayed in the multi-touch monitor window can be superimposed with the animated display of the gesture so that a user can easily perceive the difference between his hand activities and those of the ideal gesture. Additionally, a multi-touch monitor window can be used in conjunction with an application specifically designed to facilitate gesture learning as described in greater detail below. 
     An exemplary gesture learning application display is illustrated in  FIGS. 5A and 5B . A main window  501  indicates a gesture to be performed. The gesture to be performed may be indicated in a variety of ways, including, for example, animated hand  502 , motion trails  503 , an iconographic representation  504 , a textual description (which can be either a description of a gesture  505   a  or of a command performed by the gesture  505   b ), as well as various combinations of one or more of these or other indications. Multi-touch monitor window  506  displays feedback to the user indicating what the device perceives from a gesture performed by a user. This feedback may include an animated hand  502 , motion trails  508 , or other suitable representations as described above. 
     The feedback may also include a negative feedback indicator  509  if the gesture is performed incorrectly or a positive feedback indicator  510  if the gesture is performed correctly. For example, in the illustration of  FIG. 5A , the rotate gesture performed by the user was interpreted as a downward stroke and an “X” is displayed to indicate that the gesture was not correctly performed. In the illustration of  FIG. 5B , the rotate gesture performed by the user was interpreted correctly and a check-mark is displayed to indicate that the gesture was correctly performed. The feedback indicators may also include audible feedback either in conjunction with or as a substitute for the visible feedback indicators. 
     A basic flow chart for a gesture learning application corresponding to the displays of  FIGS. 5A and 5B  is illustrated in  FIG. 6 . In block  601 , a gesture is displayed to the user, e.g., in main window  501 . In block  602 , the gesture performed by the user is monitored and the feedback displayed in monitor window  506 . The gesture performed by the user is then evaluated in block  603  to determine whether it was performed correctly. If not, a negative feedback indicator is displayed (block  604 ), and the gesture can be demonstrated again (block  601 ). If the gesture was performed correctly, a positive feedback indication may be provided and/or a next gesture may be displayed (block  605 ), with the process repeating. 
     A gesture learning application may also be configured to permit a user to practice gestures that have been learned. This can add an element of repetition to help reinforce the learning of gestures. An exemplary display for such an application is illustrated in  FIGS. 7A and 7B . The display can include a main window  701  and multi-touch monitor window  502 . Alternatively, only the main window may be used as other feedback to the user can be provided as will be described below. Main window  701  can be used to present a sequence of gestures  703  to be performed by the user. Arrangement of the sequence of gestures can be organized into lessons corresponding to particular chords or sequences of chords, particular motions or sequences of motions, etc. The sequence of gestures can also include a timing component, either as from the beginning to end of a gesture or the spacing between gestures. 
     As in the examples above, gestures may be presented in a variety of ways, including animations, motion trails, verbal descriptions of the gesture and/or the command invoked, iconographic representations, etc. The example of  FIGS. 7A and 7B  uses iconographic representations. The iconographic representations include a schematic hand, a dot indicating the figures used, and a motion arrow indicating the motion to be performed. In the example illustrated, iconographic representation  704  corresponds to a thumb and two finger clockwise rotation gesture. Iconographic representation  705  corresponds to a thumb and forefinger clockwise rotation gesture. Iconographic representation  706  corresponds to a thumb and forefinger pinch gesture. Iconographic representation  707  corresponds to a thumb and three finger expansion gesture. Although the illustrated iconographic representations correspond to a left hand, either right-handed or two-handed gestures could also be used. 
     As illustrated in  FIG. 7B , a user can then perform each of the gestures in the sequence presented. As the gestures are performed, feedback can be provided in the optional multi-touch monitor window  702  as described above. Additionally or alternatively, an iconographic representation corresponding to the gesture performed can be displayed. The iconographic representation can also include positive feedback indication for correctly performed gestures and negative feedback indication for incorrectly performed gestures. 
     Correct performance of the first two gestures can be indicated by display of an iconographic representation of the gestures performed  708  and  710  that correspond to the iconographic representations  704  and  705  originally presented. Check marks  709  and  711  can serve as a positive feedback indicators. As above, positive feedback could also take other forms, including audible feedback. 
     Incorrect performance of the third gesture can be indicated by displaying iconographic representation  712 , which does not correspond to the originally presented iconographic representation  706 . In the illustrated example, instead of a two-finger pinching gesture, a three-finger pinching gesture was performed. An additional negative feedback indication, e.g., “X”  713  can also be displayed. As with the positive feedback indicator, audible feedback can also be used, either in addition to or as a substitute for visual feedback. In cases where a multi-touch monitor window is not displayed, incorrect performance of a gesture can invoke the multi-touch monitor window, which can playback the performed gesture to demonstrate the mistake made by the user. 
     A gesture learning practice application may also take the form of a game to provide a more satisfying user experience. In one embodiment, the game may take a form similar to the classic SPACE INVADERS® or MISSILE COMMAND® video games. Representations of gestures  801 , which can be in any of a variety of written or graphical forms, may descend from the top  802  of screen  803 . The representations  801  can be destroyed ( 804 ) by correctly executing the represented gesture. Destroying the representation of the gesture can add points (e.g.,  200 ) to the user&#39;s score  805 . If the representation of the gesture reaches bottom  806  of screen  803  without being destroyed, points may be deducted from the user&#39;s score or other penalty. 
     Another embodiment of a gesture learning game can be similar to the TETRIS® video game. As shown in  FIG. 9 , the block shapes  901 - 906  can correspond to particular chords  907 - 912 . Rotation of the pieces can be performed by clockwise and counter-clockwise rotation of the corresponding chord. Dropping of the pieces can be performed by downward strokes. Alternatively, the rotations and dropping can be performed by other motions. 
     In another embodiment, gesture learning can be incorporated into graphical role-playing and strategy games such as FINAL FANTASY® or CIVILIZATIONS® where each character, vehicle or group of characters is assigned a particular chord, and various gesture motions performed with that particular chord direct the movements, spells, and/or attacks of a character, vehicle, or group of characters. Failure to perform the correct chord results in punishment in the form of unwanted movements, spells or actions by unintended characters, vehicles, or groups of characters. Since the instantly performed chord selects the character, vehicle, or group of characters, with practice the player will be able to switch between characters, vehicles, or groups of characters much more quickly than the traditional method of moving the mouse cursor over or directly touching the desired character, vehicle, or group of characters. 
     Having described various formats for gesture learning techniques and applications, the following describes how a user may access and interact with such applications. In some embodiments, application programs embodying these techniques may be provided on a computer system the multi-touch gestures interact with. The program may be stored in a memory of a computer system, including solid state memory (RAM, ROM, etc.), hard drive memory, or other suitable memory. A CPU may retrieve and execute the program. The CPU may also receive input through a multi-touch interface or other input devices. In some embodiments, an I/O processor may perform some level of processing on the inputs before they are passed to the CPU. The CPU may also convey information to the user through a display. Again, in some embodiments, an I/O processor may perform some or all of the graphics manipulations to offload computation from the CPU. Also, in some embodiments, a multi-touch interface and display may be integrated into a single device, e.g., a touch screen. 
     The computer system may be any of a variety of types, including desktop computers, notebook computers, tablet computers, handheld computers, personal digital assistants, media players, mobile telephones, and the like. Additionally, the computer may be a combination of these types, for example, a device that is a combination of a personal digital assistant, media player, and mobile telephone. The gesture learning application may be started by a user using any of a variety of techniques common in GUI-based computer systems. 
     Many other variations and/or combinations of the embodiments discussed herein are also possible. For example, although the descriptions herein have centered around motions of fingers and hands performed on a surface, the principles herein may be also applied to three-dimensional spatial gestures. It is therefore intended that the following appended claims be interpreted as including all such alterations, permutations, combinations and equivalents.