Abstract:
A touch system comprises a touch panel having a touch surface and a projector presenting images onto the touch surface. A computer executing an applications program is coupled to the touch panel and the projector. The computer is responsive to contact output generated by the touch panel in response to proximity of a pointer to the touch surface and updates image data conveyed to the projector so that images presented on the touch surface reflect pointer activity. The computer executes a gesture suggestion and writing recognition routine. The gesture suggestion and writing recognition routine performs recognition to convert ink on the touch surface into another object based on a selected interpretation.

Description:
FIELD OF THE INVENTION  
         [0001]    The present invention relates generally to touch systems and in particular to a system and method for providing gesture suggestions to enhance interpretation of user input.  
         BACKGROUND OF THE INVENTION  
         [0002]    Touch systems are well known in the art and many variations exist. In all cases, touch systems include a touch panel having a touch surface on which contacts are made using a pointer. Pointer contacts with the touch surface are detected and are used to generate corresponding output that represent the positions on the touch surface where contacts are made. The contact position output is typically fed to a computer that executes one or more applications programs. The computer generates image data that is used to present images on the touch surface. The computer uses the contact position output to update the image data and thus, the images presented on the touch surface. In this manner, the images presented on the touch surface are updated to reflect the activity of the pointer on the touch surface.  
           [0003]    For example, U.S. Pat. No. 5,448,263 to Martin, assigned to the assignee of the present invention, discloses a passive analog resistive touch panel coupled to a computer. The computer provides image data to a projector that projects images onto the touch surface of the touch panel. The touch panel includes a tool tray that supports a plurality of differently coloured pens. When a user contacts the touch surface either with a fmger, other pointer or a pen, the touch panel outputs signals representing the contact position on the touch surface. The contact position data is conveyed to the computer and is mapped to the computer display. If a finger or other pointer is used to contact the touch surface, the touch system operates in a pointer mode and the contact position data is treated as a mouse event. This allows the user to operate the computer in a manner similar to using a computer mouse i.e. select menus, manipulate objects etc. simply by contacting the touch surface. If a pen is lifted from the tool tray and is used to contact the touch surface, the touch system operates in an ink mode and the contact position data is recorded as writing or drawing.  
           [0004]    When the computer is running an applications program in a Windows environment, a computer desktop image is presented on the touch surface that includes icons representing the various applications programs available for selection. When an icon is selected, a window for the selected applications program is opened. The window typically includes a frame, one or more tool bars, optional scroll bars and an active area surrounded by the frame, tool bars and scroll bars. As mentioned above, in the pointer mode, contacts on the touch surface are treated as mouse event input to the computer desktop. The computer in response to the mouse event input controls the computer desktop or selected applications program according to the touch panel output and updates the image data conveyed to the projector for display to reflect the pointer activity.  
           [0005]    In the ink mode, an acetate image identical to the computer desktop image overlies the computer desktop image to provide a surface on which ink can be drawn. When a pen contacts the touch surface, the contact position data is treated as writing or drawing (herein referred to as “writing”). In this case, the computer updates the image data conveyed to the projector for display so that the writing is displayed on the acetate image.  
           [0006]    In order to interpret writing when the touch system is operating in the ink mode, the computer can be conditioned to execute a handwriting recognition routine. As a result, writing on the touch surface can be translated into text. In many circumstances, however, the writing on the touch surface is not clear making translation difficult. To enhance text recognition, several prior systems have been considered.  
           [0007]    For example, U.S. Pat. No. 5,963,666 to Fujisaki et al. discloses a confusion matrix for aiding in the recognition of handwritten text. Handwritten characters input into a data field of an applications program are recognized and assigned a cluster code. The string of characters is characterized by the cluster codes of the individual characters. The cluster codes of the string are compared with strings of cluster codes representing words in a dictionary. All or some of the matching words are displayed to the user, from which a selection can be made.  
           [0008]    U.S. Pat. No. 5,367,453 to Capps et al. discloses a word correction system and method. Ink objects entered into a computer system with a pointer are recognized as word objects and are displayed. If a correction request is entered by a user pertaining to a displayed word object, a list of alternative words for the word object are displayed. The user can select an alternative word from the list to replace the displayed word object.  
           [0009]    U.S. Pat. No. 5,329,609 to Sanada et al. discloses a word recognition system including a dictionary order sorter that re-sorts character strings of recognition candidates stored in a high-ranking candidate memory in the order of distance into the dictionary. When a sort termination signal is received, character strings of recognition candidates stored in the memory are displayed together with their ranking numbers.  
           [0010]    Although these prior systems assist in recognizing handwriting, these systems are limited in that they simply provide the results of a fixed interpretation. As will be appreciated alternative systems to enhance the manner by which writing is interpreted are desired.  
           [0011]    It is therefore an object of the present invention to provide a novel system and method for providing gesture suggestions to enhance interpretation of user input.  
         SUMMARY OF THE INVENTION  
         [0012]    According to one aspect of the present invention there is provided a method of recognizing ink displayed on a touch panel comprising the steps of:  
           [0013]    selecting an interpretation to be used to interpret said ink from a plurality of available interpretations; and  
           [0014]    recognizing the ink using the selected interpretation.  
           [0015]    In a preferred embodiment, the selecting is performed in response to user input. The plurality of available interpretations is displayed as gesture suggestions on the touch panel and the desired interpretation is selected by contacting the touch panel at the location where the desired interpretation is presented.  
           [0016]    It is also preferred that the available interpretations include a text interpretation and a freehand interpretation. When the text interpretation is selected, the ink is converted into text during the recognizing. When the freehand interpretation is selected, the ink is converted into a graphic object during the recognizing.  
           [0017]    According to another aspect of the present invention there is provided a system for recognizing ink displayed on a touch panel comprising:  
           [0018]    means for selecting an interpretation to be used to interpret said ink from a plurality of available interpretations; and  
           [0019]    means for recognizing the ink using the selected interpretation.  
           [0020]    According to yet another aspect of the present invention there is provided a touch system comprising:  
           [0021]    a touch panel having a touch surface;  
           [0022]    a projector presenting images onto said touch surface; and  
           [0023]    a computer executing an applications program and being coupled to said touch panel and said projector, said computer being responsive to contact output generated by said touch panel in response to proximity of a pointer to said touch surface and updating image data conveyed to said projector so that images presented on said touch surface reflect pointer activity, said computer executing a gesture suggestion and writing recognition routine, said gesture suggestion and writing recognition routine performing recognition to convert ink on said touch surface into another object based on an interpretation selected from a plurality of available interpretations.  
           [0024]    According to still yet another aspect of the present invention there is provided a computer program product including a computer readable medium having a computer program for recognizing ink displayed on a touch panel embodied thereon, said computer program including:  
           [0025]    computer program code for selecting an interpretation to be used to interpret said ink from a plurality of available interpretations; and  
           [0026]    computer program code for recognizing the ink using the selected interpretation.  
           [0027]    The present invention provides advantages in that ink on the touch panel is recognized based on an interpretation selected from a plurality of available interpretations. As a result incorrect interpretations of ink are reduced especially in cases where the ink can be interpreted in a number of different, yet correct ways.  
       
    
    
     BRIEF DESCRIPTION OF THE DRAWINGS  
       [0028]    An embodiment of the present invention will now be described more fully with reference to the accompanying drawings in which:  
         [0029]    [0029]FIG. 1 is a schematic diagram of an interactive touch system including a touch panel having a touch surface;  
         [0030]    [0030]FIG. 2 is a view of a computer desktop image together with an acetate desktop image on the touch surface of the touch panel.  
         [0031]    [0031]FIGS. 3 a  to  3   d  are views of ink presented on the touch surface of the touch panel and possible interpretations of the ink;  
         [0032]    [0032]FIG. 4 is a flowchart showing the steps performed by a gesture suggestion and writing recognition routine;  
         [0033]    [0033]FIG. 5 is a view of ink presented on the touch surface of the touch panel together with a plurality of selectable gesture suggestions representing available interpretations of the ink; and  
         [0034]    [0034]FIG. 6 is another view of ink presented on the touch surface of the touch panel together with a plurality of selectable gesture suggestions representing available interpretations of the ink. 
     
    
     DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT  
       [0035]    Turning now to FIG. 1, an interactive touch system similar to that disclosed in U.S. Pat. No. 5,448,263 to Martin is shown and is generally identified by reference numeral  10 . As can be seen, touch system  10  includes a touch panel  12  having a touch surface  14 . The touch panel  12  in this embodiment is of the analog resistive type and includes a tool tray  15  that supports a plurality of differently coloured pens (not shown). Touch panel  12  is responsive to contacts on the touch surface  14  made using either a finger, other pointer or pen and generates output representative of the locations on the touch surface  14  where contacts are made. The contact position output of the touch panel  12  is fed to a computer  16  executing one or more applications programs and is treated either as mouse events or writing depending on the object used to contact the touch surface  14 . Specifically, the contact position output of the touch panel  12  is treated as mouse events if a finger or other pointer is used to contact the touch surface  14 . The contact position is however treated as writing when a pen is lifted from the tool tray  15  and is used to contact the touch surface  14 . In this manner the touch system  10  either operates in a pointer mode or ink mode depending on the object used to contact the touch surface  14 . The computer  16  is also coupled to a front or rear projector  18  and provides image data to the projector. The projector  18  in turn presents images on the touch surface  14  of the touch panel. The touch panel  12 , computer  16  and projector  18  form a closed-loop so that user contacts with the touch panel  12  can be recorded as writing or used to control execution of an applications program executed by the computer  16 .  
         [0036]    In the present embodiment, the computer  16  runs in a Windows environment and provides image data to the projector  18  so that a computer desktop image is presented on the touch surface  14 . The computer desktop image presents one or more icons that can be selected to open associated applications programs. When an applications program is selected, a window for the applications program is opened.  
         [0037]    [0037]FIG. 2 shows an applications program window  30  including an active area  32  bordered by a frame and one or more tool bars  34  presented on the touch surface  14 . When a pen is used to contact the touch surface  14  and the touch system  10  is conditioned to the ink mode, an acetate image  40  identical to the computer desktop image is displayed over the computer desktop image. The acetate image  40  provides a surface for writing (commonly referred to as “ink”) so that the ink is not used by the computer  16  to update the underlying computer desktop image  30 . Contact position data returned to the computer  16  by the touch panel  12  in response to pen contacts with the touch surface  14  are used by the computer to update the image data conveyed to the projector  18  so that the ink appears on the acetate image  40 .  
         [0038]    When the touch system  10  is conditioned to the ink mode, the computer can be conditioned to execute a gesture suggestion and writing recognition routine to interpret ink displayed on the acetate image, in a user suggested manner. As a result, incorrect interpretations of ink input by a user are reduced. For ease of illustration, a situation where ink can be interpreted a number of possibly correct but different ways will be described with reference to FIGS. 3 a  to  3   d.    
         [0039]    As can be seen in FIG. 3 a,  handwritten ink  42  resembling a circle, is shown displayed on the acetate image presented on the touch surface  14 . The shape of the ink allows the ink to be interpreted in a number of different ways. Specifically, the ink  42  may be recognized as the number “zero” or the letter “o”, as shown in FIG. 3 b.  The ink may also be recognized as a circle  46 , as shown in FIG. 3 c,  or as a freehand object  48 , as shown in FIG. 3 d.  If a fixed interpretation is used to recognize the ink, an incorrect interpretation of the ink may result.  
         [0040]    To reduce incorrect interpretations of ink, when a user enters ink  50  and the computer has been conditioned to execute the gesture suggestion and writing recognition routine by selecting an icon displayed on the touch surface, the gesture suggestion and writing recognition routine displays a plurality of gesture suggestions  52  that represent available interpretations of the ink as shown in FIG. 4. Once a gesture suggestion has been selected, the gesture suggestion and writing recognition routine uses the selected gesture suggestion and ink  54  and performs recognition of the ink based on the selected gesture suggestion  56 . The recognized ink  58  is then presented on the touch surface  14  in a form suitable for the selected gesture suggestion.  
         [0041]    [0041]FIG. 5 shows an example of ink  70  on the touch surface  14  together with a plurality of gesture suggestions  72 . In this particular embodiment, the gesture suggestions  72  are in the form of selectable buttons  80  to  90 . The buttons  80  to  90  represent various available interpretations that can be used to interpret the ink  70 . In this particular example, the buttons include a text recognition button  80 , an insert button  82 , a freehand button  84 , a delete button  86 , a rewind button  88  and a help button  90 .  
         [0042]    When the text recognition button  80  is selected, the gesture suggestion and writing recognition routine performs handwriting recognition and converts the ink into corresponding text. When the insert button  82  is selected, the gesture suggestion and writing recognition routine converts the ink into corresponding text and inserts the text at the next point on the touch surface  14  where a contact is made. If the freehand button  84  is selected, the gesture suggestion and writing recognition routine treats the ink as a graphic object, that can be moved and manipulated on the touch surface  14  in response to mouse event input generated by the touch panel  12 .  
         [0043]    When the delete button  86  is selected, the gesture suggestion and writing recognition routine removes the ink from the acetate image. When the rewind button  88  is selected, the gesture suggestion and writing recognition routine rewinds the last portion of the ink that was entered by the user to allow for immediate correction of errors. When the help button  90  is selected, the gesture suggestion and writing recognition routine displays help information concerning the other buttons  80  to  88 .  
         [0044]    [0044]FIG. 6 shows an alternative example of ink  100  drawn on the touch surface  14  together with gesture suggestions  102  representing available interpretations for the ink. In this example, the word “cat” is displayed on the touch surface  14 . The gesture suggestions include an English text selection  110 , a Chinese text selection  112  and an image selection  114 . When the English text selection  110  is selected, the gesture suggestion and writing recognition routine converts the ink into English text  116 . When the Chinese text selection  112  is selected, the gesture suggestion and writing recognition routine converts the ink into the corresponding ideographic character  118 . When the image selection  114  is selected, the gesture suggestion and writing recognition routine replaces the ink with a corresponding image  120 .  
         [0045]    As will be appreciated, by displaying available interpretations for the ink and allowing a desired interpretation to be selected, ink on the touch surface  14  is correctly interpreted and recognized to a high degree. The concise form of gesturing provides good visual feedback to assist physically or mentally challenged individuals.  
         [0046]    Although selecting the desired interpretation for ink is described as being achieved by selecting an icon displayed on the touch surface  14 , those of skill in the art will appreciate that alternatives are available. For example, the gesture suggestion and writing recognition routine can automatically analyze ink on the touch surface whenever a pen down or other designated condition occurs. In this case, the gesture suggestion and writing recognition routine categorizes ink either as text or freehand writing by calculating certainty values and performs the ink recognition in a manner based on the result of the categorization.  
         [0047]    The icons displayed on the touch surface  14  can of course be replaced with a menu listing the available interpretations. Alternatively voice recognition can be used to enable the user to select the desired interpretation for the ink.  
         [0048]    The set of available interpretations described herein with reference to FIGS. 5 and 6 is for illustrative purposes only. As will be appreciated, the set of interpretations can be expanded indefinitely. For example, interpretations corresponding to different languages, different shapes and symbols, such as math symbols, can be included.  
         [0049]    Although the touch system has been described as switching between the pointer mode and the ink mode depending on the type of pointer used to contact the touch surface, this is also for illustrative purposes only. For example, transitions between the pointer mode and ink mode can be achieved by selecting appropriate buttons on a small window that remains visible on the touch surface  14 . In this manner, the same pointer can be used to generate mouse events or ink. The gesture suggestion and writing recognition routine can be used in any touch system that includes a touch panel on which a computer image is presented. Such touch systems include pen-tablet computers, mobile computers, mobile phones as well as active touch systems that make use of special pointers that emit signals such as infrared light, visible light, ultrasonic frequencies and electromagnetic frequencies in order to activate the touch surfaces. These touch systems also include passive surface acoustic wave or capacitive-type touch systems as well as camera-based touch systems such as that disclosed in International PCT Application No. WO 02/03316 to Smart Technologies Inc. et al., the assignee of the present invention.  
         [0050]    Although preferred embodiments of the present invention have been described, those of skill in the art will appreciate that variations and modifications may be made without departing from the spirit and scope thereof as defined by the appended claims.