Abstract:
A contactless display system enables a user to interact with a displayed image by moving a finger, or pointer, toward a selected portion of the image. Images can be enlarged, or translated dynamically in response to detected movement. Operational methodology can be manually switched between contact-type and contactless operation to enhance flexibility.

Description:
FIELD 
       [0001]    The invention pertains to systems and methods to navigate small screen displays. More particularly, the invention pertains to such systems and methods with contactless sensors for tracking a trajectory of a user&#39;s finger toward a virtual keyboard. 
       BACKGROUND 
       [0002]    Various types of small screen displays can be found on products such as cellular telephones, personal digital assistants (PDAs), mobile computers and imagers. Increasingly, users must navigate small screen displays to do web browsing with a mobile phone or photo browsing using a PDA. 
         [0003]    Small touch screens are popular to support interaction with applications running on portable devices, such as PDAs and mobile phones. Small touch screens also are finding their way into home products such as the Honeywell TH8321U1006 thermostat, the Honeywell 6271V security panel, and various personal health monitoring devices. They have been used for years in parcel delivery, retail warehouse operations, and refinery field operations. 
         [0004]    Navigation schemes using zooming and panning controls or by a fish-eye viewer are known. However, these controls are inconvenient or are inefficient to use them in some situations. For example, it is hard to use fish-eye navigation with a touch screen. It is also hard to zoom and pan graphics with a mouse or a touch screen. Navigation with a stylus on a small screen is perhaps even more difficult for the user. 
         [0005]    There is thus a continuing need for alternative approaches to control navigating large and small scale graphics that is natural and easier to use. 
         [0006]    It would also be desirable to be able to use such approaches in navigating large scale graphics such as maps or building floor plans. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0007]      FIG. 1  is a diagram of a touchless interaction system in accordance with the invention; 
           [0008]      FIG. 2  is a block diagram of some of the software element of the system of  FIG. 1 ; 
           [0009]      FIG. 3  is a flow diagram illustrating a method of interaction; 
           [0010]      FIG. 4A , B illustrate two different applications of the system of  FIG. 1 ; 
           [0011]      FIG. 5  is a touchless mode entry screen; 
           [0012]      FIG. 6  is an intermediate touchless mode screen; 
           [0013]      FIG. 7  is a touchless mode regional display screen with an exit button; 
           [0014]      FIGS. 8A , B, C illustrate various aspects of the contactless sensors of the system of  FIG. 1 ; 
           [0015]      FIGS. 9A , B, C illustrate one form of virtual keyboard touchless navigation; and 
           [0016]      FIGS. 10A , B, C illustrate another form of virtual keyboard touchless navigation. 
       
    
    
     DETAILED DESCRIPTION 
       [0017]    While embodiments of this invention can take many different forms, specific embodiments thereof are shown in the drawings and will be described herein in detail with the understanding that the present disclosure is to be considered as an exemplification of the principles of the invention, as well as the best mode of practicing same, and is not intended to limit the invention to the specific embodiment illustrated. 
         [0018]    Embodiments of the invention incorporate a touchless, or contactless, interface, one that senses the position of a user&#39;s finger or hand in three dimensions. In a disclosed embodiment, a plurality of capacitive sensors can be arranged at the edges of a display device. The trajectory of the finger or hand can be tracked toward a point on a virtual keyboard being displayed on the device. This enables the associated system to anticipate the point on the screen of the display device that the user is attempting to select, before the finger actually touches the screen. 
         [0019]    Z-axis finger position data can be used, in accordance with the invention, to control the zoom ratio or zoom range on, for example, a map display. Alternately, a fish eye on a map display can be controlled with this touchless pointing approach. Multiple parameters of the fish eye can be modified during the process, such as zoom ratio, zoom range, zoom shape (rectangle, round rectangle, ellipse, etc.), and proportion of the distorted edge around the fisheye. 
         [0020]    As the user moves his/her finger, the graphical content in the display is updated accordingly. Finger movement can also control zooming/spanning operation or fish-eye effect on the map display. This process should be highly efficient and intuitive for users. 
         [0021]    The same approach can also be used to control and interact with a virtual keyboard on a small screen display. It overcomes the chronic problem associated with small virtual keyboards in that the keys are always much smaller than the human finger tip. 
         [0022]    Precise interaction requires magnification of only the target region of the keyboard (e.g. some small subset of the keys). The touchless interface uses z-axis data pertaining to hand position to infer the desired target region on the keyboard and automatically zooms in or magnifies the desired region on the virtual keyboard. 
         [0023]    In some applications, the input signal from the touchless device might disturb interactions which don&#39;t need to be touchless. In an aspect of the invention, several different approaches can be used to intuitively and quickly disable/enable the touchless interaction. In one embodiment, the user&#39;s right hand can be used for pointing and controlling the zoom control or fish eye control and the left hand can be used to operate a button that enables or disables the touchless navigation. In this process, the left hand also can be used to quickly change fisheye or zoom parameters on the fly simultaneous with the right hand doing the pointing and dragging to provide a very efficient two-handed interaction. 
         [0024]      FIG. 1  illustrates a touchless, or contactless interaction system  100 . The system  100  includes a programmable processing unit  151  connected over data bus  152  with touch screen input buffer  153 , touchless input buffer  154 , display buffer  155 , buffer  156  and storage unit, or, memory  157 . Touch screen  158  is coupled to the processor through touch screen buffer  153 . Touchless sensing device  159 , for example a plurality of capacitive based non-contact sensors, is coupled to the processor through touchless input buffer  154 . 
         [0025]    Graphical display  160  is coupled to the processor through display buffer  155 . Display device  160  includes a display screen upon which various images are presented. The touchless sensors  159  are located around the periphery of the display screen of the display device  160  as discussed in more detail subsequently. 
         [0026]    I/O devices  161  are coupled to the processor through I/O buffer  156 . I/O devices may include any combination of devices allowing the system to interact with external information. 
         [0027]    Storage unit  157  contains the information and/or programs or executable software necessary for the processor  151  to implement a touchless interaction system. For example, display control software  157   a  can be stored in a computer readable form in unit  157 . Other system control software can also be stored in unit  157 . 
         [0028]      FIG. 2  illustrates various software modules  200  of the system  100  which are executed by the processor  151 . Modules  200  can be stored in unit  157  in magnetic or optical computer readable form. Software  200  includes a Command Executive module  202 , a Command Recognizer module  204 , a Data Receiver  206 , a Graphical System Display module  208  and Domain Models  210  which provide information as to the region being displayed. Operation of various modules is discussed relative to the process  250  of  FIG. 3 . 
         [0029]    As illustrated in the flow diagram of  FIG. 3 , data from the sensors, such as sensors  158 , or  159  is loaded into respective receivers such as  206   a,  b from buffers  153 ,  154 , as at  252 . That data is loaded into input buffer  204   b  as at  254 . 
         [0030]    Gesture Analyzer  204   a  analyzes the data as at  256 . The Gesture Analyzer  204   a  sends a system command to the Command Executive  202  as at  258 . The Command Executive modifies the status of Domain Objects, of the Models  210  as at  260 . 
         [0031]    The Command Executive  202  notifies the Graphical System module  208  to alter the status of the visual image on the display  160 , as at  262 . The Graphical system module  208  updates the image on the display unit  160 , as at  264 . The Command Executive module  202  then updates the system status, as at  266 . 
         [0032]      FIG. 4A  illustrates one embodiment of the invention, a contactless, navigational regional display such as might be used to evaluate an alarm condition of a building. In a second embodiment,  FIG. 4B  illustrates small display contactless navigation in accordance with the invention. 
         [0033]      FIG. 5  illustrates a display of the type of the embodiment, or application, of  FIG. 4   a  in an initial display state. In Step  1 , as indicated, a User can click on a button to enter a touchless navigation state.  FIG. 6  is a confirmation screen presented to the User on the display unit  160 . The User can enter the touchless navigation state as illustrated at Step  3 .  FIG. 7  illustrates a screen present on the display unit  160  when in the touchless navigation state. A button is provided to exit the touchless state. 
         [0034]      FIGS. 8A , B, C illustrate various aspects of the characteristics of the contactless sensors  159  arranged about the perimeter of the screen  160   a  of the display unit  160 . As illustrated therein, sensors  159  define an outer frusto-conical sensing region  160   b  and an inner region  160   c.    
         [0035]    When the User&#39;s finger or pointing device is in the outer region  160   b,  the regional display or map can be navigated or scrolled as well as zoomed. As the User&#39;s finger approaches the screen, in the region  160   b,  and  160   c,  the presented image zooms from one level to a to a more detailed level, for example. When the User&#39;s finger enters the interior region  160   c  the User can, in one embodiment, only zoom in and out on the map or display. The region  160   c  could help the end user zoom in/out of the map or display smoothly without dithering or jittering. 
         [0036]      FIGS. 9A , B, C illustrate steps of one solution to navigating virtual keyboards using the system  100 . A region can be selected and enlarged by the User to activate a key. Then, a second region can be selected and enlarged by the User to activate a different key, and so on until the desired entry has been completed. 
         [0037]      FIGS. 10A , B, and C illustrate steps of another solution to navigating virtual keyboards using the system  100 . In the embodiment of  FIGS. 1A , B, and C, any portion of the keyboard to which the User&#39;s finger moves can be magnified or enlarged to enable the User to seamlessly activate a sequential group of keys. It will be understood that embodiments of the present invention can be incorporated into electronic devices such as wireless phones, mobile computers, or imaging devices, all without limitation which might have relatively small keyboards. 
         [0038]    From the foregoing, it will be observed that numerous variations and modifications may be effected without departing from the spirit and scope of the invention. It is to be understood that no limitation with respect to the specific apparatus illustrated herein is intended or should be inferred. It is, of course, intended to cover by the appended claims all such modifications as fall within the scope of the claims.