Abstract:
A method and apparatus for user input on a handheld device with a virtual keyboard using secondary surfaces. On the primary surface of the device (e.g., front), the user interacts via touch sensors and a display element. Secondary surfaces (e.g., back) include additional touch sensors through which the user can also provide input. The display element is used to present information appropriate to the device&#39;s function (e.g., email messages) and control elements, including a virtual keyboard. The user interacts with the touch sensors on the first surface to bring up the virtual keyboard. Once displayed, the user can interact with this keyboard using either the primary surface or secondary surfaces. When used on appropriately sized device, the user can hold the device with the palms and thumbs of both hands and use their fingers on the touch sensors on the secondary surfaces to type. The selection of a key on the virtual keyboard is accomplished the combination of contacts made on the touch sensors on the secondary surfaces. The selected key, or region of the keyboard, is visually indicated on the front surface. Input of the keystroke is recorded when the user removes their touch from certain touch sensors on the secondary surfaces.

Description:
BACKGROUND 
       [0001]    The invention relates generally to user input for computer systems and more particularly to efficient data input into handheld devices. An emerging class of handheld devices use a display element to present a virtual keyboard to the user for input. The user touches the display to enter data on this keyboard. This input method allows changes in the keyboard design without requiring changes in the physical device. However, this approach limits the rate of input based on the speed and accuracy of the user&#39;s touches and the system to sense these inputs. The first generation of these platforms are the Apple iPhone, iPod Touch, and Motorola Droid (iPod and iPhone are trademarks of Apple, Inc, and Droid is a trademark of Motorola). A second generation of handheld devices, generically referred to as tablet computers, have recently been released, including the Apple iPad (iPad is a trademark of Apple, Inc). These devices are larger than the first generation and allow for more conveniently holding the device with two hands. 
         [0002]    The user expects to be able to input data into these devices while holding it. For example, a user may want to enter notes from a lecture or a meeting on this device. If the user holds the device in portrait mode and calls up a keyboard to enter data, the user could use a thumb typing and reach across the screen. If the user is in landscape mode, the virtual keyboard may need to split to allow the user to use thumb typing since the distance across the device in landscape mode may exceed the user&#39;s reach with their thumbs. If a virtual keyboard is the full width of the screen in landscape mode, the user will need to use two hands to type effectively and will need to rest the device on something. 
         [0003]    Another approach to typing on these devices is to use the back of the device as a touch sensitive surface that acts as if touches on the back correspond to touches on the front (See USPTO Patent Application 20070103454). If the locations of the touches on the back of the unit have a one-to-one correspondence with the keys on the virtual keyboard, the user will have to accurately position their hands for each individual key. This is difficult to accomplish for the average user. 
         [0004]    Other approaches may use add-on keyboards (e.g., bluetooth keyboard), but they suffer all the problems of physical keyboards. In a handheld device, the ergonomics of viewing the screen while typing becomes problematic. A stand could be used, but this adds additional components for using this portable device. Likewise, the addition of an external keyboard makes using this portable device cumbersome. A slide-out keyboard now makes the device larger and more prone to failure and limits the orientations that the device can be used in. Both external keyboards and slide-out keyboards limit the availability of unique virtual keyboard layouts for various software applications. 
       SUMMARY 
       [0005]    In one embodiment the invention provides a method to interact with a virtual keyboard while holding the device with both hands and using touch input on secondary surfaces to select keys. The touch input on the secondary surfaces does not require highly accurate placement of the fingers to reach distinct locations for each key. Instead, the touch input requires combinations of touch patterns to represent the various keystrokes. The system can provide visual feedback to the user to allow them to discover the right pattern for each keystroke. The system supports the use of customized keyboard layouts with a consistent method for identifying keystrokes. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0006]      FIG. 1  shows a front view of a prior art handheld device. 
           [0007]      FIG. 2  shows an embodiment of a handheld device in accordance with the invention. 
           [0008]      FIG. 3  shows another embodiment of a handheld device in accordance with the invention wherein multiple sensor areas are provided. 
           [0009]      FIG. 4  shows a virtual keyboard and three zones the user selects using two touch areas. 
           [0010]      FIG. 5  show the left zone of a virtual keyboard as controlled by the user for column selection. 
           [0011]      FIG. 6  show the middle zone of a virtual keyboard as controlled by the user for column selection. 
           [0012]      FIG. 7  show the left zone of a virtual keyboard as controlled by the user for row selection. 
           [0013]      FIG. 8  shows the touch sensors involved with calibrating the position of L 1 , L 2 , L 3 , and R 1 , R 2 , R 3  relative to the user&#39;s grasp of the device. 
           [0014]      FIGS. 9A ,  9 B, and  9 C show a view from above of three possible embodiments of the handheld device in accordance with the invention, illustrating possible placements of secondary surfaces on the device. 
       
    
    
     DETAILED DESCRIPTION 
       [0015]    The following description is presented to enable any person skilled in the art to make and use the invention as claimed and is provided in the context of the particular examples discussed below, variations of which will be readily apparent to those skilled in the art. Accordingly, the claims appended hereto are not intended to be limited by the disclosed embodiments, but are to be accorded their widest scope consistent with the principles and features disclosed herein. 
         [0016]    Small multi-media handheld devices with touch screens such as mobile telephones and tablet computers typically use a virtual keyboard for user input. A device can have many virtual keyboard layouts to assist in a variety of data entry tasks. An illustrative prior part device that is laid out in this manner is the iPad from Apple, Inc. As shown in  FIG. 1 , the main face of the iPad  100  comprises of a touch-sensitive LCD  110 . Within this display element, a virtual keyboard  120  is illustrated. 
         [0017]    In contrast, a multi-media handheld device in accordance with the invention include additional touch sensors on secondary surfaces. More specifically, touch-sensitive sensors are provided on surfaces on the device that can be interacted with while holding the device. These sensors are used to augment the input accomplished by the touch sensors on the display element. When the device is activated or placed into an operational state where it is appropriate, control elements (e.g. soft keys and menus) are displayed on the display element. Prior art devices would require the user to touch the display element to indicate their input. This can be awkward to use the entire keyboard and simultaneously hold the device. 
         [0018]    Referring to  FIG. 2 , a view of the back of multi-media handheld device  200  in accordance with one embodiment of the invention includes six touch-sensitive areas,  210 ,  220 ,  230 ,  240 ,  250  and  260 . As used herein, these six touch-sensitive areas may be created using several technologies. These may include discrete capacitive touch detectors, discrete pressure switches, or a touch-sensitive surface that is adapted to simultaneously detect where one or more object touch it (e.g., fingers) and the effects those objects create on the sensors. The location of these touch sensitive areas L 1 , L 2 , L 3 , R 1 , R 2 , R 3  may be in fixed positions, or they may be adjustable. The location of these touch-sensitive areas may be physically marked on the outside of the case or may be unmarked. The marking may be accomplished in a variety of manners, including using indentations or raised markers. 
         [0019]    Referring to  FIG. 3 , a view of the back of multi-media handheld device  300  in accordance with one embodiment of the invention includes twenty-four touch-sensitive areas for use in four orientations. The areas for portrait use  330 ,  331 ,  332 ,  340 ,  341 ,  342  correspond to the areas  210 ,  220 ,  230 , and  240  in the  FIG. 2 . Additional touch-sensitive areas  310 ,  311 ,  312 ,  320 ,  321 ,  322 , and  323  are added to allow the device to be used in portrait mode when inverted. To use the device in landscape mode, touch-sensitive locations are added in  350 ,  351 ,  352 ,  360 ,  361 , and  362  as well as at  370 ,  371 ,  372 ,  380 ,  381 ,  382 . In this embodiment, some touch sensitive locations overlap. The unit  300  may include a sensor to detect the orientation of the device to determine how to interpret touches at these overlapping locations. For example, if the device is in landscape mode, a touch in the area of  372  and  320  could be considered a touch at location  372  to better reflect the overall situation and likely intent of the user. The overlap of touch-sensitive areas in  300  make the use of a touch sensitive surface a practical implementation for the sensors. 
         [0020]    Referring to  FIG. 4 , the virtual keyboard on the display element is illustrated with additional divisions  410 ,  420  and  430 , referred to as zones. This keyboard layout is for inputting the Roman alphabet, but the invention is not limited to this alphabet or keyboard layout. The invention allows for application to any virtual keyboard layout that can be divided into up to three zones and have up to 16 keys in each zone, when using only 6 touch sensitive areas. The addition of two additional touch sensitive areas, using all four fingers, can expand the addressable number of keys in any keyboard as a natural variation of this invention to those skilled in the art. The user interacts with one embodiment of the device  100  through its touch-sensitive display element  110  to bring up a virtual keyboard  120 . The virtual keyboard, extended to support the concept of input from secondary surfaces and the selection of zones of the keyboard, controls the zone boundaries for each particular keyboard layout within the constraints of no more than 16 keys per zone. The zones are not required to be precisely square. In fact, most keyboard layouts have a natural staggered key design and the invention can easily accommodate this, as shown in zones  410 ,  420 , and  430 . Zone  410  has four rows. If the key  440  (numeric keyboard) is considered to cover two key spaces, then each column in zone  410  has three columns. Similarly, zone  420  has four rows. The first and third row have four columns; the second row has three columns, and the final row can be considered to have four columns which all map to the same key  450 . In zone  430 , if the key  470  is considered to cover two key spaces and key  460  covers three key spaces, then the zone can be considered to have four rows with three columns. In embodiments with different keyboard layouts, those skilled in the art could easily apply this invention to determine appropriate zone boundaries, and row and column assignments for each key. 
         [0021]    Once in the state to accept keyboard input, the user can use finger touches on the secondary surfaces to select keys for input. The touch sensitive areas are assigned symbolic names L 1 , L 2 , L 3 , R 1 , R 2 , and R 3 . One embodiment  200  for this may associate touch areas  210 ,  220 ,  230 ,  240 ,  250 , and  260  with R 1 , R 2 , and R 3  and L 1 , L 2 , and L 3 . These associations can be controlled by software to meet various users preferences. For the following discussion, specific mappings will be used, but other mappings are within the scope of this invention. Touches to  200 ,  210 ,  220 ,  230 ,  240 ,  250 , and  260  will be mapped to R 1 , R 2 , R 3 , L 1 , L 2 , L 3  respectively, for this discussion. The invention uses L 1  and R 1  to select the zone of the keyboard  400 . Touch on L 1  is used to indicate that the user wants to select a key in zone  410 . Touch on R 1  is used to indicate that the user wants to select a key in zone  430 . Touching both L 1  and R 1  indicates that the user wants to select a key from zone  420 . No zone or key selection is made if the user does not touch either L 1  or R 1 . If the number of keys in virtual keyboard is sufficiently small, it may be made of only two zones, each selected by pressing individually L 1  or R 1 . To allow the user to learn the required touches, the virtual keyboard  400  can react to touches by highlighting the selected zone. For example, if a user touched L 1 , one embodiment would highlight zone  410 . In other embodiments, the zone  410  could be highlighted and the other zones  420  and  430  could be dimmed. In other embodiments, zone  410  could be left unaltered and the other zones could be dimmed. This highlighting or dimming is generally indicating the user&#39;s selected area of focus. Likewise, if the user touched R 1 , in one embodiment zone  430  could indicate the user&#39;s focus and if both L 1  and R 1  are touched, the zone  420  could indicate the user&#39;s focus. If L 1  is pressed and then R 1  is pressed, the user might be shown initially the focus on  410  and then the focus indication would shift to zone  420 . 
         [0022]    Once a zone has been selected, the virtual keyboard indicates to the user the selected row and column based on the state of L 2 , R 2  and L 3 , R 3 . In  FIG. 5 , the left portion of the virtual keyboard is shown. In accordance with the invention, L 2  and R 2  select the column within the selected zone. In one embodiment, the virtual keyboard indicates the selected column to assist the user in learning the touches to select a specific key. Touching neither L 2  nor R 2  will select the middle column  520 . Touching L 2  only will select the left column  510 . Touching R 2  only will select the right column  530 . Touching both may be ignored, effectively touching neither. This results in selecting the middle column  520 . As the user changes their touches, the virtual keyboard  500  can animate their changing selections to assist the user in understanding the impact each touch has on the selection.  FIG. 6 . shows a keyboard with four columns instead of the three shown in  FIG. 5 . The behavior of the effect on touches to the selection of columns is equivalent to that in  FIG. 5 , with an extended meaning of touching both L 2  and R 2 . Touching neither L 2  nor R 2  will select the left-middle column  620 . Touching L 2  only will select the left column  610 , effectively shifting the selection one to the left. Touching R 2  only will select the right column  640 , effectively shifting the selection two to the right. Touching both select the right-middle column  630 . This is consistent with the meaning of L 2  moving the selection one to the left and R 2  moving the selection two to the right. Selecting both causes both actions, with a combined effect of moving the selection one to the right. As the user changes their touches, the virtual keyboard can animate their changing selections to assist the user in understanding the impact each touch has on the selection. 
         [0023]    Referring to  FIG. 7 , the left zone of the virtual keyboard is shown. In accordance with the invention, L 3  and R 3  select the row within the selected zone. When the selected zone has four rows, the following behavior is performed. Touching neither L 3  nor R 3  will select the upper-middle row  720 . Touching L 3  only will select the top row  710 , effectively shifting the selection one up. Touching R 3  only will select the bottom row  740 , effectively shifting the selection two down. Touching both select the lower-middle row  730 . This is consistent with the meaning of L 3  moving the selection one up and R 3  moving the selection two down. Selecting both causes both actions, with a combined effect of moving the selection one down. As the user changes their touches, the virtual keyboard can animate their changing selections to assist the user in understanding the impact each touch has on the selection. 
         [0024]    When the selected zone has three rows, the following behavior is performed. Touching neither L 3  nor R 3  will select the middle row  720 . Touching L 3  only will select the top row  710 . Touching R 3  only will select the bottom row  740 . 
         [0025]    After the user selects a zone, the invention always has a row and column selected. In some embodiments, this is visually indicated to the user. The intersection of these selections selects the place where the effective touch will be generated on the virtual keyboard. As the user changes the selection, the effective touch changes. The virtual keyboard can react to this. Prior art devices such as the Apple iPhone highlight the key being selected with a touch. Moving the point of contact while still holding the finger down allows the selection to change without generating the actual keystroke. The keystroke is generated upon release of the touch. For the invention, the keystroke is generated when the prior touch to at least L 1  and R 1  are released. The user can move between zones without causing a keystroke by maintaining at least one finger on either L 1  or R 1 . So, a user can start with a touch on L 1 , then add R 1 , then release L 1  to move the zone selection from the left to the right, as needed. Once the rest of the key selection is completed, the user can release R 1  to generate the desired keystroke. Other variations of this invention may require the user to release all touches on L 1 , L 2 , L 3 , R 1 , R 2 , and R 3  or other subsets before generating a keystroke. 
         [0026]    Referring to  FIG. 8 , possible embodiments may use surfaces capable of detecting multiple touches simultaneously. Some embodiments will not have physical indicators of the preferred location for the touch areas. This allows for easier use of the device in multiple orientations and multiple grasp locations in an orientation. Instead, the invention needs to calibrate the location of L 1 , L 2 , L 3 , R 1 , R 2 , and R 3  to the user&#39;s grasp. The invention has two approaches to accomplish this. Various embodiments can use either approach. One approach to calibrating the grasp of the user recognizes that the user will likely switch between using the primary surface and the secondary surface for input. At the beginning of a transition, the user must touch six fingers to the back of the unit and release them. The invention records the centroids of these locations as the centroids for the six touch locations  810 ,  820 ,  830 ,  840 ,  850 , and  860 . After this grasp calibration, touches to the unit will select the zones and then keys as described earlier. The keyboard can block reacting to all touches on the secondary surfaces, or even change its physical appearance to provide feedback to the user, to indicate that a calibration touch is needed. 
         [0027]    A second approach to calibration uses sensors  870  and  880  near to the edge of the unit. When the user grasps the unit with their palms, these sensors will be able to detect the extent of the contact  875  and  885 . This will allow the system to compute the location of the L 1 , L 2 , L 3 , R 1 , R 2 , and R 3  locations relative to the palm placements. This approach to calibration uses  875  and  885  to compute the locations for  810 ,  820 ,  830 ,  840 ,  850 , and  860 . 
         [0028]    In order to compensate for shifts in the user&#39;s grip, the invention tracks the location of touches and can adjust these touch locations. If the system detects touches outside of these areas, the invention allows the system to re-enter the calibration process. 
         [0029]    Embodiments of the system can combine these approaches. The initial calibration can use both the six finger contact and the palm placement to better estimate the location of the hands and their angle across the back of the unit. The system can then track both the palm positions as the grip drifts over time and track relative locations of touches to detect angular drift over time of the finger position relative to the palm placement. 
         [0030]      FIGS. 9A ,  9 B, and  9 C show various embodiments for the secondary surfaces described in this invention as seen from a top view looking down at the device with the display element on the surfaces  910 ,  940 , and  970 , respectively. In  FIG. 9A , embodiments of this invention could use surface  920  for the L 1 , L 2 , and L 3  touch locations and surface  930  for the R 1 , R 2 , and R 3  touch locations. Embodiments that are using palm placement calibration may include touch sensors on surfaces  925  and  935 . In  FIG. 9B , embodiments of this invention could use surface  950  for the L 1 , L 2 , and L 3  touch locations and surface  960  for the R 1 , R 2 , and R 3  touch locations. Embodiments that are using palm placement calibration may include touch sensors on surfaces  955  and  965 . In  FIG. 9C , embodiments of this invention could use surface  980  for the secondary surface touch locations L 1 , L 2 , L 3 , R 1 , R 2 , and R 3 . Embodiments that are using palm placement calibration may include touch sensors on surfaces  980 , or use  985  and  995 , or both. 
         [0031]    Embodiments of the invention may be integrated into an electronic device or be an accessory to an electronic device. When the embodiment is an accessory, the embodiment may communicate with the electronic device via a wired or a wireless mechanism. The accessory may be powered from the electronic device or may have its own power, or may even offer additional power to power both the accessory and the electronic device. 
         [0032]    In a typical implementation, touch surface is comprised of a number of sensing elements arranged in a two-dimensional array. Each sensing element (aka, ‘pixel’) generates an output signal indicative of the electric field disturbance (for capacitive sensors), force (for pressure sensors), or optical coupling (for optical sensors) at the sensor element. The ensemble of pixel values at a given time represents a ‘proximity image’. Touch surface controllers provide this data to a processor. The processor, in turn, processes the proximity image information to correlate the user&#39;s finger movements across the touch surface. 
         [0033]    Various changes in the materials, components, circuit elements, techniques described herein are possible without departing from the scope of the following claims. For instance, illustrative hand-held device  200  may include physical buttons and switches in addition to those described herein for auxiliary functions (e.g., power, mute, reset buttons). In addition, the processor performing the method may be a single computer processor, a special purpose computer processor (e.g., a digital signal processor), a plurality of processors coupled by a communications link or a custom designed state machine. Custom designed state machines may be embodied in hardware devices such as in integrated circuit, including but not limited to application specific integrated circuits (“ASICs”) or field programmable gate arrays (“FPGAs”).