Patent Description:
Mobile computing continues to grow quickly as mobile devices, such as smart phones, add more power and more features. Users of such devices may now access various services on the internet, such as mapping applications, electronic mail, text messaging, various telephone services, general web browsing, music and video viewing, and similar such services. The small size of mobile devices may make it difficult to interact with such services, because a full QWERTY keyboard may not be available on a mobile device or may not be available at all times. Thus, input to a mobile device may occur via a standard telephone dialing pad (e.g., via multi-tap inputs) or a touch screen.

In addition, interaction with a mobile device may occur in a variety of situations, in varying levels of concentration for a user. At one end of a spectrum, a user may be able to provide full attention to their device, such as when they are at their desk or riding on mass transit. At the other end of the spectrum, a user may be busy having a conversation or driving their automobile, so that any interaction with their mobile device should require a minimum amount of attention from the user.

<CIT> discloses techniques for text input in various apparatuses, which may be useful for selecting one of items from a group of candidate items displayed on a display screen. In one example, for a <NUM>*<NUM> grid of items, the center item is selected by default. If this is the desired item, the user only needs to confirm this default selection. If one of the items around the center is desired, a selection may be made by directional movement using a 2D position sensor, which may be a 2D touch sensor. The directional movement is then measured in relation to an array of angle segments. For example, a down-left movement would select the bottom left item, independent of the position of the user's finger on the 2D position sensor.

<CIT> discloses an apparatus and method for inputting a text that inputs a text corresponding to a relative coordinate value that is generated by touch and touch position movement, wherein the method, based on touch position movement along a touch pad or touch screen, generates a series of relative coordinate values sequentially, temporarily displays the texts that correspond to said values on a display, and, upon termination of touch with said touch pad or touch screen, finally inputs a text in response to a touch termination signal.

<CIT> discloses a system associated with a text entry application, which comprises an optional onscreen representation of a circular keyboard. The circular keyboard is manipulated via a hardware joystick, although as an alternative any input device that provides the possibility of directional input with a sufficient precision can be used, including for example a touchpad or touchscreen. The user points the joystick in the general direction of the desired letter, and then continues pointing roughly to each letter in the desired word. It is possible to navigate through multiple concentric rings of characters. As soon as a direction has been established by some degree of tilt from the center, the input may be registered and recorded. It may still be beneficial to the user, however, to allow the direction to be altered slightly before recording it. Therefore, the last effective direction is preferably only recorded after the joystick is returned to its resting position.

This document describes systems and techniques that a user may employ in order to enter information into a mobile computing device. In general, an application or operating system on a mobile device may establish certain gestures on a touch screen that will result in particular inputs being used by the application or another application on the device. The initial point of contact by a user on a display (e.g., with the user's fingertip) may establish an anchor point that is then used to identify a subsequent direction of input by a user (e.g., via a dragging motion or a second press on the screen). The direction may be mapped to a keypad, with the anchor point representing a key or area on the keypad (e.g., a center key of the keypad) and the dragging direction representing a desired key in the relevant direction from the center key. Where a keypad is larger, so that multiple keys may be located in the same direction from an anchor key, additional user input may indicate the appropriate key, from the multiple keys in the particular direction, that is intended by the user. In addition, feedback may be provided to a user to indicate the keypress that has been registered in response to their input, such as by having a device speak the input associated with the keypress. Such feedback may come in a number of forms, such as spoken (e.g., synthesized or digitized speech), auditory (e.g., tones or clicks), and tactile (e.g., rumble or tap) feedback, where such feedback is synchronized to registration of the inputs. Thus, in addition to hearing the value of a pressed key spoken, a user may also hear a short sound, synchronized with tactile feedback such as vibration. Such feedback may improve the sensation of "moving over" or "pressing" a button.

In one example, a telephone dialing system may be provided. A user's initial point of contact with a touchscreen may be taken by the system as the location of the "<NUM>" key on a telephone keypad. Subsequent dragging by the user in a radial direction may select a key on the keypad that is in the radial direction from the "<NUM>" key, whether or not the user's contact on the screen is actually over a "<NUM>" key that might be displayed on a virtual keyboard on the screen. For example, dragging downward relative to the initial point of contact may indicate entry of an "<NUM>," while dragging in a upper-right or Northeast direction may represent entry of a "<NUM>. " Particular strokes may indicate entries of keys that are not directly radially related to the "<NUM>" key, such as a tap indicating entry of the "<NUM>" key, a circle indicating entry of the "<NUM>" key, and other special strokes indicating entry of the pound or asterisk keys. Other inputs, such as shaking and/or tilting the device, can provide other commands.

In certain embodiments, the features discussed here may provide one or more advantages. For example, a user can enter data into a device without having to look at the device, where the data input is determined by a direction of input, and where the initial point of input from the user is independent of the keypress to be entered. In particular, the user may operate a mobile device without having to take their visual attention away from another activity. Moreover, such a system may be learned in certain implementations without a user having to learn an input lexicon. Also, blind users may more easily interact with a computing device having a touchscreen interface. Such features may have benefits in terms of better usability and safety.

In one implementation, a computer-implemented method is provided according to claim <NUM>.

In another implementation, a non-transitory computer readable storage device is provided according to claim <NUM>.

In yet another implementation, a computing device is provided according to claim <NUM>.

This document describes systems and techniques for receiving user input on a touchscreen of a computing device in a manner that is relatively quick and involves reduced user visual attention to the input process. The user input may begin with a user contact (e.g., a finger touch) on a touchscreen and may proceed with an additional user input in a direction from the initial user contact, such as by the user dragging their finger in a direction measured radially from the initial point of contact. The particular direction of the dragging may represent an intended input that corresponds to a key on a virtual keyboard or keypad that is located in the respective direction from an anchor key on the keyboard. Thus, for example, the point of initial contact may be assumed to be a center key of a keypad, whereas a direction of the subsequent user input may represent a key, in that direction relative to the center key, that the user would like to press. In certain scenarios, the keypad need not be shown on the display in order for the inputs to be mapped to the keypad, or the inputs need not be aligned with the keypad if it is displayed.

<FIG> shows example screenshots of a mobile device providing for touchscreen user input. In general, four different displays <NUM>-<NUM> are shown to provide examples demonstrating how the techniques described here can be implemented. Each of the displays <NUM>-<NUM> are shown in a mobile device having a touchscreen graphical user interface, where the device may be loaded with a number of computer applications, including a telephone dialing application. Telephone dialing may be a particularly appropriate application for the techniques described here because talking on a telephone traditionally does not require eye contact with the device, while dialing of a telephone typically requires visual or tactile contact. As a result, people may feel comfortable making a telephone call when they do not have such eye contact, but may unnecessarily take their attention away from their other work while they are dialing.

Display <NUM> shows such an example of user telephone dialing input. The display <NUM> looks like a standard smart phone virtual dialing pad, having a standard telephone keypad <NUM> displayed on a touchscreen, and a data entry area <NUM> showing numbers entered by a user. In a typical configuration, the user may simply tap each number on the keypad to enter the numbers. However, such entry requires the user to know where each of the numbers is located, and such awareness may require the user to look at the keypad - something that is difficult to do if the device is in a pocket or holster, or at the user's ear.

As a result, an alternative (or in this case, additional) data entry technique is provided. In particular, because normal dialing on the interface <NUM> involves tapping, the system can recognize dragging motions as involving an alternative user intent. In particular, dragging motions on the display <NUM> may be interpreted to be directed at entering dialing keypad information based on a direction of the dragging.

Two such dragging examples, shown on display <NUM>, may illustrate this point. The first dragging example <NUM>, labeled "A," represents a path of a user's finger dragging across the display <NUM> - here, in an upper-right or Northeast direction. The dragging motion starts with an initial user contact over the "<NUM>" key on the display <NUM>, but the ultimate input is determined independent of that particular correspondence between the contact point and the currently-displayed selection under the contact point. Rather, the character that is deemed to be input by the user is the character - on the virtual keypad shown on the display <NUM> - that is in the dragged direction relative to the center of the keypad. In this example, the center of the keypad is the "<NUM>" key, and the key to the upper-right of that key is the "<NUM>" key. Therefore, the data entry area <NUM> shows a "<NUM>" in its first position, so that a telephone number ultimately dialed by the device will start with the digit <NUM>.

The second dragging example <NUM>, labeled "B," represents a follow-up input by the user. This time, the user starts in the upper-right corner of the display <NUM> and drags to the upper-left, or Northwest. Again, the interpreted input is deemed to have started at the center of the keypad, and to have selected the key on the dragged direction on the keypad, which here is the "<NUM>" key (i.e., the "<NUM>" key is Northwest of the "<NUM>" key). As a result, the second displayed digit in the data entry area is a "<NUM>. " The actually starting point of dragging example <NUM> did not matter - rather, what mattered was the direction of the dragging.

In certain examples, the general location of the start of a dragging operation may have some effect on the input, as that input is interpreted by a device. For example, a screen may be split into two portions so that selections in directions on one half of the screen are interpreted in a different manner than selections in the same directions on the other half of the screen. Still, however, the selections may be independent of the actual initial location of the contact, in that any selection in the appropriate direction in the designated area, will have the same result, regardless of where in that designated area the selection began.

In certain situations, like that shown in display <NUM>, not all of the relevant keys will be arrayed in a single radial direction from an anchoring point, such as the "<NUM>" key. In those situations, additional gestures may be used to signal an intent to enter such additional keys. For example, a single tap may represent an intent to select the middle key. Other times, multiple taps or dragging other than in a radial direction may indicate such an intent. For example, drawing a circle - though it may be slower than a straight radial drag - may be used to indicate an intent to press the "<NUM>" key. Other tapping or curved gestures may represent an intent similar to pressing a "control" or "caps lock" key on a standard keyboard.

Display <NUM> provides an example similar to that in display <NUM>, but over a desktop for a mobile operating system. In particular, the display <NUM> shows a number of selectable application icons such as icon <NUM>, and a widget <NUM> in the form of a clock, displayed together on a single level of a desktop. This may be the display that is generally shown to a user of a device when no particular application is the focus of the device's operating system (e.g., when no applications are launched or when all the applications' windows are minimized). In this example, two user dragging selections are again shown. The first dragging selection <NUM>, labeled "A," is made in a downward direction, and the second dragging selection <NUM> is made in a rightward direction. Thus, if the relevant input mode of the device at the time of the entries were for a dialing application or another application relating to a telephone keypad (e.g., texting), the user selections may represent an intent to select the "<NUM>" key (which is below the center, "<NUM>" key) and the "<NUM>" key (which is to the right of the center, "<NUM>" key).

The user data entries may be interpreted in other contexts, however, and the particular context may be set in a number of different ways. As one example, user entries on the desktop may be assumed, initially, to relate to the elements on the desktop, so that, for example, tapping an icon causes an application associated with the icon to be launched, and dragging on an icon causes the icon to be relocated. However, if a user makes an input action to display <NUM>, where the action does not correspond to an action supported by the display <NUM> at the current time, the action may be taken by the device as directed at an alternative application. For example, if a user presses down on an icon and drags it to a location at which it cannot be dropped, such a seemingly incongruous action may be interpreted as being directed to an alternative input mechanism on the device, which will involve interpreting the input according to the direction of the dragging.

Also, a particular user gesture may be used to activate such an alternative input mechanism. For example, if a user traces a circle over a desktop, such an action may activate the alternative input mechanism. Also, accelerometer input may be taken as an indication to provide for an alternative input mechanism, such as by a user shaking the device in a particular manner. Also, if the device senses other environmental variables, such as by sensing that its front side or screen is close to an object (e.g. it is in a user's pocket or close to the user's head), the alternative input mechanism may be automatically enabled.

A user may be able to specify which application, of several applications, will be enabled when an alternative input mechanism is acting over a desktop. For example, in an operating system configuration screen or in a configuration screen for the alternative input mechanism, the user can identify an application to which input data is to be passed (such as a telephone dialer) when no other application is an obvious recipient of the input (such as when no other application is active). Also, a user may speak the name of an application to enable alternative touch input for that application.

One such user example may better highlight the concepts discussed here. For example, a user may speak the word "dialer" to have their device change to a mode in which dialing may take place. While the user could also speak the numbers to dial, they may be in a noisy area (or a quiet area) in which they would like to minimize their level of voice input. The voice input may launch a dialer application and may also cause that application to be displayed in place of a desktop that was previously shown on the display, as with display <NUM>. Alternatively, the dialer may be invoked or may not be displayed, as with display <NUM>. The user may then make dragging motions on the display in particular directions to enter keys to be dialed, and may shake their phone or press a dial key (e.g., such as one of the buttons below display <NUM> or display <NUM>) to cause the numbers they have entered to be dialed.

Alternatively, a user wanting to send a text message may speak the words "text," and may then enter characters for a text message. Such entry may occur in a number of ways. For example, the user may make entries that correspond to text entry via a standard telephone keypad. In one example, dragging in various compass directions or orientations may result in the entry of numbers, while dragging in a direction and then tapping one to four times may result in a particular letter associated with the key in that direction being entered.

A device may also provide audible feedback in the form of the content of an interpreted data entry intent. As one example for entering text, when a user drags to the right, their device may speak "six. " If they then lift their finger and tap once, the device may speak "m," to cycle through the various values for the "<NUM>" key. If the user then drags their finger down and right, the device may speak "nine," and if they tap again, the device may speak "w. " Another tap may cause "x" to be spoken, and another may cause "y" to be spoken. In this manner, a form of multi-tap input may be provided without the user having to look at the touchscreen. The spoken value may also be combined with an audible tone or other similar sound so as to more quickly indicate when an input value has been registered by a system.

Additional feedback may be provided tactilely. For example, a device may vibrate slightly when a character entered by a user is registered by a device, so that the user will know that they can move on to entering their next character. Such feedback may be in place of audible feedback or may be in addition to audible feedback, so as to reinforce the message provided by the audible feedback. In this manner, the user may be provided an improved sensation of pressing or rolling over a certain selection (even though their actually finger may not be rolling over any visible element on the touchscreen).

Deletion of characters that have been entered and other such functions may be performed in various manners, including by the functions being assigned to one of the virtual keyboard keys and being selected in the manners discussed above. Alternatively, special input mechanisms may be used. For example, deleting a character may be accomplished by shaking a mobile device (where the device includes an accelerometer) or by jerking the device to the left (to signify shifting of the cursor to the left). Such deletion may also be accompanied by audible and/or tactile feedback, such as by speaking all of the characters that remain after the deletion, or by speaking something like, "Letter 'S' deleted. " Also, a special button on the device may be used for such functions, such as an off-screen soft button.

Referring now to the third display in the figure, display <NUM>, a scenario like that shown in display <NUM> is provided. In this example, the presence of an alternative data entry interface is made known visually to a user. The interface relates to dialing on a telephone number but does not entirely replace a desktop with a dialer. Rather, an interface overlay <NUM> is shown in a translucent manner with an instruction of "dial a number" to indicate to a user who may be looking at the display <NUM> that the device is currently in an alternative data entry mode.

A data input box <NUM> may also be provided as part of the overlay <NUM> so as to provide an indication of the data that the user has entered. The data input box <NUM> may be translucent like the rest of the box or opaque so as to better display the data that a user has input to the device. In addition, other data display objects may be provided with the overlay <NUM> to provide visual input to a user, though the visual input may be optional when considering that the user may not be looking at the device while entering data into it.

In certain implementations, the data may be associated with a particular application only after it has been entered by a user. For example, if a user enters ten numeric digits and then provides an indication that they are finished entering data (e.g., by tapping or shaking their device), an interface module may infer that the person entered a telephone number and may provide the number to a dialer on the device. In a like manner, if the user enters a number of words, the interface module may provide the text to a text messaging application, to a note taking application or to another similar application, where the selection of an application can depend on the context (e.g., on the content or size of the text entered). Alternatively, a device may identify several possible uses for the text and may ask a user for an additional entry indicating how they would like the data used.

In the example in the figure, three different entries are shown. A first dragging selection <NUM> is made in an upper-left direction, thus indicating selection of a "<NUM>" key on a dialing keypad. If the user had subsequently tapped on the display <NUM>, an alternative entry may have been provided, or the system may have interpreted the tapping as entry of a "<NUM>" key, since the "<NUM>" key has no letters assigned to it on a standard keypad. A second dragging selection <NUM> is labeled "B," and represents a selection of the "<NUM>" key, while a third dragging selection <NUM>, labeled "C," represents a selection of the "<NUM>" key. Thus, the data entry area shows the numbers "<NUM>. " The user may continue entering numbers until the data they intended to enter is complete.

Display <NUM> shows an example of alphanumeric data entry. A virtual keypad <NUM> is shown in the middle of display <NUM> and represents a circular representation of the alphabetic keys on a standard Dvorak keyboard. Such a keyboard is designed to permit fast typing, so that frequently-used letters are placed along the middle row of the keyboard, and infrequently-used letters are placed in the upper and lower rows, and also at the edges. In a similar manner, the frequently-used letters from the middle row of a Dvorak keyboard are shown here in a middle circle of the virtual keypad <NUM>. Such letters may be selected by a user simply by dragging in a particular direction. The number of characters in a particular row may be selected so as to permit a system to discern a user's intended input, so that a user does not frequently drag in the direction of a letter they did not intend to submit.

Other, less frequently-used letters are shown in a second ring, where selection of such letters may require a dragging motion followed by a single tap (or a tap followed by a dragging motion), to indicate that the outer row of letters is the user's intent. An indicator <NUM> in the middle of the keypad <NUM> can provide a number of different indications. For example, here the indicator <NUM> shows the number "<NUM>", corresponding to the particular keypad shown here. A "<NUM>," in contrast, may represent that a different portion of the keypad is shown. For example, punctuation is not currently shown on display <NUM>, but a user may switch so that the keypad <NUM> changes to show, and accept input for, punctuation. Such a change may occur by a variety of gestures, such as a circular dragging, by tapping in all four corners or two opposed corners of the display <NUM>, or by swiping along the length of one side of the display (e.g., along the entire length or substantially the entire length). Certain frequently used punctuation, such as commas and periods, or other frequently-used keys may also be included in a main ring of a virtual keyboard.

Alternatively, a user intent to enter a character that is on an outer ring may be inferred from user actions during the entry of dragging operation. In one example, the dragging operation may contain multiple segments, where the first segment indicates a direction, and a second segment indicates which row of a multi-row (or multi-circle) keypad to select. For example, a user may simply drag in a straight line to indicate entry from an inner circle, but may double-back (or hook to the left or right) before lifting their finger to indicate an intent to enter a character from an outer circle.

As with the other displays, display <NUM> shows a data entry box <NUM> to provide visual feedback of characters that have been entered, or at least registered by the user's device. Such characters may also be spoken audibly to the user as each character is registered.

Three different dragging entries, labeled A to C and numbered <NUM>, <NUM>, and <NUM>, respectively, correspond to the three letters shown in the data entry box <NUM>. Dragging entry <NUM> is a complex entry because it involves an initial tap, and two entry segments that are separated by a change in direction. The initial tap is shown by a dotted circle and may be employed by a user to indicate that the upcoming dragging motion relates to the outer circle rather than the inner circle. A hook 142a in the entry may represent a variety of intents. For example, the hook may indicate which circle the user would like the application to select a letter from. The right hook shown here may represent the inner circle.

The hook may also represent a mid-course correction by the user. For example, as the user dragged upward, their device may have interpreted the input as a "p" rather than a "q" and may have spoken the letter "p" as soon as the entry was sufficient to indicate such an intent. Upon hearing the device say "p," the user may have been alerted that the device was interpreting their intent in an improper manner, and may have turned slightly to the right to indicate that they intended to select the key to the right (or clockwise of) the "p" - i.e., the "q" key.

Subsequent entries <NUM> and <NUM> indicate intents to enter the letters "u" and "i. " The user's intent to select from the inner circle may be inferred from the lack of a tapping input preceding or following the dragging entries.

The device may also make intelligent contextual selections from user inputs. For example, a disambiguation dictionary may be used to determine which of the two rows of characters in display <NUM> is intended, based on the frequency with which the particular letters appear together in ordinary usage, much like typical text entry completion methods work to disambiguate text entry. For example, entry B may represent, ambiguously, the entry of a "u" or a "b", and reference to a disambiguation dictionary may indicate that the letters "qu" are used together much more often than the letters "qb," so that the system will disambiguate the entry to represent the letter "u.

<FIG> is a block diagram of a system <NUM> for providing quick touchscreen user input. In general, the system is represented by a mobile device <NUM>, such as a smart phone, having a touchscreen user interface <NUM>. In addition, the device <NUM> may have alternative input mechanisms, such as a clickable trackball <NUM> and other selectable buttons, in addition to a physical keyboard that can be uncovered by sliding the display outward.

The device <NUM> may communicate via a wireless interface <NUM>, through a network <NUM> such as the internet and/or a cellular network, with servers <NUM>. For example, the device <NUM> may carry telephone calls through the telephone network or using VOIP technologies in familiar manners. Also, the device <NUM> may transmit regular data over the internet, such as in the form of HTTP requests directed at particular web sites, and may receive responses, such as in the form of mark-up code for generating web pages, as media files, as electronic messages, or in other forms.

A number of components running on one or more processors installed in the device <NUM> may enable a user to have simplified input on the touchscreen interface <NUM>. For example, an interface manager <NUM> may manage interaction with the touchscreen interface <NUM>, and may include a display manager <NUM> and an input manager <NUM>.

The display manager <NUM> may manage what information is shown to a user via interface <NUM>. For example, an operating system on the device <NUM> may employ display manager <NUM> to arbitrate access to the interface <NUM> for a number of applications <NUM> running on the device <NUM>. In one example, the device <NUM> may display a number of applications, each in its own window, and the display manager may control what portions of each application are shown on the interface <NUM>.

The input manager <NUM> may control data that is received from a user via the touchscreen <NUM> or other input mechanisms. For example, the input manager <NUM> may coordinate with the display manager <NUM> to identify where, on the display, a user is entering information so that that the device may understand the context of the input. In addition, the input manager <NUM> may determine which application or applications should be provided with the input. For example, when the input is provided within a text entry box of an active application, data entered in the box may be made available to that application. Likewise, applications may subscribe with the input manager so that they may be passed information entered by a user in appropriate circumstances. In one example, the input manager <NUM> may be programmed with an alternative input mechanism like those shown in <FIG> and may manage which application or applications are to receive information from the mechanism.

A simple gesture interface <NUM> may also be provided for similar purposes. In particular, the simple gesture interface <NUM> may be a program module or other form of application, such as a widget or gadget, that serves as an intermediary between other applications on a device and the interface manager <NUM>. Applications <NUM> may initially register with the simple gesture interface <NUM> when they are originally launched. In certain embodiments, the applications <NUM> may identify one or more directions for which it would like to receive inputs, e.g., by designating downward movement as an "<NUM>" for a telephone-based application. Using a known API, for example, the application may submit an array or other data structure of parameters of direction and input keys to be associated with user inputs in those directions. Where multi-direction inputs are to be interpreted, the applications <NUM> may submit information in a similar, but more expansive, manner.

The simple gesture interface <NUM> may then interact with the interface manager <NUM>, such as by registering itself with the interface manager <NUM>. When a user performs activities associated with such a registration, so as to indicate that the user would like to use the input mechanisms described here (e.g., by placing the device <NUM> in a pocket, by shaking the device <NUM> in a particular manner, or by dragging across display <NUM> in a particular manner), the interface manager <NUM> may report subsequent inputs by the user to the simple gesture interface <NUM>. For example, the interface manager may report the X and Y coordinates of each line traced by a user or of points along a curve or other pattern traced by the user. The interface manager <NUM> may also report if the user entered any taps and where those taps occurred on the display <NUM>.

The simple gesture interface <NUM> may then interpret such input and report it in an appropriate manner to the relevant application or applications <NUM>. For example, the simple gesture interface <NUM> may report a direction of a dragging input and the occurrence of any taps relevant in time to the dragging, and the application may interpret such data. Alternatively, the simple gesture interface <NUM> may interpret the data in a greater manner, such as by correlating a certain dragging direction with a keystroke that was previously registered by the application as corresponding to the direction. The simple gesture interface <NUM> may then pass the keystroke to the application.

The simple gesture interface <NUM> may also reformat data in other manners. In one example, a dialer application may not have been written to work with the input mechanisms described here. Instead, the dialer may receive information about which objects (in the form of virtual telephone keypad buttons) have been pressed by a user. A simple gesture interface <NUM> may be programmed to reside in a communication channel between the interface manager <NUM> and the application, and may convert directional dragging inputs into the form of messages that the dialing application expects to see from the interface manager <NUM>. In this manner, the system <NUM> can provide a number of different manners in which to provide quick user touchscreen input to one or more applications running on device <NUM>.

Finally, a user data database <NUM> may store information about particular user preferences or parameters. For example, the database <NUM> may store an identifier of an application that is to receive input from the simple gesture interface <NUM> in various contexts. As one example, a dialer application may be set by default to receive such input when the input is made over an operating system desktop. The user data database <NUM> may also include information such as the type of virtual keypad a user prefers to have their inputs mapped to (e.g., QWERTY or Dvorak), and other relevant data needed to provide an alternative mechanism for providing input.

<FIG> is a flowchart of a process for receiving touchscreen user inputs. In general, the process involves the use of a gesture tracking program to determine the form of user inputs on a touchscreen interface, and then to convert those inputs into particular commands (e.g., key presses) to be executed on computing device.

At box <NUM>, the process begins by the initiation of a gesture tracker <NUM>. The gesture tracker may, for example, be a component of an operating system that launches when the device is powered up, and that runs in the background to provide for gesture-based input to the system. The tracker may alternatively be an application separate form the core operating system, such as a gadget or widget, that communicates with touchscreen managing components of an operating system.

At box <NUM>, the device receives a gesture input. Any contact with the touchscreen may initially be interpreted as a gesture input. Such an input may then go through a filtering process, such as by filtering out contacts that are too broad-based to be intentional inputs because they likely represent accidental contact with something other than a fingertip or stylus. Also, certain inputs may be filtered and passed to various different applications, where one of the applications processes gestures like those described above.

For gestures that are determined to relate to inputs that are to be judged by their direction of dragging relative to a base point, the direction of dragging may be computed at box <NUM>. In particular, endpoints for a dragging operation may be determined in a familiar manner, and the angle between the endpoints may be computed. The angle may be generalized, in addition, such as to a nearest round angle (e.g., the nearest <NUM> degrees) or nearest compass direction (e.g., one of the eight or sixteen main compass directions). Other representations for an angle or direction of a dragging input, or a pair of tapping inputs, may also be employed.

At box <NUM>, a command or key press that has previously been correlated with the direction may be assigned to the received gesture, and may be executed at box <NUM>. The command may be assigned, for example, by a helper application that is dedicated to receiving data on such user inputs and passing commands to associated applications. Also, applications themselves may correlate a direction with a command or keystroke.

<FIG> is a flowchart of a process for interpreting touchscreen inputs on behalf of applications for which a user makes the inputs. In general, the process involves tracking user inputs, determining which inputs can be passed literally to an application and which require interpretation, and interpreting the relevant inputs before passing them, as interpreted, to an application.

The process in this example begins at box <NUM>, where relevant applications are launched. The applications may include end-user application such as telephone dialers, chat programs, note taking programs, and the like. The application may also include intermediary applications, such as helper applications that work to translate inputs from a touchscreen for direct use by the end-user applications. The applications may perform relevant processes, and at some point, may await events triggered by a user. For example, an event manager may receive information about contacts made with a touchscreen and may alert one or more relevant applications. For example, at box <NUM>, a user touchscreen input is received. The input could take a variety of forms, such as one or more taps, and one or more dragging motions that may occur in straight lines, curves, or more complex shapes. At box <NUM>, the process first determines whether the input is in a form that needs to be interpreted. For example, if the input is a tap on a program object that is intended to receive user inputs, such as a selectable button, such an action may be reported directly to the application that generated the object (box <NUM>).

If the input needs to be interpreted, the process may determining whether it is a substantive input or an input to stop the acceptance of such interpreted inputs (box <NUM>). If it is the latter, a helper application that assists in interpreting inputs may be closed or disabled (box <NUM>). For example, a user may have previously had their mobile device in their pocket, and thus been using a helper application that would interpret their inputs that did not require visual attention, but may have then taken the device out. Such a user may be able to use a keyboard directly by tapping on the actual or literal objects such as individual keys on a keypad. As such, the user may take an action to move the helper application out of the way in such a situation.

If the interpreted input is not an ending input, then the input needs to be interpreted and passed to the relevant application (box <NUM>). For example, a user may have entered a dragging input in a particular direction on the touchscreen face, where the user understands a particular direction to represent a particular keypress. The correlation of that direction to the keypress may have been previously registered, and a look-up may be used to identify the keypress from the direction. An identifier for the keypress may thus be passé to the application, so that the application may register the intended user input.

In this manner, a computing device may be provided with a flexible system by which to aid various applications in receiving input from a user who cannot use their device in an originally-intended manner that requires visual contact with the device. Rather, each application (or the helper application itself) may define a number of alternative input gestures that do not require the level of visual attention required by the original gestures, and may receive notice that the original gestures were received, by the helper application that translates the alternative gestures to the results associated with the original gestures.

<FIG> is a swim lane diagram of a process by which a gesture tracking module interfaces between a computer application and a touchscreen. In general, the process is shown here to highlight one example by which various components may execute to receive user inputs where the user need not be looking at a display on which they entered the inputs. Other arrangements and cooperation between and among components may also be used.

The process begins at box <NUM>, where an application is launched. The application may take a variety of forms, such as a dialing application for a smart phone. Upon launching, the application may call a gesture tracking module (box <NUM>) for assistance in managing certain inputs such as directional dragging inputs on a touchscreen display. At box <NUM>, the gesture tracking module launches and registers its presence with an operating system, such as a touchscreen manager component of a device operating system that is responsible for reporting contact activity entered through the touchscreen. The touchscreen manager, at box <NUM>, then confirms back to the gesture tracking module, that it has recognized the latter's registration.

At box <NUM>, the touchscreen manager receives an input that is not of the type handled by the gesture tracking module. For example, the input may be in an area on the touchscreen that is outside an area registered to the gesture tracking module. Or the input may be a curved input or a tap, where the gesture tracking module registered to receive information only about substantially straight-lined dragging inputs. In such a situation, the touchscreen manager may look to registration information from various applications running on a device, and may report the input from the user to the appropriate application (boxes <NUM> and <NUM>). The relevant application can then respond to the received input, such as by responding to a button push on a virtual keypad on the touchscreen.

A subsequent user input is received at box <NUM> (and follow up inputs may be received repeatedly after the processing of other inputs). This subsequent user input is determined to be an input in a form that is relevant to the gesture tracking module, and thus is reported to that module at box <NUM>. For example, perhaps a user initially tapped a first digit on a telephone dialer, and then needed to glance away from their device, and was able to continue dialing using directional dragging motions. The gesture tracking module may then, based on information previously received from, or on behalf of, the application, convert the information from the touchscreen manager into a command or keypress for the application (box <NUM>).

The gesture tracking module then sends such information to the application, which may respond to the input at box <NUM>. In this example, the gesture tracking module may have been brought forward on the device via some particular action by the user, and at box <NUM> it is again minimized and moved out of the way. In other words, a visual representation of the module on the touchscreen may be removed, and no inputs on the touchscreen may be interpreted as being directed to the module - even if they would otherwise match a signature registered with the touchscreen manager by the gesture tracking module.

<FIG> shows an example of a generic computer device <NUM> and a generic mobile computer device <NUM>, which may be used with the techniques described here. Computing device <NUM> is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. Computing device <NUM> is intended to represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, and other similar computing devices.

Also, multiple computing devices <NUM> may be connected, with each device providing portions of the necessary operations (e.g., as a server bank, a group of blade servers, or a multiprocessor system).

The information carrier is a computer- or machine-readable medium, such as the memory <NUM>, the storage device <NUM>, memory on processor <NUM>, or a propagated signal.

The information carrier is a computer- or machine-readable medium, such as the memory <NUM>, expansion memory <NUM>, memory on processor <NUM>, or a propagated signal that may be received, for example, over transceiver <NUM> or external interface <NUM>.

A number of embodiments have been described. Nevertheless, it will be understood that various modifications may be made without departing from the scope of the invention. For example, much of this document has been described with respect to a telephone dialing application, but other forms of applications and keypad layouts may also be addressed, such as keypads involving graphical icons and macros, in addition to alphanumeric characters.

Claim 1:
A computer-implemented method comprising:
receiving data indicating a first user input gesture on a touchscreen (<NUM>) of a computing device (<NUM>), wherein the first user input gesture is a tapping input gesture to select a subset of keys of a reference keyboard, and wherein the indication of the first user input gesture is independent of a location of the tapping input gesture on the touchscreen;
receiving data indicating a second user input gesture on the touchscreen of the computing device, wherein the second user input gesture is a dragging input gesture, wherein the first input gesture is followed by the second input gesture;
identifying that the second input gesture dragged in a particular direction;
identifying, based on the first input gesture and the particular direction of the second input gesture, that the first and second user input gestures represent a first user selection, the first user selection being a selection that is associated with a first key that is located in the particular direction away from an area of the reference keyboard; and
providing data indicating the first user selection.