Input method application for a touch-sensitive user interface

In an aspect, a method includes the actions of displaying, at a touch-sensitive display, a plurality of characters that each represent a distinct consonant included in a language script; receiving, at the touch-sensitive display, a touch selection of a first character from the plurality of characters, the touch selection being received at a location on the touch-sensitive display at which the first character is displayed; and in response to receiving the touch selection: detecting a gesture input, selecting a vowel marker based at least in part on an association of the gesture input with one or more vowel markers; and displaying, at the touch-sensitive display, an updated representation of the first character based at least in part on the selected vowel marker. Other embodiments of this aspect include corresponding systems, apparatus, and computer programs, configured to perform the actions of the methods, encoded on computer storage devices.

TECHNICAL FIELD

This document relates to touch-sensitive user interfaces.

BACKGROUND

English and other Latin-based scripts define text based on an aggregation of individual building blocks (letters. In such scripts, each letter can have a number of modified states (e.g., capitalized, modified by an accent, etc.). Device keyboards designed to enable text entry in such languages are generally suited to stateful input methods based on a one-to-one mapping between a key press and an output. Such keyboards are generally implemented on touchscreen devices as input method editors (“IMEs”)—system components and/or programs that allow a user to enter input by interacting with a touch-based user interface that is output at a display device. For example, an on-screen/virtual, touch-based keyboard can substantially replicate the functionality of a physical keyboard (such as a typewriter keyboard). Other touchscreen IMEs are based on gesture input, in which an input gesture is mapped to a single output character. Such keyboards often have a static layout with keys divided into a first group of typing keys and a second group of state toggle keys. In such examples, the typing keys may each correspond to a single output character, while the state toggle keys (e.g., the “Caps Lock” key) can be configured to change the key layout and accordingly assign a different set of letters to the keyboard.

Because non-Latin-based languages often include more characters than can be included within the dimensions of a typical, physical QWERTY keyboard, an English-based computer keyboard is sometimes adapted to enable text input in such languages.

SUMMARY

This specification describes technologies relating to an input method editor for a touch-sensitive user interface.

One innovative aspect of the subject matter described in this document can be implemented as a method performed by data processing apparatus, the method including the actions of displaying, at a touch-sensitive display, a plurality of characters that each represent a distinct consonant included in a language script; receiving, at the touch-sensitive display, a touch selection of a first character from the plurality of characters, the touch selection being received at a location on the touch-sensitive display at which the first character is displayed; and in response to receiving the touch selection: detecting a gesture input, selecting a vowel marker based at least in part on an association of the gesture input with one or more vowel markers; and displaying, at the touch-sensitive display, an updated representation of the first character based at least in part on the selected vowel marker. Other embodiments of this aspect include corresponding systems, apparatus, and computer programs, configured to perform the actions of the methods, encoded on computer storage devices.

This, and other aspects, can include one or more of the following features. In response to detecting the gesture input, a selection of other characters displayed in the touch sensitive display can be precluded for the duration of the gesture input. It can be determined that the gesture input is dependent on the touch selection of the selected character while the touch selection is maintained during the gesture input. It can be determined that the gesture input is dependent on the touch selection of the selected character if the gesture input is detected within a predefined time after a cessation of the touch selection. In response to detecting the touch selection, a bounded region surrounding the selected character can be displayed in the touch-sensitive user interface. An image corresponding to the gesture input can be displayed within the bounded region. The association of gesture inputs with vowel markers can define association of gesture inputs with vowel markers for each consonant. The language script can be one selected from a group consisting of Korean, Arabic, and Hindi.

Another innovative aspect of the subject matter described in this document can be implemented as a method that includes the actions of displaying, at a touch-sensitive display, a plurality of characters that each represent a distinct consonant included in a language script; receiving, at the touch-sensitive display, a touch selection of a first character from the plurality of characters, the touch selection being received at a location on the touch-sensitive display at which the first character is displayed; detecting a gesture input that is dependent on the touch selection of the first character; selecting a vowel marker based at least in part on the association of the gesture input with one or more vowel markers; and displaying, at the touch-sensitive display, an updated representation of the first character based at least in part on the selected vowel marker. Other embodiments of this aspect include corresponding systems, apparatus, and computer programs, configured to perform the actions of the methods, encoded on computer storage devices.

Another innovative aspect of the subject matter described in this document can be implemented as a method that includes the actions of displaying, at a touch-sensitive display, a plurality of characters that each represent a distinct character included in a language script; detecting a touch selection of a first character displayed in the touch-sensitive display; in response to detecting the touch selection of the first character, designating the first character for association with subsequent inputs detected at the touch-sensitive display; detecting a gesture input at the touch-sensitive display, the gesture input representing an accent marker to be associated with the first character; in response to detecting the gesture input: identifying the accent marker represented by the gesture input, and displaying, in the touch-sensitive display, an output character representing a combination of the first character and the accent marker. Other embodiments of this aspect include corresponding systems, apparatus, and computer programs, configured to perform the actions of the methods, encoded on computer storage devices.

Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following potential advantages. The techniques described here can be implemented as an Input Method Editor (IME) that is contextual and adaptive, and which changes the user interface based on user input. In addition, the application that executes the techniques described here can predict an intended outcome by recognizing gesture inputs. The touch-sensitive user interface can serve as a viable alternative to keyboards for non-Latin languages that mimic the keyboards of Latin-based languages (for example, QWERTY keyboards). The user interface can be implemented as keyboard that is not crowded and that does not display characters in the display device that may be illegal according to rules of the language script. For example, for languages that have a phoneme structure of CVC (C is a consonant or is empty and V is a vowel or diphthong), an input of two initial consecutive vowels is illegal. The user interface described in this document can avoid a display of such illegal characters in the display device. More particularly, a system bandwidth that would otherwise be used to display such illegal characters can be re-purposed for legal input. In addition, using the user interface described in this document, a speed of user input can be increased and errors decreased as the characters in the user interface change in response to the context in a predictable fashion. Further, the adaptive system described in this document can provide feedback to a user, and hence serve as a teaching system or a tutorial or both.

DETAILED DESCRIPTION

FIGS. 1A-Dillustrate an example mobile device100. Among other components, the mobile device100includes a touch-sensitive display102in which is rendered a touch-sensitive user interface103. The mobile device100can include data processing apparatus that executes a computer application. The application can receive inputs from a user through the user interface103and can display outputs based on the inputs in the touch-sensitive display102. The mobile device100can be, for example, a personal digital assistant (PDA), a smart phone, a navigation system, a music player, a tablet computer, an e-book reader, or other type of computing device that utilizes a touch-sensitive display. The other components of the mobile device100can include, for example, a hardware controller (to turn the device on/off), speakers, and the like.

As used by this document, a “touch-sensitive user interface”103(or “user interface,” or “touchscreen,” or “touch screen display”) is an electronic visual display displayed on the touch sensitive display102and in conjunction with the display102detects the presence and location of a touch within the display area. Also, as used in this document, “a gesture input” is a form of motion input received by the user interface103and the display102that is interpreted by computer software instructions executed by the data processing apparatus. The touch, which defines a user input, can occur using a finger, hand, stylus, or other object. The touch can represent a single touch at a particular location on the touch-sensitive display102, or the touch can represent a gesture input, e.g., a motion or “swipe,” across multiple locations on the touch-sensitive display102, or combinations of such motions or swipes. The user can control the application executing on the mobile device100through touches and gesture inputs.

The application can be, for example, an input method editing application that enables a user to input characters of a language script (for example, a non-English language script such as Korean, Arabic, Hindi). As described below, the user can provide touch and gesture inputs in the display102, in response to which the application can display output characters in the touch-sensitive display102.

In some implementations, the application, when executed, displays characters that each represents a respective consonant of a language script, in the touch-sensitive user interface103. As shown inFIG. 1A, the application displays consonant characters of the Korean language including a first character104(), a second character106() and a third character108(). The application displays each character of the language script in a corresponding selectable object that a user can select by a touch selection. In some implementations, a touch selection is a prerequisite to a gesture input, and is a selection of a particular object displayed in the touch-sensitive display in response to a touch at a location on the touch-sensitive display at which the object is displayed. As will be described in more detail below, the gesture input is dependent on the touch selection.

The application can display the characters in one or more rows and columns. In addition to the characters, the application can display additional selectable objects representing for example, a space bar, a backspace key, a shift key, and one or more keys the selections of which will cause the application to display additional characters in the user interface103or in the display device102or both.

The application can receive a touch selection of one of the characters, for example, the third character108(). For example, a user of the mobile device100can touch a location on the touch-sensitive display102at which the third character108() is displayed. In response to receiving the touch selection, the application can designate the character that was touched, i.e., the third character108(), as a selected character. Designating the touched character as the selected character results in subsequent user inputs received through the touch-sensitive display102being associated with the selected character. For example, as described below, subsequent user inputs that are associated with vowel markers will be associated with the third character108().

In some implementations, the application can designate the character that was touched as a selected character by displaying a bounded region110surrounding the selected character in the touch-sensitive display102. For example, the application can display a rectangular region surrounding the third character108(), and, in certain instances, display the third character108() at the center of the rectangular region. For example, a color of the bounded region110can be different from the remainder of the display102and the user interface103. Alternatively, or in addition, the bounded region110can have a translucent appearance.

In general, the application can display the bounded region110such that the region110is visually discernible from a remainder of the touch-sensitive display102and the touch-sensitive user interface103. In the example shown inFIG. 1B, the bounded region110occupies less than an area occupied by the display102and the user interface103. In some implementations, the application can display the bounded region110to occupy the entire area otherwise occupied by the display102and the user interface103.

In some implementations, in addition to displaying the bounded region110, the application can display the selected character, i.e., the third character108(), in a visually discernible manner, either as originally rendered, as depicted inFIG. 1B, or in increased size (not shown). For example, the application can display a region112that is within the bounded region110and that surrounds the third character108() in a color that is different from a color of the remainder of the bounded region. Alternatively, or in addition, the application can display the third character108() in a color that is different from a color of the bounded region.

By displaying the bounded region110, the application notifies the user that the touched character has been designated as the selected character, and that subsequent inputs that the user will provide through the touch-sensitive display102will be associated with the selected character. In other words, once the application designates the touched character as the designated (selected) character, the application precludes selection of other characters displayed in the display102for the duration of subsequent inputs associated with the touched character. In addition, the application can present the user with a bounded region110within which all or substantial portions of subsequent inputs (for example, gesture inputs) are to be provided.

In response to receiving the touch selection and designating the touched character as the designated character, the application monitors for a gesture input that is dependent on the touch selection of the selected character. As described above, a gesture input can be a motion input (for example, a swipe) received within the user interface103. For example, the user can provide a gesture input using a finger by swiping the finger within the bounded region110. Because the user first provides the touch selection and then provides the gesture input, the latter is dependent on the former.

In some implementations, the gesture input is dependent on the touch selection of the selected character while the touch selection is maintained during the gesture input. For example, if the application receives the touch selection of the third character108() and determines that the touch with the display102has been maintained after the touch selection, then the application associates any gesture input received for the duration that the touch is maintained is to be associated with the third character108. In variations of these implementations, if the application detects a cessation of the touch (for example, because the user has removed his/her finger from the display102), then the application will not associate a subsequent gesture input with the third character108. To notify the user that a subsequent gesture input will not be associated with the third character108, the application can cease to display the bounded region110in the user interface103.

In alternative implementations, the gesture input is dependent on the touch selection of the selected character if the gesture input is detected during the touch or within a predefined time after the cessation of the touch selection or subsequent gesture input is detected and/or ceased. For example, the application can receive the touch selection of the third character108() and can determine a cessation of touch (for example, because the user has removed his/her finger from the display102). Despite the cessation of touch, the application can associate a subsequently received gesture input with the third character108() as long as the gesture input is received within a pre-defined duration after the touch selection was received. This allows for gesture inputs that include multiple and separate gestures, e.g., a circular gesture input followed by a separate swipe across the circle. If no gesture input is received within the pre-defined duration, then a subsequent gesture input will not be associated with the selected character. The predefined duration can be, for example, 200 milliseconds, or some other duration.

When a subsequent gesture input is not associated with a selected character, for example, the application can treat the beginning of the subsequent gesture input as a new touch selection. To notify the user that a subsequent gesture input will not be associated with the third character108(), the application can cease to display the bounded region110in the user interface103.

In some implementations, the bounded region110is for notification only, and the gesture input is not required to be confined to the bounded region110. For example, even though a bounded region is shown, the entire display102can be used to receive the gesture input.

In some implementations, the gesture input can be associated with vowel. In response to monitoring the gesture input, the application can access an association of gesture inputs with vowel markers. The gesture input can include touching different areas of the screen, can be continuous or discontinuous (i.e., be composed of multiple gestures contact with the touch surface is not continuous) or can be combinations of them. The association of gesture inputs and vowel markers can be stored, for example, as computer software instructions, in the mobile device100. The associations can be defined by the software developer and stored in data associating the gesture inputs with vowel markers for each consonant, and particular gesture inputs can be selected to correspond with particular vowel markers.

For each consonant, each gesture input associated with each vowel marker for the consonant is different from each other gesture input associated with each other vowel marker for the consonant. However, a vowel marker for a first consonant and a vowel marker for a second consonant can be associated with a same gesture input. For example, a gesture input of a “u” shape can be associated with a first vowel marker, a gesture input of a “z” shape can be associated with a second vowel marker, and so on. Additionally, for a first consonant, a gesture input of a “u” shape can be associated with a particular vowel marker, and for a second, different consonant, the gesture input of the “u” shape can be associated with the same particular vowel marker. Finally, a gesture input can also be associated with different markers for different consonants. For example, for a first consonant, a gesture input of a “u” shape can be associated with a first vowel marker, and for a second, different consonant, the gesture input of the “u” shape can be associated with a second, different vowel marker. The associations can be stored, for example, in a look-up table that receives data indicating the selected character and gesture input as input.

Upon accessing the associations, the application can select a vowel marker that is associated with the gesture input that is detected, and select a vowel marker. In addition, the application can display a representation of the selected character as modified by the vowel marker in the user interface103.

For example, the gesture input shown inFIG. 1Cis associated with the Korean vowel,, which modifies the third character108() resulting in the character116. Thus, the third character108() represents a base character and the vowel,, represents a modifier that modifies the base character to result in an output character, for example, the character116shown inFIG. 1C. The application displays the character116in the display102or in the user interface103. As shown inFIG. 1C, as the user provides the gesture input (i.e., swipes the finger), the application can display an image114that corresponds to the gesture input in the display102, for example, within the bounded region110. For example, the gesture input includes a touch selection of any location within the bounded region followed by a swipe in a vertically upward direction and then by a cessation of the gesture input (for example, a removal of the finger from the display102). In response to the gesture input, the application displays the image114as a vertical line that tracks the swipe, i.e., originating from the location of the touch selection, extending in the direction of the swipe, and terminating at the location at which the gesture input ceased.

FIG. 1Dshows another example in which the gesture input is associated with the Korean vowel,, which modified the third character108() resulting in the character120shown inFIG. 1D, which the application displays in the display102or in the user interface103.

The representation can be generated based on the rules of the language script of the characters. For example, the rules of the language script can define a representation of a consonant represented by the selected character and the vowel marker received through the gesture input. In particular, the rules can define associations of vowel markers for each consonant. The rules can also be stored, for example, as computer software instructions in the mobile device100. The application can execute the rules to determine the representation of the selected character as modified by the vowel marker, and to display the representation in the user interface103. The language script can be one in which vowels modify the consonants to which the vowels are attached. For example, the language script can be any non-Latin language script, and, in particular, can be selected from a group consisting of Korean, Arabic, and Hindi.

In some implementations, the application can display the image114to be visually discernible from the bounded region110. For example, the application can display the image114in a color that is different from that of the bounded region. In addition, the application can display the color as a tapered gradient in a direction of the gesture input. For example, the color of the vertical line shown inFIG. 1Ccan be most intense at a location of the touch selection, can be relatively less intense at a location at which the finger swipe ceased, and can have a tapered gradient of color in between. In general, a gradient of color of the image114from most intense to least intense can represent a direction of the gesture input.

FIGS. 2A-Dillustrate portions of a touch-sensitive user interface200receiving gesture inputs related to Korean characters. The Korean (Hangul) alphabet is written in syllabic blocks that include letters (jamo). Each block includes at least one consonant and at least one vowel. The application can display the fourteen simple consonants as base characters in the touch-sensitive user interface. Five of these consonants can be doubled to form the five doubled (or tense) consonants by receiving touch selections, for example, by taping. In Hangul, each vowel is written as a vertical or horizontal main stroke with zero, one, or two smaller strokes written perpendicularly to the main stroke. In the application, the characters that represent consonants are modified by gesture inputs that resemble the smaller strokes. The gesture inputs disambiguate the direction of the main stroke.

For example, the gesture input represented by the image204represents the vowel marker,, that modifies the character202() resulting in character206. Similarly, the gesture input represented by the image208represents the vowel marker,, that modifies the character202() resulting in character210. In addition, the gesture input represented by the image212represents the vowel marker,, that modifies the character202() resulting in character214. Also, the gesture input represented by the image212represents the vowel marker,, that modifies the character202() resulting in character218.

FIGS. 3A-Dillustrate portions of a touch-sensitive user interface300receiving gesture inputs related to Arabic characters. In the Perso-Arabic script, letters can be grouped by shape, such that letters within each group are distinguished from one another by diacritical marks or “dots.” The base characters are then the shape-groups, and individual characters can be formed by using a gesture input which begins with the base character corresponding to the group. The characters shown inFIGS. 3A-Dbelong to the he-jimi group, which consists of four letters distinguished by corresponding dots. Using the application, each character can be entered by making a gesture beginning on the he-jimi character that corresponds to the location and number of the dots. Selecting the he-jimi character, which itself has no dots, without further without additional gesturing can designate the he-jimi group as the selected characters.

For example, the application displays the character302() in the user interface300. In the absence of any gesture input, the character302() represents the base character304(). The gesture input represented by the image306represents vowel marker consisting of one diacritic dot, that modifies the character302() resulting in character308(). Similarly, the gesture input represented by the image310represents another diacritic dot, that modifies the character302() resulting in character308(). In addition, the gesture input represented by the image314represents three diacritic dots, that modifies the character302() resulting in character316().

Examples of gesture inputs and associations with vowel markers have been described with respect to Korean and Perso-Arabic scripts. However, the same principals can be applied to other languages and scripts, such as Non-Latin scripts, for example, Indic scripts.

FIG. 4illustrates an example of a system400implementing the application for a touch-sensitive user interface. The system400can include data processing apparatus402, a computer-readable medium404, and a display device406. The computer-readable medium404can store computer software instructions executable by the data processing apparatus402to cause the application described above in the display device406. The example system400can be implemented on a mobile device, as described above. Alternatively, the system can be implemented in any computer, for example, a desktop computer, a laptop computer, and the like. For example, in a desktop computer, the data processing apparatus and the computer-readable medium can be implemented in a central processing unit that is separate and distinct from a display device. The display device can be a cathode ray tube (CRT) or liquid crystal display (LCD) monitor that includes a touch-sensitive user interface. Examples of processes implemented by the system400are described with reference toFIGS. 5 and 6.

FIG. 5is a flowchart showing an example of a process500for displaying characters of a language script. The process500displays characters that each represents a respective consonant of a language script at502. The process500receives a touch selection of one of the characters at504. The process500detects a gesture input that is dependent on the touch selection of the selected character at506. The process500accesses an association of gesture inputs with vowel markers at508. The process500selects a vowel marker that is associated with the gesture input that is monitored at510. The process500displays a representation of the character as modified by the vowel marker at512.

FIG. 6is a flowchart showing an example of a process600for displaying characters of a language script. The process600displays characters that each represents a respective consonant of a language script602. The process600receives a touch selection of one of the characters at604. The process600designates the selected character as one to which subsequent inputs are associated at606. The process600detects a gesture input representing a vowel marker to be associated with the selected character at608. The process600identifies the vowel marker represented by the gesture input at610. The process600displays an output character representing a combination of the selected character and the vowel marker at612.

Although the examples above are described with reference to consonants and vowel markers, the processes can be applied to associations of characters and accent markers, and are not necessarily limited to consonants and vowel markers.

Embodiments of the subject matter and the operations described in this specification can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions, encoded on computer storage medium for execution by, or to control the operation of, data processing apparatus. Alternatively or in addition, the program instructions can be encoded on an artificially-generated propagated signal, for example, a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.

To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, for example, a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user. All or portions of the display device can include a touch screen or any touch-sensitive user interface. The computer can also include a keyboard and a pointing device, for example, a mouse or a trackball or touch pad, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, for example, visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user's client device in response to requests received from the web browser.

The gesture inputs described above are exemplary gesture inputs. As alternatives or in addition to the gesture inputs described above and illustrated in the figures, other gesture inputs can be associated with vowel markers. In some implementations, instead of a touch selection, a first gesture input can be associated with a character that represents a consonant and a second subsequent gesture input can be associated with a vowel marker to be associated with the character. In some implementations, instead of or in addition to gesture inputs, touch selections and/or selections of hardware components of the mobile device can be associated with the vowel markers. Similarly, instead of touch selections, selections of hardware components on the mobile device can be associated with the characters that represent the consonants.