Proactive input method editor switching

Method and apparatus for proactive IME switching are provided. Embodiments receive a first input using a first input method editor (IME) of a plurality of IMEs. A first string corresponding to the first input is determined using the first IME. Upon receiving an indication that the first string is incorrect, a second IME of the plurality of IMEs is selected based on a mapping between a context of the first input and the second IME, wherein the context of the first input comprises: a first application that the first input was intended for; the first input; and a series of strings determined just before receiving the first input. Finally, a second string for a second input is determined using the second IME.

BACKGROUND

The present invention relates to the field of input devices, and more specifically, to automatically and proactively switching between input method editors.

Written languages using large numbers of characters that represent words or morphemes, sometimes referred to as ideographic characters, face complication when trying to provide input from standard computing input devices, such as a standard keyboard. Some languages, such as Mandarin Chinese, may have thousands of characters that can be used in written expression. Phonetic Romanization has been used in which the phonetic pronunciation of the character is represented in Romanized or Latinized characters that are produced using the American Standard Code for Information Interchange (ASCII). Romanization or Latinization is the conversion of writing from a different writing system to the Roman (Latin) script. Methods of Romanization of written text include transliteration, converting text from one written type of script to another, and transcription, which is the systematic representation of language in written form. For example, a given input could be transcribed by a computing device by generating a string corresponding to the input. Transcription methods include the first transcription (such as phonetic, radical, image (handwriting), audio (speech) transcripts), which associate with well-defined and mapped meaningful ideographic symbol (second transcription). Any of above transcription mapping methods can be developed as an input method. Users can input the first transcription into a special input method program to produce a second transcription. For instance, pinyin input method is one of most popular input methods for generating Chinese characters using a traditional Latin keyboard.

For Mandarin Chinese, for example, pinyin is a phonetic system for transcribing the sound of Chinese characters into Romanized script. Pinyin enables a user to identify a Chinese character by the phonetic pronunciation of the Chinese character, whose sound is described by the transcribed Romanized script. A pinyin-based input method editor allows users input Romanized characters to generate Chinese characters. Generally, numerous input method editors may exist, and a user may install multiple distinct input method editors for Chinese input.

SUMMARY

According to one embodiment of the present invention, a first input is received to be transcribed using a first input method editor (IME) of a plurality of IMEs. Next, a first transcription is determined for the first input using the first IME. Upon receiving an indication that the first transcription is incorrect, a second IME of the plurality of IMEs is selected based on a mapping between a context of the first input and the second IME, wherein the context of the first input comprises a first application that the first input was intended for, the first input, and a series of transcriptions that were determined just before receiving the first input. Finally, a second transcription is determined for a second input using the second IME.

DETAILED DESCRIPTION

Input method editors (IMEs) are programs or applications that enable users to input complex characters to a computing system even when the desired character does not have a corresponding key on the user interface. For example, Chinese, Japanese, Korean, Vietnamese (CJKV), and many other complex languages each contain many thousands of unique characters, and it is not feasible to provide a dedicated key for each character. Users who wish to input such complex characters, therefore, must rely on one or more IMEs to transcribe some other kind of input into the desired characters. There are a large number of IMEs today, many relying on QWERTY keyboards, audio input, handwriting analysis, or some other form of input. As is discussed briefly below, each type of IME has advantages and disadvantages, so most users routinely utilize multiple IMEs on a single device, and switch between them frequently in order to facilitate transcription. This manual IME switching is time-consuming and frustrating, and an automated process for selecting a more advantageous IME is desired.

One popular type of IME is a handwriting IME that uses handwriting recognition to transcribe a user's input to the corresponding character(s). These handwriting IMEs may scan writing that a user has previously written, but more frequently they provide a space for the user to write directly on an input section of a device and the input is transcribed in real time. This input section may be a portion of a touch screen, a dedicated device used only for handwriting input, or any other suitable method of inputting handwriting. Handwriting IMEs are convenient in that they often require no extra hardware that the user does not already possess, and they are relatively quick, especially if the user has a stylus. Unfortunately, handwriting IMEs can be difficult to use as well. If a particular user's handwriting is sloppy or has unique quirks, many handwriting IMEs will struggle to provide a correct transcription. Similarly, handwriting IMEs can be difficult to use if the user is moving or being jostled, for instance on a bus or train. Often, a handwriting IME will work well for many characters but will frequently fail to provide an accurate transcription for more complex characters. Thus, users frequently switch to another type of IME when they know they need to input a character that the handwriting IME cannot recognize.

Another popular type of IME is an audio IME. Audio IMEs use a microphone to receive verbal input from a user, and transcribe the spoken speech into text. Speech recognition is useful to quickly transcribe input, but can also be undesirable to use. Using an audio IME requires the user to speak out loud, which can compromise privacy and annoy others if the user is in a public setting. Additionally, the speech recognition employed by audio IMEs may frequently fail to accurately transcribe input if the user has an accent. Similarly, audio IMEs are difficult to use if there is ambient noise in the area. As above, audio IMEs may frequently work well for some characters while repeatedly failing to recognize other characters or phrases, often because of accents and unique quirks of each user. Thus, users frequently switch between audio IMEs and other types of IMEs.

A third popular type of IME is a romanization IME that allows a user to input Latin characters (often English characters on a QWERTY keyboard or a T9-style keyboard) for transcription. One popular example is a pinyin IME for Chinese transcription, though similar IMEs exist for other languages as well. For purposes of brevity, these IMEs are described using pinyin IMEs as representative. Pinyin IMEs and other related IMEs rely on romanization of the target language. For example, the pinyin method of romanization is an established system of writing Chinese characters using the Roman (e.g., Latin or English) alphabet and frequently uses a standard QWERTY keyboard. Pinyin approximates the pronunciation of Chinese characters and phrases using English/Latin characters. For example, the Chinese character “” meaning “photograph” can be written as “zhào piàn” in pinyin. Pinyin IMEs are advantageous in that they require nothing more than a standard keyboard and input can be provided silently and relatively quickly.

A key disadvantage to such systems, however, is that the user must know how to accurately pronounce the desired character and know how to describe that pronunciation using Latin letters. Different individuals may frequently pronounce the same phrase or character quite differently, which leads to obvious problems using a pinyin IME. Additionally, many characters are pronounced very similarly, and it can be tedious to go through dozens of transcription suggestions to arrive at the desired character. In particular, one issue in the use of pinyin or phonemic based scripts, is the number of homonyms that may exist for the same Latinized or Romanized (hereafter referred to as Romanized or Romanization), script characters. Many phonetic input systems use a multiple-choice method to address this homonym issue, presenting a candidate list of possible characters with the same pronunciation for each syllable entered, for selection by a user. For example, the pronunciation of “yi” in Mandarin Chinese can correspond to over 100 Chinese characters, which would generate a large list of characters from which a user must choose the most appropriate for the intended use. For these reasons, many users switch from pinyin IMEs to other IMEs to transcribe particular phrases or characters.

Another popular option for users is a stroke IME that is based on the structure of the desired character. Often, the order of strokes that a user uses when writing the character by hand is used. Common Chinese stroke IMEs use five keys for entry, one key for each type of stroke. These IMEs are advantages in that they do not rely on pronunciation, but can be difficult to use, especially for characters with similar appearances. There are many other types of IMEs, and many different specific IMEs within each type. As is clear, users frequently desire to use multiple IMEs for input, switching manually between them frequently based on the character that they wish to enter.

Various IMEs can be provided to a user in a variety of ways. Some transcription can be provided locally on the user's device, but may also be provided as a service on a remote device, e.g., in the cloud. Frequently, IMEs are used on mobile devices such as cell phones, but they are also useful (and often required) for other devices, including a laptop, desktop computer, wearable device, or any other device which receives input from a user. A single user may have multiple IMEs for multiple languages, but users also frequently install multiple IMEs for a single language in order to facilitate transcription. Because each IME has advantages and disadvantages, users frequently switch to a different IME for the same target language manually for a single or few characters, only to switch back to their preferred IME immediately afterwards. Such manual switching is time-consuming and frustrating. Embodiments of the present disclosure generally utilize context-sensitive intelligent switching between IMEs based on prior manual IME switches, transcription errors, predefined rules, and user customization, even if the multiple IMEs provide transcription for a single target language.

FIG. 1is a block diagram illustrating a system for intelligently switching IMEs, based on an embodiment of the present disclosure. As shown inFIG. 1, the system100includes a User System101and a Remote System151connected by a Network130. In some embodiments, all transcription is performed locally and there is no need for a Remote System151. In other embodiments, one or more of the IMEs that a user wishes to access are provided as a service on Remote Server151. Generally, User System101may send input provided by a user over Network130to Remote System151where it is transcribed. The resulting transcription is then sent back over Network130to User System101. As shown, User System101contains a Computer Processor102, Storage media103, Memory104, and a Network Interface111. Computer Processor102may be any processor capable of performing the functions described herein. The User System101may connect to the Network130using the Network Interface111. Furthermore, as will be understood by one of ordinary skill in the art, any computer system capable of performing the functions described herein may be used.

In the pictured embodiment, Memory104contains an Operating System105, a Client Application106, an Error Analysis Module107, a plurality of IME Switch Logs108, a User Profile109, and one or more Local IME(s)110. Of course, in some embodiments all IMEs that the user accesses are located remotely and no Local IME(s)109are present. Although Memory104is shown as a single entity, Memory104may include one or more memory devices having blocks of memory associated with physical addresses, such as random access memory (RAM), read only memory (ROM), flash memory, or other types of volatile and/or non-volatile memory. The Operating System105may be any operating system capable of performing the functions described herein.

The Client Application106is any application requiring input from a user, where the user wishes to use an IME to provide said input. The Error Analysis Module107monitors manual IME switches and input from the user to Client Application106, determines whether an IME switch is required, and switches to a second IME when required, as will be discussed in greater detail below. The Error Analysis Module107also records transcription errors in the IME Switch Logs108, as will also be discussed in more detail below. Additionally, in some embodiments Error Analysis Module107also generates IME Switch Logs108when a user manually switches to a second IME, even when no transcription error was made. The IME Switch Logs108may include a reference to the IME that was selected by the user, the input that triggered the faulty transcription, the context of the input, a reference to the Client Application106, and any other useful data as will be described in more detail below. The Local IME(s)110may be any IME capable of transcribing any type of user input, including an audio IME, pinyin IME, stroke IME, and handwriting IME. User Profile109generally contains information about user preferences and IME configurations, as will be discussed in more detail below.

The Remote System151contains a Computer Processor152, Storage media153, Memory154, and a Network Interface161. Computer Processor152may be any processor capable of performing the functions described herein. The Remote System151may connect to the Network130using the Network Interface161. Furthermore, as will be understood by one of ordinary skill in the art, any computer system capable of performing the functions described herein may be used.

In the pictured embodiment, Memory154contains an Operating System155and one or more Remote IME(s)159. Although Memory154is shown as a single entity, Memory104may include one or more memory devices having blocks of memory associated with physical addresses, such as random access memory (RAM), read only memory (ROM), flash memory, or other types of volatile and/or non-volatile memory. The Operating System155may be any operating system capable of performing the functions described herein. The Remote System151is generally configured to receive input from Network130, transcribe the input using the one or more Remote IME(s)159, and return the transcription back to the device that sent the input.

FIG. 2illustrates a flow diagram for a method200of one embodiment of the present disclosure. The method200begins at block201, where the system receives a first input to be processed using a first input method editor (IME) of a plurality of IMEs. Then, at block202, the system determines a first string corresponding to the first input using the first IME. The method200next proceeds to block203, where, upon receiving an indication that the first string is incorrect, the system, e.g., the Error Analysis Module107, selects a second IME of the plurality of IMEs based on a mapping between a context of the first input and the second IME, wherein the context of the first input comprises: a first application that the first input was intended for; the first input; and a series of strings determined just before receiving the first input. Finally, the method200concludes at block204, where the system determines a second string corresponding to a second input using the second IME.

FIG. 3is a flow chart illustrating one embodiment of the present disclosure. As illustrated,FIG. 3shows a method300for intelligent automatic IME switching based on context. The method300begins at block301, where the system receives input from a user. The input is to be transcribed using an IME, and may be audio, a handwriting image, text, a series of strokes, or any other appropriate input that can be transcribed. At block302, the system optionally determines whether the desired IME is located locally or on a remote system, e.g. Remote System151. As discussed herein, a particular IME may be executed entirely on a user's local device, entirely on a remote server, e.g., in the cloud, or may be split between local and remote execution. AsFIG. 3illustrates, if the desired IME service is provided on a remote system, the input is sent to the remote system at block303. The remote system may be communicatively coupled to the local system in any manner sufficient to perform the functions disclosed herein. Additionally, the remote IME service may in fact be located on the same device as the local system, e.g., in another virtual machine on the same physical device. At block304, the local system receives a suggested transcription from the remote system, and proceeds with the method300.

Alternatively, if the system determines at block302that the desired transcription can be executed locally, the method300proceeds to block305where a suggested transcription is generated based on the input using the local IME. As illustrated, the method300proceeds to block306where the system determines whether the suggested transcription is correct. In some embodiments, Error Analysis Module107is configured to make the determination regarding the correctness of the transcription. In an embodiment, the suggested transcription is provided to the inputting user and the user may manually or verbally confirm the transcription as correct or discard the transcription as incorrect. Additionally or alternatively, the system or Error Analysis Module107may be configured to detect that a user confirmed a transcription but immediately deleted it after inserting it. This may indicate that the transcription was not correct, but the user accidentally or intentionally inserted it only to delete it and switch to a different IME for entering the correct character. If the transcription was correct, the method300returns to block301where the user provides input for further transcription. Alternatively, if the system or Error Analysis Module107determines that the transcription was incorrect, the method300proceeds to block307.

At block307, the Error Analysis Module107compares the context of the input to prior IME switching records and/or user profiles. In some embodiments, the input context and prior IME switch records include the correct transcriptions that were entered just before the input which led to the incorrect transcription. For example, the Error Analysis Module107may be configured to determine whether a series of characters entered just before the current error corresponds to a series of characters leading up to a prior error and manual switch. In another embodiment, the input context includes the input itself, and the Error Analysis Module107compares the input that led to the current failed transcription to prior input. For example, the Error Analysis Module107may be configured to determine that a handwriting image is similar to some previous handwriting image that is stored in a prior IME switch record. In another embodiment, the input context includes the application that the user is interacting with when the transcription error occurs. For example, the Error Analysis Module107may be configured to compare the current context to prior IME switches that occurred when using the same application as the current input.

In some embodiments, the user can select what is and is not stored in the prior IME switch records and compared to the present input context. For example, a user may not care whether the application being used is the same, or may want error records from other applications to be used, but with a discounted weight. Alternatively, a user may be uncomfortable with storing the transcriptions leading up to prior errors due to privacy concerns, and may configure the system to only consider the current input itself. These preferences can be stored in a configurable user profile, and can be taken into account by Error Analysis Module107at block307. In some embodiments, the user profile may also contain information about preferred or default IMEs for the user. The default IME may be specific to each application, or may be the same across the entire device. In another embodiment, the user profile includes a second IME that is preferred if a switch is required. For example, a user may indicate that their preferred or default IME is a handwriting IME, but that if a switch is required, it is preferred to switch to a stroke IME. Additionally, in some embodiments the user may indicate that the system should never switch to a different IME when certain applications are being used.

At block308, after comparing the input context to prior IME switches, the Error Analysis Module107determines whether or not to switch to a second IME. If the Error Analysis Module107determines that another IME is better suited to transcribe the input, e.g., because the input context is similar to at least one prior IME switch, the method300proceeds to block309where the new IME is selected for future transcriptions. In some embodiments, the switch is accomplished by updating a pointer which points to the IME that is to be used for a given input. For example, the IME pointer may indicate a section of disk space or a location on a network where the IME service is located, and block309may comprise updating that pointer to point to a different section. After switching to a second IME, the method300returns to block301, where the user may attempt to enter the desired character again.

In some embodiments, the Error Analysis Module107is configured to immediately switch back to the first IME after the user enters the desired character. For example, a user may indicate that the suggested transcription is wrong when using a handwriting IME. The Error Analysis Module107, in response, may automatically switch to an audio IME. In some embodiments, the system continues to use the audio IME until the user switches back manually, or until another incorrect transcription occurs. In some embodiments, however, the Error Analysis Module107is configured to use the audio IME only for transcription of a single character or a small number of characters. The Error Analysis Module107may then automatically switch back to the original IME so that the user may continue entering characters using their preferred IME.

In some embodiments, the Error Analysis Module107is configured to continue to monitor the user's actions even after determining at block308to switch to a different IME. For example, if, after the automatic switch, the user does not enter new input but instead manually switches back to the first IME, the Error Analysis Module107may be configured to record an indication of that as well. Additionally or alternatively, the Error Analysis Module107may be configured to weaken the mapping that led to the automatic switch. For example, the Error Analysis Module107may be less likely to make the same automatic switch based on the same context in the future, because the user was not pleased with the result. In this way, the Error Analysis Module107evolves and learns from the user's patterns and provides intelligent switching decisions based on the context of the input and the user's preferences.

If, at block308, the Error Analysis Module107decides not to switch IMEs, e.g., because the input context is not similar to any prior IME switch records, the method300proceeds to block310where the Error Analysis Module107monitors the system to determine whether the user has manually switched to a second IME in response to the failed transcription. If the user goes on to provide input for transcription and does not switch to a second IME, the method300returns to block301. For example, the transcription may have been inaccurate because of loud background noises, mistyping by the user, or any other reason other than inadequacy of the IME. Alternatively, if the user manually switches to a second IME, the method300proceeds to block311. At block311, the Error Analysis Module107creates a record which logs the context of the failed input, as well as an indication of the IME that the user selected. As discussed above, the input context may include a variety of information. In some embodiments, the input context includes the transcribed characters that led up to the failed transcription. In another embodiment, the input context includes the input itself that was improperly transcribed. In yet another embodiment, the input context includes the input provided by the user which was properly transcribed, as well as the resulting transcriptions. In some embodiments, the context includes the application that the input was intended for. Additionally, in some embodiments the input context includes a reference to the IME that provided the incorrect transcription.

In some embodiments, the input context also includes information about what the user did after the manual IME switch. For example, the log may include the transcribed character or characters that were entered just after the switch. Additionally, the log may include an indication that the user entered a single character or a small number of characters, and then switched back to the original IME. In some embodiments, this additional information enables the Error Analysis Module107to intelligently switch to a second IME upon determining that the first IME is incapable of correct transcription, and then determine when to return to the first IME based on learning from the user's patterns. After logging the context of the failed input and the IME switch, the method300returns to301to allow the user to continue to input information to be transcribed using the selected IME. In this way, the Error Analysis Module107will be better suited to intelligently switch IMEs for future transcriptions.

FIG. 4illustrates a flow chart for a method400implementing another embodiment of the present disclosure. The method400begins at block401, where the system receives input from a user. At block402, the system generates a string based on the received input and provides the string to the user. Block402broadly represents the transcription process illustrated inFIG. 3, but the method illustrated inFIG. 4applies only when the suggested string is correct and the user confirms its entry. The illustrated embodiment ofFIG. 4departs from the above discussed embodiments at block403. At block403, rather than simply returning to block401to receive input from the user, the Error Analysis Module107determines a context of the provided string. In some embodiments, the context includes the generated string itself. In other embodiments, the context also includes previously generated strings that led up to the current correct string. Additionally, in some embodiments, the context includes the application that the transcriptions are being generated for. The context may also include a reference to the IME that was used for each of the immediately preceding generated strings, as well as for the current correct string.

At block404, the Error Analysis Module107determines whether the current context is similar to or corresponds to prior IME switch records. The IME switch records contain logs of previous IME switches and the associated context. For example, as discussed above, the Error Analysis Module107could determine whether the sequence of strings or inputs of the current situation are similar to the sequence of strings that immediately preceded an IME switch in the past. Additionally or alternatively, the Error Analysis Module107may be configured to compare the application being used, the IME being used before the switch, and any other context disclosed herein. If the Error Analysis Module107determines that switching IMEs would be inappropriate, the method400returns to block401to receive input from the user. This determination may be made, for example, because no prior IME switches match the current context. In some embodiments, the current context may be similar to one or more prior switches, but the Error Analysis Module107may determine that a switch would be inappropriate for other reasons. For example, the Error Analysis Module107may determine that the current context occurs frequently and only rarely does the user intend to switch to a different IME for the next input.

Alternatively, the Error Analysis Module107may determine, at block404, that the IME should be switched based on the current context and the context of prior IME switches. For example, a user may repeatedly enter a series of characters such as a place, a name, or a phrase, and then switch manually to another IME because the next character to be entered will likely be difficult with the current IME. This pattern can recur frequently, with the Error Analysis Module107logging each switch. Additionally or alternatively, as discussed above, the Error Analysis Module107may automatically switch to a second IME based on receiving an indication that the suggested transcription is incorrect. Each of these automatic switches may also generate an IME switch record. Based on these records, at block404the Error Analysis Module107may determine that the user will likely want to switch to a different IME for the next character. If so, the method400proceeds to block405where the Error Analysis Module107automatically switches to the second IME. As discussed above, this switch could be accomplished through updating a pointer to the current IME, or any other appropriate method to achieve the functions disclosed herein.

In the illustrated embodiment ofFIG. 4, the Error Analysis Module107intelligently switches to a second IME before a transcription error is even made, based on recognizing a pattern of switches in the past. As discussed above, the Error Analysis Module107may further automatically switch back to the first IME after the next character is inputted, again based on the historic records and context. This enables the system to seamlessly transcribe input from the user using multiple IMEs, switching back and forth between them as needed, with minimal user intervention.

In some embodiments, the Error Analysis Module107can determine that the sequence of generated strings or inputs of the current situation constitutes an invalid sequence of transcriptions in the current IME but also identify a different IME for which the sequence constitutes a valid sequence of transcriptions. Determinations of what constitutes a valid or invalid sequence of transcriptions of a particular IME can be based on a predefined set of validity rules that are specific to the particular IME. In instances where a different, intended IME is unambiguously identified (e.g., where the sequence is valid only for a single one of the available IMEs and invalid for all the others), the Error Analysis Module107may switch to the identified IME in a manner that retroactively accepts the sequence as input—even absent past IME switches as guidance. Depending on the embodiment, such an IME switch may occur with or without triggering a formal error in the current IME. Doing so relieves the user from having to manually switch to the different IME and reenter the sequence of transcriptions.

Typically, cloud computing resources are provided to a user on a pay-per-use basis, where users are charged only for the computing resources actually used (e.g. an amount of storage space consumed by a user or a number of virtualized systems instantiated by the user). A user can access any of the resources that reside in the cloud at any time, and from anywhere across the Internet. In context of the present invention, a user may access applications (e.g., one or more IME transcription services) or related data available in the cloud. Additionally, the proactive IME switching application could available in the cloud. For example, the Error Analysis Module107could execute on a computing system in the cloud and intelligently switch which IME service should be provided to a user. In such a case, the Error Analysis Module107could determine which IME is appropriate to use for a given input and store IME switch logs at a storage location in the cloud. Doing so allows a user to access this information from any computing system attached to a network connected to the cloud (e.g., the Internet).