Patent Description:
Smartphones and mobile computing devices are configured to support voice typing which can be enabled when users activate a microphone function of the mobile device. In general, mobile computing devices can include at least two input method editors (IMEs), namely, a keyboard or text IME and a voice or speech IME. The text IME supports physical input and display of digital text, while the voice IME supports voice input and transcription of speech audio. For some mobile or user devices, the keyboard IME can be configured as a default IME and, thus, is the preselected input method option adopted by the device.

When a user of the mobile device activates the microphone function, the user can cause the device to experience a switch from the keyboard IME to the voice IME. In some instances, the switch can be indicated by an illuminated microphone icon viewable on a display of the mobile device. Similarly, while in voice dictation, manual correction of an incorrectly transcribed word can trigger an IME switch to the touch keyboard input method. In some instances, a user can input or type text via the keyboard IME and, when a particular word spelling is unknown, the user can activate the device microphone and elect to input the word by way of voice transcription.

<CIT> discloses a computer-implemented method for correcting words in transcribed text including receiving speech audio data from a microphone. <CIT> discloses a system and method for entering text from a user that includes a programmed processor that receives inputs from the user and disambiguates the inputs to present word choices corresponding to the text.

A computing system is described that includes at least a mobile device having a keyboard IME and voice IME. The described system receives user input by way of a voice input method of a mobile device. The system recognizes the user input and generates a transcription that includes a particular term spoken by the user. The system further generates an input context data structure that references at least the particular term. The input context data structure may comprise a data structure which comprises the particular term, along with other data representative of the input context or modality in which the particular term is received.

The input context data structure can generally include a time and/or date parameter, an indication of an application program associated with the received user input, and one or more n-grams that can include contiguous context items, e.g., letters or words, that are associated with a speech audio input. The speech audio corresponds to the user input received by the voice input method and can include a human speech utterance of the particular term.

The system then transmits the generated input context data structure to a keyboard IME of the mobile device for use in updating one or more language models accessible by the keyboard IME as well as by the voice IME. The input context data structure can also be used to update a global language model that is accessible globally by multiple users of the computing system. The updated language models enable keyboard IMEs and voice IMEs to recognize the particular term when the particular term is once again received as user input by either a voice input method or the keyboard input method of a mobile device.

In one innovative aspect of the specification, a computer-implemented method is described, that includes activating a first modality user input mode in which user inputs by way of a first modality are recognized using a first modality recognizer; and receiving a user input by way of the first modality. The method includes, obtaining, as a result of the first modality recognizer recognizing the user input, a transcription that includes a particular term; and generating an input context data structure that references at least the particular term. The method further includes, transmitting, by the first modality recognizer, the input context data structure to a second modality recognizer for use in updating a second modality recognition model associated with the second modality recognizer. As used in this specification, a modality can be a particular input mode, communication channel, or input signal path in which user input of a particular type is received and/or processed by a user device.

In some implementations, the method further includes, activating a second modality user input mode in which user inputs by way of the second modality are recognized using the second modality recognizer; receiving a user input by way of the second modality, the user input including the particular term; and in response to transmitting, recognizing, by the second modality recognizer, the particular term received by way of the second modality. In some implementations, recognizing the particular term by the second modality recognizer includes providing, by at least a display of a user device, an indication that the particular term is associated with a language model accessible by the second modality recognizer.

In some implementations, the method further includes, activating the first modality user input mode in response to receiving a user input by way of the second modality, wherein the received user input includes the particular term, and the particular term is not recognized by the second modality recognizer.

In some implementations, the second modality recognizer is configured to: detect an occurrence of user inputs received by way of the second modality that include input context data structures that reference at least the particular term; increment a first data count that tracks a number of occurrences in which input content that reference the particular term is received by way of the second modality; and increment a second data count that tracks a number of occurrences in which user inputs that correspond to the particular term are received by way of the second modality.

In some implementations, the method further includes, generating a database that includes multiple user inputs received by way of the first modality; and using at least one user input of the database of multiple user inputs to update one or more global language models accessible by at least one of the first modality recognizer or the second modality recognizer.

In some implementations, the first modality user input mode includes a voice input mode in which user inputs corresponding to human speech are recognized using the first modality recognizer. In some implementations, the first modality recognizer is a voice input method editor (IME) configured to receive an audio input signal corresponding to human speech that includes an utterance of the particular term.

Other implementations of this aspect include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the methods. A system of one or more computers can be configured to perform particular operations or actions by virtue of having software, firmware, hardware, or a combination of them installed on the system that in operation causes or cause the system to perform the actions. One or more computer programs can be configured to perform particular operations or actions by virtue of including instructions that, when executed by data processing apparatus, cause the apparatus to perform the actions.

In another innovative aspect of the specification, a computer-implemented method is described, that includes, activating, in a computing device, a voice user input mode in which user inputs by way of a voice modality are recognized using a voice modality recognizer; and receiving a user input by way of the voice modality. The method includes, obtaining, by the computing device and as a result of the voice modality recognizer recognizing the user input, a transcription that includes a particular term; and generating an input context data structure that references at least the particular term. The method further includes, transmitting, by the voice modality recognizer, the input context data structure to a keyed input modality recognizer for use in updating a keyed modality recognition model associated with the second modality recognizer.

The subject matter described in this specification can be implemented in particular implementations and can result in one or more of the following advantages. The computing system of this specification removes the need to configure or define separate learning models or logic constructs to enhance keyboard IME learning in computing devices. By not coding a multitude of keyboard learning models, computing device processes are optimized and processing efficiency is improved by minimizing unnecessary computations.

Received audio inputs are transcribed and transmitted to a local keyboard IME as well as a global language model for use by a multitude of user devices globally. Keyboard IME enhancements can be efficiently accomplished based on, for example, server-based or local device analysis of audio input signals corresponding to new or evolving speech utterances. Hence, redundant signal analysis of speech and keyboard user inputs of common words is avoided, thereby providing enhanced system bandwidth for other computations and system transmissions.

In addition to the common words, based on the described subject matter, the keyboard IME is now enabled to learn new words using the speech recognition functions of the computing device. For example, a new word can correspond to a term that did not exist before within a particular spoken language or computer language model (e.g., "selfie" or "bae") or a naming for a new place/location.

<FIG> illustrates multiple interfaces related to cross-modality learning in an example computing system. The multiple interfaces include interface <NUM>, <NUM>, <NUM>, and <NUM>. Each illustrated interface corresponds to an example user input screen that can be displayed on a user device <NUM>. As depicted in <FIG>, in some implementations, user device <NUM> can correspond to a mobile smartphone device.

In alternative implementations, user device <NUM> can be one of a variety of computing devices including devices such as laptop/desktop computers, smart televisions, electronic reading devices, streaming content devices, gaming consoles, tablet devices or other related computing devices that are configured to execute software instructions and application programs associated with a voice input method editor (IME) and a keyboard IME.

Interface <NUM> can be displayed on user device <NUM> and can include an example user interface of an application program that receives user input from a user <NUM>. In some implementations, the received user input is speech or voice input. As discussed in more detail below with reference to <FIG>, user device <NUM> can include at least two IMEs, namely, a keyboard or text IME and a voice or speech IME.

In some implementations, functions associated with each IME can be executed in an example cloud-based computing system accessible by user device <NUM>. In the implementation of <FIG>, user device <NUM> can be configured such that the keyboard IME is the default IME and, thus, is the preselected input method option adopted by device <NUM>. Interface <NUM> can include a digital representation of a microphone <NUM> that illuminates when user <NUM> causes device <NUM> to experience a switch from the keyboard IME to the voice IME.

User <NUM> of user device <NUM> can activate a microphone function of device <NUM> to enable voice dictation. Moreover, interface <NUM> can be configured to display a message that states "Voice Input Active. " The displayed message indicates to user <NUM> that device <NUM> is in a voice input mode and can receive speech or voice input. The received speech input can be transcribed locally by device <NUM> (i.e., client side), or by a cloud-based computing system (i.e., server side), to produce transcription <NUM>.

User <NUM> can de-activate the microphone function of device <NUM> to disable voice dictation and switch to the keyboard IME of user device <NUM>. Hence, interface <NUM> can correspond to a text, touch, keyboard, or physical input mode in which user inputs to device <NUM> are received by way of a digital or physical keyboard input method. In some implementations, user device <NUM> is a touch screen device that displays a digital keyboard. The digital keyboard can be configured to receive motion inputs <NUM> that correspond to swiping motions, graffiti motions, or gesture motions.

Touch or physical inputs received by user device <NUM> can be depicted as text <NUM>. In some implementations, user <NUM> attempts to use functionality associated with the keyboard IME to type or enter a particular term. For example, the particular term can be the word "Milpitas. " In some implementations, user <NUM> may type an example text or email message to a friend, Bob. Although not depicted in interface <NUM>, the message can indicate that user <NUM> suggests to meet Bob in an example location, "Milpitas," a city in Santa Clara County, California.

As discussed in more detail below, the keyboard IME of user device <NUM> can be coupled to an example language model that includes multiple words associated with multiple languages. However, in this instance the language model does not recognize the typed word "Milpitas. " Hence, because the word "Milpitas" is not recognized by the model, autocorrect logic associated with the keyboard IME may, for example, suggest to change or autocorrect Milpitas to "mimosas," as depicted in text <NUM> of interface <NUM>.

Similarly, autocorrect or spell-check logic associated with the keyboard IME may also indicate, to user <NUM>, that the entered word, "Milpitas," is spelled incorrectly. Thus, as depicted by interface <NUM>, sample words such as "mimosas," "Milos," or "miles" can be suggested by example text suggestion logic associated with the keyboard IME of device <NUM>. In response to user device <NUM> suggesting to change a particular entered word to another word, user <NUM> can activate the microphone function of device <NUM> to enable voice dictation.

Interface <NUM> and interface <NUM> provide depictions of one or more operations that are associated with cross-modality learning. Interface <NUM> depicts an illuminated microphone <NUM> which occurs when user <NUM> causes device <NUM> to experience a switch from the keyboard IME to the voice IME. In some implementations, a cross-modality learning operation can include activating a voice user input mode in which user inputs by way of a voice modality are recognized using a voice modality recognizer.

For example, the switch from the keyboard IME to the voice IME can generally correspond to activating the voice user input mode to enable voice dictation. Further, the voice modality recognizer can generally correspond to the voice IME, while the voice modality can generally correspond to voice input functions of user device <NUM> in which voice dictation functionality is enabled. As used in this specification, a modality can be a particular input mode, communication channel, or input signal path in which user input of a particular type is received and/or processed by user device <NUM>.

Referring again to the cross-modality learning operation, user input by way of the voice modality can be received by user device <NUM>. The voice IME can be configured to recognize user inputs such as audio input related to human speech that includes multiple word utterances. Further, as a result of the voice IME recognizing the user input, device <NUM> can obtain a transcription that includes a particular term. For example, in the depiction of interface <NUM>, the particular term can be input provided by user <NUM> in the form of a human speech utterance of the word "Milpitas.

The learning operation can include obtaining a transcription of the particular term or speech utterance. Hence, as shown by text <NUM>, a transcription of the spoken word "Milpitas" is obtained during the example cross-modality learning operation. In some implementations, user device <NUM> obtains the transcription based, in part, on data processing operations that occur locally within user device <NUM>. While in some implementations, user device <NUM> obtains the transcription based, in part, on data processing operations that occur remotely within an example cloud-based or server-based computing system.

In some implementations, although the voice IME can properly recognize the user input and obtain an accurate transcription, the voice IME language model may not include the particular term, "Milpitas. " Accordingly, spell-check logic that references the language model associated with the voice IME may not recognize the transcribed term, "Milpitas. " Hence, because the word "Milpitas" is not recognized, the spell-check logic may, for example, indicate to user <NUM> that the transcribed word, "Milpitas," is spelled incorrectly.

In response to receiving this indication, user <NUM> can disregard the incorrect spelling indication provided by the spell-check logic. Alternatively, in some implementations, user device <NUM> may prompt user <NUM> to affirmatively accept the transcribed spelling of the particular term "Milpitas. " In interface <NUM>, the depiction of text <NUM> can be interpreted as an indication that user <NUM> has accepted the spelling of "Milpitas" as correct.

Upon indication of user <NUM> accepting the transcribed spelling, the particular term, "Milpitas," received by way of the voice modality will then be added or saved to one or more language models associated with the voice IME. Once added to the language models, the particular term can be accessible for use in subsequent speech-to-text communications. For example, once stored in the language models, the word "Milpitas" can be used in subsequent communications without triggering the occurrence of autocorrect logic or spell-check logic.

The cross-modality learning operation can further include, generating an input context data structure that references at least the particular term. For example, an input context data structure can be generated that includes at least the term "Milpitas" as well as multiple other items associated with the received user input. In some implementations, the multiple other items can include an example application program used to enter the particular term and a time and/or date that indicates when the particular term was received.

The cross-modality learning operation can further include the voice modality recognizer transmitting the input context data structure to a keyboard or physical input modality recognizer for use in updating a keyboard modality recognition model associated with the keyboard modality recognizer.

For example, an input context data structure can be transmitted by the voice IME to the keyboard IME. The input context data structure can include the term "Milpitas," an indication of a text/email message application program used to input "Milpitas," and the data/time in which user <NUM> entered "Milpitas" by way of the voice input method. The keyboard IME can be associated with a keyboard modality recognition model that includes at least a spatial model (described below) and a language model.

Interface <NUM> shows text <NUM> being input, to user device <NUM>, via the keyboard or physical input mode. In some implementations, the transmitted input context data structure can be used to update a keyboard language model that is accessed by the keyboard IME. The updated keyboard language model enables user <NUM> to input text communication that includes the particular term "Milpitas" so that the term is appropriately recognized by spell-check and/or autocorrect logic associated with the keyboard IME. Further, as indicated by text <NUM>, user <NUM> can swipe or gesture the term "Milpitas" based on the spatial model and language model of the keyboard IME being updated to include the particular term "Milpitas.

<FIG> illustrates a system diagram of an example computing system <NUM> for cross-modality learning. System <NUM> generally includes a speech modality recognition model <NUM> (speech model <NUM>), a keyboard modality recognition model <NUM> (keyboard model <NUM>), a cross-modality learning module <NUM> (learning module <NUM>), and a global language model <NUM> (global LM <NUM>).

As used in this specification, the term "module" is intended to include, but is not limited to, one or more computers configured to execute one or more software programs that include program code that causes a processing device(s) of the computer to execute one or more functions. The term "computer" is intended to include any data processing device, such as a desktop computer, a laptop computer, a mainframe computer, a tablet device, a server, a handheld device, a mobile or smartphone device or any other device able to process data.

Speech model <NUM> can include an acoustic model <NUM>, a speech language model <NUM>, and a speech IME <NUM>. Speech model <NUM> is generally configured to receive audio input <NUM> and execute a variety of data and signal processing functions to identify and extract one or more words associated with human speech spoken in a particular language.

Speech model <NUM> can be used in conjunction with one or more application programs that are accessible from user device <NUM>. In some implementations, speech model <NUM> can be formed, in part, from software or program code executing in modules, processor devices, or circuit components that are disposed locally within user device <NUM>. While, in other implementations, speech model <NUM> can be associated with non-local, cloud, or server-based computing systems that receive and process audio signal transmissions from device <NUM>.

Acoustic model <NUM> can be an example acoustic model used in speech recognition to associate relationships between an audio signal and phonemes or other linguistic properties that form speech audio. In general, acoustic model <NUM> can interact with speech IME <NUM> to identify and associate certain received utterances that exhibit acoustical characteristics that align with the acoustics associated with an example spoken word such as "MILPITAS.

Language model <NUM> can be an example language model used in speech recognition to specify or identify certain word combinations or sequences. In some implementations, model <NUM> can be configured to generate a word sequence probability factor which can be used to indicate the likely occurrence or existence of particular word sequences or word combinations. The identified word sequences correspond primarily to sequences that are specific to speech corpus rather than to written corpus.

Speech IME <NUM> can include a speech buffer <NUM>, a recognizer <NUM> and a LM manager <NUM>. Speech buffer <NUM>, and buffer <NUM>, can each include one or more memory units configured to temporarily buffer or store speech or audio signals for data or signal processing by speech model <NUM>. Speech buffers <NUM>, <NUM> can include one or more non-transitory machine-readable storage mediums. The non-transitory machine-readable storage mediums can include solid-state memory, magnetic disk, and optical disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (e.g., EPROM, EEPROM, or Flash memory), or any other tangible medium capable of storing information.

In addition to their respective buffers, <NUM> and <NUM>, speech IME <NUM> and keyboard IME <NUM> (described below) can each include multiple processing devices. The processing devices can include one or more processors (e.g., microprocessors or central processing units (CPUs)), graphics processing units (GPUs), application specific integrated circuits (ASICs), or a combination of different processors. In alternative implementations, system <NUM> can include other computing resources/devices, e.g., cloud-based servers, that provide additional processing options for performing one or more of the determinations and calculations described in this specification.

The processing devices can generally include one or more memory units or memory banks that are non-transitory machine-readable storage mediums. In some implementations, the processing devices execute programmed instructions stored in the memory units to cause system <NUM> and its associated components to perform one or more functions described in this specification.

In some implementations, recognizer <NUM> can be example speech recognition logic, programmed instructions, or algorithms that are executed by one or more processors of speech IME <NUM>. For example, recognizer <NUM> can execute program code to manage identification, extraction, and analysis of characteristics of the received audio input <NUM>. Further, recognizer <NUM> can execute comparator logic to compare characteristics of the received audio input <NUM> to various model parameters stored in acoustic model <NUM> and language model <NUM>. Results of the comparison can yield text transcription outputs that correspond substantially to speech utterances provided by one or more users <NUM> of system <NUM>.

LM manager <NUM> can include example access or management logic that controls and/or manages access to one or more model parameters of language model <NUM>. For example, LM manager <NUM> can be configured to access certain parameters of language model <NUM> based on particular characteristics of received audio input <NUM> that are identified and analyzed by recognizer <NUM>. For example, recognizer <NUM> may identify a characteristic of the received audio input <NUM> as including one or more word utterances that correspond to the English or Spanish languages. Thus, LM manager <NUM> will access model parameters of language model <NUM> that are associated with either the spoken English language, the spoken Spanish language, or both.

In general, recognizer <NUM> and LM manager <NUM> can interact or cooperate to execute a variety of data processing and signal processing functions. Execution of these functions enable completion of process steps necessary to perform speech audio input recognition and to convert the speech audio to text transcription.

As noted above, speech model <NUM>, as well as keyboard model <NUM>, can each be used in conjunction with one or more application programs that are accessible from user device <NUM>. Example application programs can include an email application, a text message application, an instant messaging application, a web browsing application, a mapping application, or any other application program configured to receive user input such as speech audio input, digital text input, alpha-numeric input, character input, or digital image input.

Cross-modality learning module <NUM> can be configured to execute program code to, in part, generate input context data structures for transmission between speech model <NUM> and keyboard model <NUM> as well as between learning module <NUM> and global language model <NUM>. In some implementations, learning module <NUM> can aggregate multiple parameter values based on parameter signals received from processors of user device <NUM>, voice IME <NUM>, and keyboard IME <NUM>.

For example, learning module <NUM> can receive parameter values that indicate particular application programs used in conjunction with the respective IMEs of system <NUM> to receive text or speech user inputs. Moreover, user device <NUM> can provide date and time parameters that indicate when a particular speech or typed term and associated context words are received by the respective IMEs. Additionally, the respective IME's can provide n-gram contexts or a full transcription lattice associated with received speech or typed input.

Learning module <NUM> can generate input context data structure(s) <NUM> based on the received parameter values and facilitate transmission of the generated data structures <NUM> between the respective IMEs <NUM>, <NUM> and between learning module <NUM> and global LM <NUM>. In some implementations, global LM <NUM> receives input context data structures <NUM> from learning module <NUM> that are used, by global LM <NUM>, to update language models that are accessible globally to a multitude of users.

In some implementations, system <NUM> can provide one or more parameter values or data structures <NUM> to generate a database that includes multiple user inputs received, by system <NUM>, through a keyboard or voice modality. The parameter values and data structures <NUM> can include one or more particular terms or new words. The database can be associated, at least in part, with global language model <NUM>. Further, in some implementations, global LM <NUM> can include a variety of individual language models that correspond to a variety of different spoken languages and input modalities. System <NUM> can use at least one user input of the database of multiple user inputs to update the one or more language models of global LM <NUM>.

In alternative implementations, global LM <NUM> can provide, to learning module <NUM>, data structures and/or parameter values that can include new words or particular terms received from other global users. Learning module <NUM> can then generate one or more input context data structures <NUM> which are then transmitted to one of speech model <NUM> or keyboard model <NUM> to update their respective language models <NUM> and <NUM>. Hence, in some implementations, keyboard IME <NUM> and speech IME <NUM> can learn new particular terms based on parameters or data structures received from global LM <NUM>.

Keyboard model <NUM> can include a spatial model <NUM>, a language model <NUM>, and a keyboard IME <NUM>. Keyboard model <NUM> is generally configured to receive touch/physical keyboard inputs <NUM> that correspond to letters, numbers, and other characters that are displayed as digital text that form words or phrases.

Much like speech model <NUM> discussed above, keyboard model <NUM> can also be formed, in part, from software or program code executing in modules, processor devices, or circuit components that are disposed locally within user device <NUM>. While, in other implementations, keyboard model <NUM> can be associated with non-local, cloud, or server-based computing systems that receive and process audio signal transmissions from device <NUM>.

Aside from spatial model <NUM>, technical descriptions of the functions for language model <NUM> and keyboard IME <NUM> can be similar to descriptions of language model <NUM> and speech IME <NUM> discussed above. For clarity and brevity, language model <NUM> can be described by noting technical distinctions relative to language model <NUM>. Likewise, keyboard IME <NUM> can be described by noting technical distinctions relative to speech IME <NUM>.

Language model <NUM> can be used in keyboard text recognition to identify certain letter combinations or sequences. In some implementations, model <NUM> can be configured to generate a letter or word sequence probability factor that can be used to indicate the likely occurrence or existence of particular letter sequences or word combinations. The identified letter and word sequences correspond primarily to sequences that are specific to written corpus rather than to speech corpus.

In some implementations, recognizer <NUM> can be text recognition logic, programmed instructions, or algorithms that are executed by one or more processors of keyboard IME <NUM>. For example, recognizer <NUM> can execute program code to manage identification, extraction, and analysis of characteristics of the received text input <NUM>. Further, recognizer <NUM> can execute comparator logic to compare spatial characteristics of the received text input <NUM> to various model parameters stored in spatial model <NUM> and language model <NUM>.

Spatial model <NUM> can be an example spatial model used in text prediction to associate spatial coordinates of letters or spatial relationships between letters to predict typed, swiped, or gestured words that are input via a keyboard of user device <NUM>. In general, spatial model <NUM> can interact with keyboard IME <NUM> to identify and associate keyboard inputs that correspond spatially with letters that form words associated with certain written corpus.

System <NUM> generally can include the following operational processes and functions. User <NUM> can speak to user device <NUM> or provide speech input that includes word utterances that are not included in or known by language model <NUM> or speech IME <NUM>. For example, user <NUM> and speak to user device <NUM> by saying a particular term such as "Milpitas. " Acoustic model <NUM> can interact with other components of speech model <NUM> to accurately transcribe the spoken input <NUM> so that "Milpitas" is displayed as text <NUM> in an example application program.

In some implementations, user <NUM> will indicate to the application program that user <NUM> accepts the transcribed spelling of the particular term. For example, speech model <NUM> can execute program code to detect or determine if user <NUM> has modified the transcription generated by speech model <NUM>. In some implementations, is user <NUM> proceeds to enter additional speech input or manually type/input text that precedes or comes after "Milpitas," without modifying the proposed transcription text <NUM>, then speech model <NUM> can determine that user <NUM> has accepted the speech to text transcription <NUM>.

If speech model <NUM> determines that user <NUM> has accepted the transcribed term "Milpitas," system <NUM> can store the particular term in language model <NUM> and/or global LM <NUM>. In some implementations, when system <NUM> stores previously unknown particular terms in the various respective language models of the system, these storage operations can effectively constitute real-time learning functions.

In general, system <NUM> can execute data processing and storage operations such that the system, and its associated IME's, can learn new spoken terms both through server-side cloud based learning process as well as local client side learning processes. Stated another way, the first time user <NUM> says a new word to device <NUM> and speech model <NUM> is able to recognize an utterance that aligns with parameters of acoustic model <NUM> and that is accepted as a correct transcription by the user; system <NUM> will recognize the word, save the word to speech LM <NUM>, and transmit a data structure that includes the word.

The transmitted data structure will be received by at least keyboard model <NUM> for use by keyboard IME <NUM>. Thus, when user <NUM> subsequently and accurately types, gestures or swipes the particular text string for "Milpitas," keyboard IME <NUM> will recognize the word as being known by language model <NUM>. Hence, system <NUM> will learn the particular term, save to the term for use by voice/speech IME <NUM> and transfer it to keyboard IME <NUM> so that the particular term can also be learned by keyboard model <NUM> while user <NUM> types or speaks other input content to device <NUM>.

In some implementations, after a particular input context data structure <NUM> is transmitted by speech IME <NUM>, and received by keyboard IME <NUM>, user <NUM> can then activate the keyboard modality input mode. In this mode, user inputs by way of the keyboard/text modality are recognized using keyboard modality recognizer, i.e., keyboard IME <NUM>. System <NUM> can then receive user input <NUM> by way of the keyboard modality and input <NUM> can include the particular term "Milpitas. " Keyboard model <NUM> learns the particular term "Milpitas" in response to speech IME <NUM> and/or learning module <NUM> transmitting input context data structure <NUM>. Subsequent to learning the particular term, keyboard IME <NUM> can then recognize the particular term received by way of the keyboard/text modality.

In some implementations, recognizing the particular term "Milpitas" by keyboard IME <NUM> can include a display of user device <NUM> providing an indication to user <NUM>. For example, the display can indicate to user <NUM> that the particular term has been added or saved to language model <NUM>. In some implementations, after "Milpitas" is added to LM <NUM>, user <NUM> can type an example text phrase such as "Drive to Milpitas" and receive a general indication that the word is recognized by keyboard model <NUM>. For example, the indication can correspond to a text display <NUM> that includes the word "Milpitas" without triggering, for example, spellcheck or autocorrect logic associated with model <NUM>.

In some implementations, system <NUM> can be configured to detect the occurrence of user inputs received by keyboard model <NUM> that include text content or a text lattice that references at least the particular term, e.g., "Milpitas. " For example, system <NUM> can be configured to detect when an example phrase or text lattice such as "Drive to Milpitas" is received by keyboard IME <NUM>. In general, system <NUM> will detect occurrences of the particular term after first having learned the particular term.

In response to detecting text content that references the particular term, system <NUM> can increment a first data count that tracks a number of occurrences in which text content that references the particular term is received by way of the keyboard modality. In some implementations, system <NUM> can also increment a second data count that tracks a number of occurrences in which user inputs that correspond to the particular term are received by way of the second modality. For example, in addition to detecting and incrementing data counts for a received text lattice that includes the particular term, system <NUM> can also detect and increment data counts that track individual occurrences of the particular term rather than occurrences of a text lattice that includes the particular term.

In some implementations, the first and second data counts can be used, by system <NUM>, to generate data sets of aggregated statistics that are associated with the particular term. Other statistics in the data sets can include, for example, variations on spelling and capitalization of the particular term. Further, statistical data can be aggregated relating to contextual variations that indicate use of the particular term in a variety of different text or speech contexts, e.g., "Drive to Milpitas," "Meet at MILPITAS," "Let's eat at milpitaas. " In some implementations, the generated data sets of aggregated statistics can be used by system <NUM> to bias, improve or enhance keyboard input or voice input learning functions within the respective models <NUM>, <NUM>.

In other implementations, the generated data sets of aggregated statistics can be transmitted to global LM <NUM>. For example, global LM <NUM> can receive a variety of inputs, from system <NUM>, associated with disparate users that may be attempting to enter the particular term "Milpitas. " As indicated in the preceding paragraph, in some instances, one or more users of system <NUM> may spell "Milpitas" incorrectly or may use improper capitalization. Such incorrect or improper uses of the particular term may not be used to update the one or more language models of global LM <NUM>. Alternatively, for instances in which the particular term is used correctly by a threshold number of users <NUM>, system <NUM> will cause the language models of global LM <NUM> to be updated with the most appropriate use of the particular term.

<FIG> is a flow diagram of an example process <NUM> for cross-modality learning. At block <NUM> process <NUM> includes activating a first modality user input mode in which user inputs by way of a first modality are recognized using a first modality recognizer. In some implementations, activating the first modality user input mode includes switching from a keyboard IME to a voice IME in an example mobile device such as device <NUM>. The first modality can correspond to a voice modality relating to voice input functions of user device <NUM> in which voice dictation is enabled. Additionally, the first modality recognizer can correspond to voice IME <NUM>.

At block <NUM> process <NUM> receives user input by way of the first modality. In some implementations, the received user input can be audio input corresponding to human speech that includes one or more word utterances. Further, the received user input can include one or more particular terms that are recognized by voice IME <NUM>.

At block <NUM>, as a result of the first modality recognizer recognizing the user input, process <NUM> obtains a transcription that includes the particular term. In some implementations, recognizing the user input can include a voice recognition model of system <NUM> processing the audio input to parse out one or more words. The parsed words can include the particular term and system <NUM> can generate text transcriptions based on the parsed words that are recognized from the received speech utterance. In some implementations, transcriptions are generated, in part, by a remote server or cloud-based computing system. The generated transcriptions can be subsequently obtained, by device <NUM>, from the computing system.

At block <NUM> process <NUM> includes generating an input context data structure that references at least the particular term. In some implementations, the input context data structure can include the particular term as well as other items such as an example application program used to enter the particular term, one or more n-grams of the speech utterance of the user input, and a time and/or date that indicates when the particular term was received.

At block <NUM>, process <NUM> includes transmitting, by the first modality recognizer, the input context data structure to a second modality recognizer. At block <NUM> of process <NUM>, the transmitted input context data structure is used to update a second modality recognition model associated with the second modality recognizer. The second modality can correspond to a keyboard or physical input modality relating to keyboard input functions of user device <NUM> in which a digital or physical keyboard is used to input text content. Additionally, the second modality recognizer can correspond to keyboard IME <NUM>.

In some implementations, the second modality recognition model can correspond to keyboard modality recognition model <NUM> that includes at least spatial model <NUM> and language model <NUM>. In some implementations, the transmitted input context data structure can be used to update a keyboard language model that is accessed by the keyboard IME. The updated keyboard language model can enable user device <NUM> to receive input text communication that includes the particular term such that the term can be appropriately recognized by, for example, spell-check and/or autocorrect logic associated with the keyboard IME.

Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory program carrier for execution by, or to control the operation of, data processing apparatus.

<FIG> is a block diagram of computing devices <NUM>, <NUM> that may be used to implement the systems and methods described in this document, either as a client or as a server or plurality of servers. Computing device <NUM> is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. Computing device <NUM> is intended to represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, smartwatches, head-worn devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations described and/or claimed in this document.

In various different implementations, the storage device <NUM> may be a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid state memory device, or an array of devices, including devices in a storage area network or other configurations.

The high-speed controller <NUM> manages bandwidth-intensive operations for the computing device <NUM>, while the low speed controller <NUM> manages lower bandwidth-intensive operations.

The processor <NUM> can process instructions for execution within the computing device <NUM>, including instructions stored in the memory <NUM>. The processor may also include separate analog and digital processors.

The display <NUM> may be, for example, a TFT LCD display or an OLED display, or other appropriate display technology. External interface <NUM> may provide, for example, for wired communication (e.g., via a docking procedure) or for wireless communication (e.g., via Bluetooth or other such technologies).

Expansion memory <NUM> may also be provided and connected to device <NUM> through expansion interface <NUM>, which may include, for example, a SIMM card interface. Thus, for example, expansion memory <NUM> may be provided as a security module for device <NUM>, and may be programmed with instructions that permit secure use of device <NUM>.

The memory may include for example, flash memory and/or MRAM memory, as discussed below. The information carrier is a computer- or machine-readable medium, such as the memory <NUM>, expansion memory <NUM>, or memory on processor <NUM>.

In addition, GPS receiver module <NUM> may provide additional wireless data to device <NUM>, which may be used as appropriate by applications running on device <NUM>.

Various implementations of the systems and techniques described here can be realized in digital electronic circuitry, integrated circuitry, specially designed ASICs, computer hardware, firmware, software, and/or combinations thereof.

These computer programs, also known as programs, software, software applications or code, include machine instructions for a programmable processor, and can be implemented in a high-level procedural and/or object-oriented programming language, and/or in assembly/machine language. As used herein, the terms "machine-readable medium" "computer-readable medium" refers to any computer program product, apparatus and/or device, e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs) used to provide machine instructions and/or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal.

To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer.

The systems and techniques described here can be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component such as an application server, or that includes a front-end component such as a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here, or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication such as, a communication network.

Further to the descriptions above, a user may be provided with controls allowing the user to make an election as to both if and when systems, programs or features described herein may enable collection of user information (e.g., information about a user's social network, social actions or activities, profession, a user's preferences, or a user's current location), and if the user is sent content or communications from a server. In addition, certain data may be treated in one or more ways before it is stored or used, so that personally identifiable information is removed. For example, in some embodiments, a user's identity may be treated so that no personally identifiable information can be determined for the user, or a user's geographic location may be generalized where location information is obtained (such as to a city, ZIP code, or state level), so that a particular location of a user cannot be determined. Thus, the user may have control over what information is collected about the user, how that information is used, and what information is provided to the user.

A number of embodiments have been described. Nevertheless, it will be understood that various modifications may be made without departing from the scope of the invention. For example, various forms of the flows shown above may be used, with steps re-ordered, added, or removed. Also, although several applications of the payment systems and methods have been described, it should be recognized that numerous other applications are contemplated. Accordingly, other embodiments are within the scope of the following claims.

Claim 1:
A computer-implemented method, comprising:
receiving, by a computing device, user input of a particular term via a first user input mode associated with the computing device;
generating, by the computing device, parameter values associated with the particular term, wherein the parameter values are based on a first modality recognition model corresponding to the first user input mode;
generating, based on the parameter values, an input context data structure that references at least the particular term; and
transmitting, by the computing device and to a second modality recognition model corresponding to a second user input mode associated with the computing device, the input context data structure for use in updating the second modality recognition model.