Patent Description:
Users may exchange messages through messaging applications. In one example, a messaging application may allow a sender to type in a message that is sent to a recipient. Messaging applications may also allow the sender to speak a message, which the messaging applications may transcribe before sending to a recipient. <CIT> relates to a method for embedding voice mail in a spoken utterance using a natural language processing computer system. <CIT> relates to systems and methods for intelligent call transcription.

Example implementations deploy audio slicing to provide efficient voice-activated audio messaging. Example implementations are capable of responding to a voice command by actively isolating an audio message included in the voice command, and sending the audio message to a recipient. By extracting the audio message payload from the received voice command, a data-efficient (and therefore bandwidth-efficient) messaging system may be provided.

When sending a text message to a recipient, a sender may choose to speak a messaging-related command to the device rather than entering a message using a keyboard. For example, a sender may say "Text Liam good luck. " In response, the device would transcribe the sender's speech and recognize "text" as the voice command trigger term, "Liam" as the recipient, and "good luck" as the payload, or object of the voice command trigger term. The device would then send the message "good luck" to a contact of the sender's, named "Liam.

Just sending the transcription of the message may be insufficient to capture the intonation in the sender's voice. In this instance, it may be helpful to send the audio data of the sender speaking "good luck" along with the transcription. In order
to send only the audio data of the object of the voice command trigger term and not audio data of the recipient's name or of the voice command trigger term, the device first identifies the voice command trigger term in the transcription and compares it to other trigger terms that are compatible with sending audio data and transcriptions of the audio data (e.g., "text" and "send a message to," not "call" or "set an alarm"). The device then classifies a portion of the transcription as the object of the voice command trigger term and isolates the audio data corresponding to that portion. The device sends the audio data and the transcription of the object of the voice command trigger term to the recipient. The recipient can then listen to the sender's voice speaking the message and read the transcription of the message. Following the same example above, the device isolates and sends the audio data of "good luck" so that when Liam reads the message "good luck," he can also hear the sender speaking "good luck.

According to an innovative aspect of the subject matter described in this application, a method for audio slicing includes the actions of receiving audio data that corresponds to an utterance; generating a transcription of the utterance; classifying a first portion of the transcription as a voice command trigger term and a second portion of the transcription as an object of the voice command trigger term; identifying a language of the utterance;.

These and other implementations can each optionally include one or more of the following features. The actions further include classifying a third portion of the transcription as a recipient of the object of the voice command trigger term; and transmitting the data structure to the recipient. The data structure is generated based on determining the language of the utterance. The voice command trigger term is a command to send a text message. The object of the voice command trigger term is the text message. The actions further include generating, for display, a user interface that includes a selectable option to generate the data structure that includes the transcription of the object of the voice command trigger term and the audio data of the object of the voice command trigger term; and receiving data indicating a selection of the selectable option to generate the data structure. The data structure is generated in response to receiving the data indicating the selection of the selectable option to generate the data structure. The actions further include generating timing data for each term of the transcription of the utterance. The audio data of the object of the voice command trigger term is isolated based on the timing data. The timing data for each term identifies an elapsed time from a beginning of the utterance to a beginning of the term and an elapsed time from the beginning of the utterance to a beginning of a following term.

Other embodiments of this aspect include corresponding systems, apparatus, and computer programs recorded on computer storage devices, each configured to perform the operations of the methods.

The subject matter described in this application may have one or more of the following advantages. The network bandwidth required to send the sound of a user's voice and a message may be reduced because the user can send the audio of the user speaking with the message and without additionally placing a voice call, thus saving on the overhead required to establish and maintain a voice call. The network bandwidth required may also be reduced because the transcription and the audio data can be sent within one message packet instead of a message packet for the audio data and message packet for the transcription. The network bandwidth may be reduced again by extracting only the audio data of the message for transmission to the recipient instead of sending the audio data of the entire utterance.

<FIG> illustrates an example system <NUM> where a device <NUM> sends a data structure <NUM> that includes audio data <NUM> and a transcription <NUM> of the audio data to another device <NUM>. Briefly, and as described in more detail below, the device <NUM> receives audio data corresponding to an utterance <NUM> that is spoken by the user <NUM>. The device <NUM> transcribes the audio data corresponding to the utterance <NUM> and generates a data structure <NUM> that includes the transcription <NUM> of the message portion of the utterance <NUM> and the audio data <NUM> of the message portion of the utterance <NUM>. Upon receipt of the data structure <NUM>, the user <NUM> is able to read the transcription <NUM> on a display of the device <NUM>, and the device plays the audio data <NUM> so the user <NUM> can hear the voice of the user <NUM> speaking.

The user <NUM> activates a messaging application on the device <NUM>. The device <NUM> may be any type of computing device that is configured to receive audio data. For example, device <NUM> may be a mobile phone, a tablet, a watch, a laptop, a desktop computer, or any other similar device. Once the user <NUM> activates the messaging application, the device <NUM> may prompt the user to begin speaking. In some implementations, the device <NUM> may prompt the user to select from different messaging options. The messaging options may include sending a transcription only, sending a transcription and audio data, sending audio data only, or automatically sending audio data if appropriate. The user speaks the utterance <NUM> and the device <NUM> receives the corresponding audio data. The device <NUM> processes the audio data using an audio subsystem that may include an A-D converter and audio buffers.

The device <NUM> processes the audio data <NUM> that corresponds to the utterance <NUM> and, in some implementations, generates a transcription <NUM> of the audio data <NUM>. In some implementations, while the user speaks, the device <NUM> generates the transcription <NUM> and the recognized text appears on a display of the device <NUM>. For example, as the user <NUM> speaks "text mom," the words "text mom" appear on the display of the device <NUM>. In some implementations, the transcription <NUM> does not appear on the display of the device <NUM> until the user <NUM> has finished speaking. In this instance, the device <NUM> may not transcribe the audio data until the user <NUM> has finished speaking. In some implementations, the device <NUM> may include an option that the user can select to edit the transcription. For example, the device <NUM> may have transcribed "text don" instead of "text mom. " The user may select the edit option to change the transcription to "text mom. " In some implementations, the display of the device <NUM> may just provide visual indication that the device <NUM> is transcribing the audio data <NUM> without displaying the transcription <NUM>. In some implementations, the device <NUM> provides the audio data <NUM> to a server, and the server generates the transcription <NUM>. The server may then provide the transcription <NUM> to the device <NUM>.

Once the device <NUM> has generated the transcription <NUM>, the device <NUM>, in some implementations, generates timing data <NUM>. The timing data <NUM> consists of data that indicates an elapsed time from the beginning of the audio data <NUM> to the start of each word in the transcription <NUM>. For example, T0 represents the elapsed time from the beginning of the audio data <NUM> to the beginning of the word "text. " In some implementations, the device <NUM> may pre-process the audio data <NUM> so that T0 is zero. In other words, any periods of silence before the first word are removed from the audio data <NUM>. As another example, T2 represents the time period from the beginning of audio data <NUM> to the beginning of "I'll. " T6 represents the time period from the beginning of the audio data <NUM> to the end of "soon. " In some implementations, the device <NUM> may pre-process the audio data <NUM> so that T6 is at the end of the last word. In other words, any periods of silence after the last word are removed from the audio data <NUM>. In some implementations, the device <NUM> generates the timing data <NUM> while generating the transcription <NUM>. In some implementations, instead of device <NUM> generating the timing data <NUM>, the device <NUM> provides the audio data <NUM> to a server. The server generates the timing data <NUM> using a process that is similar to device's <NUM> process of generating the timing data <NUM>. The server may then provide the timing data <NUM> to the device <NUM>.

In some implementations, the device <NUM> may display an interface that provides the transcription <NUM> and allows the user to select different words of the transcription <NUM>. Upon selection of each word, the device <NUM> may play the corresponding audio data for the selected word. Doing so will allow the user to verify that the audio data for each word was properly matched to each transcribed word. For example, the device <NUM> may display "Text Mom I'll be home soon. " The user may select the word "home," and in response to the selection, the device <NUM> may play the audio data <NUM> between T4 and T5. The user may also be able to select more than one word at a time. for example, the user may select "text mom. " In response, the device <NUM> may play the audio data <NUM> between T0 and T2. In the case of errors, the user may request that the device generate the timing data <NUM> again for the whole transcription <NUM> or only for words selected by the user.

The device <NUM>, in some implementations, analyzes the transcription <NUM> and classifies portions of the transcription <NUM> as the voice command trigger term, the object of the voice command trigger term, or the recipient. The voice command trigger term is the portion of the transcription <NUM> that instructs the device <NUM> to perform a particular action. For example, the voice command trigger term may be "text," "send a message," "set an alarm," or "call. " The object of the voice command trigger term is the portion of the transcription <NUM> that instructs the device <NUM> to perform the particular action on the object. For example, the object may be a message, a time, or a date. The recipient instructs the device <NUM> to send the object or perform the particular action on the recipient. For example, the recipient may be "mom," Alice," or "Bob. " In some instances, a transcription may only include a voice command trigger term and a recipient, for example, "call Alice. " In other instances, a transcription may only include a voice command trigger term and an object of the voice command trigger term, for example, "set an alarm for <NUM> AM. " In the example shown in <FIG>, the device <NUM>, analyzes transcription <NUM> "text mom I'll be home soon," and classifies the term "text" as the voice command trigger term <NUM>, the term "mom" as the recipient <NUM>, and the message "I'll be home soon" as the object of the voice command trigger term <NUM>. The recipient <NUM> includes a phone number for "mom" based on the device <NUM> accessing the contacts data of the user <NUM>. In some implementations, a server analyzes and classifies the transcription <NUM>. The server may be the same server, or group of servers, that generated the timing data <NUM> and transcription <NUM>.

With the portion of the transcription <NUM> identified as the voice command trigger term <NUM> and the object of the voice command trigger term <NUM>, the device <NUM> provides the timing data <NUM>, the audio data <NUM>, and the voice command trigger term <NUM> and the object of the voice command trigger term <NUM> to the audio slicer <NUM>. The audio slicer <NUM> compares the voice trigger term <NUM> to a group of voice command trigger terms <NUM> for which audio data of the object of the voice command trigger term is provided to the recipient. Some examples <NUM> of voice command trigger terms <NUM> for which audio data of the object of the voice command trigger term is provided to the recipient include "text" and "send a message. " For "text" and "send a message" the transcription of the message and the audio data of the message are transmitted to the recipient. Another example <NUM> of voice command trigger terms <NUM> for which audio data of the object of the voice command trigger term is provided to the recipient includes "order a pizza. " For "order a pizza," the pizza shop may benefit from an audio recording of the order in instances where the utterance was transcribed incorrectly. As illustrated in <FIG>, the device <NUM> accesses the group of voice command trigger terms <NUM> and identifies the voice command trigger term <NUM> "text" as a voice command trigger term for which audio data of the object of the voice command trigger term is provided to the recipient. The group of voice command trigger terms <NUM> may be stored locally on the device <NUM> and updated periodically by either the user <NUM> or an application update. As illustrated in <FIG>, the group of voice command trigger terms <NUM> may also be stored remotely and accessed through a network <NUM>. In this instance, the group of voice command trigger terms <NUM> may be updated periodically by the developer of the application that sends audio data and a transcription of the audio data.

If device <NUM> determines that the voice command trigger term <NUM> matches one of the terms in the group of voice command trigger terms <NUM> for which audio data of the object of the voice command trigger term is provided to the recipient, and that the object of the voice command trigger term was spoken in a different language than the trigger term, then the audio slicer <NUM> isolates the audio data corresponding to the object of the voice command trigger term <NUM> using the timing data <NUM>. Because the timing data <NUM> identifies the start of each word in the audio data <NUM>, the audio slicer is able to match the words of the object of the voice command trigger term <NUM> to the corresponding times of the timing data <NUM> and isolate only that portion of the audio data <NUM> to generate audio data of the object of the voice command trigger term <NUM>. In the example shown in <FIG>, the audio slicer <NUM> receives data indicating the object of the voice command trigger term <NUM> as "I'll be home soon. " The audio slicer <NUM> identifies the portion of audio data <NUM> that corresponds to "I'll be home soon" is between T2 and T6. The audio slicer <NUM> removes the portion of the audio data <NUM> before T2. If the audio data <NUM> were to include any data after T6, then the audio slicer would remove that portion also. The audio slicer <NUM> isolates the message audio of "I'll be home soon" as the audio data corresponding to the object of the voice command trigger term <NUM>. Upon isolating the message audio, the device <NUM> may display a user interface that includes a play button for the user to listen to the isolated audio data.

With the audio data corresponding to the object of the voice command trigger term <NUM> isolated, the device <NUM> generates the data structure <NUM> based on the data <NUM>. The data structure <NUM> includes the transcription of the object of the voice command trigger term <NUM> and the corresponding audio data <NUM> that the audio slicer <NUM> isolated. In <FIG>, the data structure <NUM> includes the transcription "I'll be home soon" and the corresponding audio data. The device <NUM> transmits the data structure <NUM> to the device <NUM>. When the user <NUM> opens the message that includes the data structure <NUM>, the transcription of the object of the voice command trigger term <NUM> appears on the display of the device <NUM> and the audio data <NUM> plays. In some implementations, the audio data <NUM> plays automatically upon opening the message. In some implementations, the audio data <NUM> plays in response to a user selection of a play button or selecting the transcription of the object of the voice command trigger term <NUM> on the display. In some implementations, the audio data <NUM> may be included in an audio notification that the device <NUM> plays in response to receiving the data structure <NUM>.

In some implementations, the device <NUM> may provide the user <NUM> with various options when generating the data structure <NUM>. For example, the device <NUM> may, at any point after receiving the audio data of the utterance <NUM>, provide an option to the user to send audio data along with the transcription of the utterance. For example, as illustrated in user interface <NUM>, the device <NUM> displays a prompt <NUM> with selectable buttons <NUM>, <NUM>, and <NUM>. Selecting button <NUM> causes the recipient to only receive a transcription of the message. Selecting button <NUM> causes the recipient to receive only the audio of the message. Selecting button <NUM> causes the recipient to receive both the transcription and the audio. The device <NUM> may transmit the selection to a server processing the audio data of the utterance <NUM>. In some implementations, the device processing the utterance <NUM> does not perform or stops performing unnecessary processing of the utterance <NUM>. For example, the device <NUM> or server may stop or not generating timing data <NUM> if the user selects option <NUM>.

The device <NUM> may present the user interface <NUM> to send audio data upon matching the voice command trigger term <NUM> to a term in group of voice command trigger terms <NUM>. In some implementations, the user <NUM> may select particular recipients that should receive audio data and the transcription of the audio data. In this instance, the device <NUM> may not prompt the user to send the audio data and instead check the settings for the recipient. If the user <NUM> indicated that the recipient should receive audio data, then the device <NUM> generates and transmits the data structure <NUM>. If the user <NUM> indicated that the recipient should not receive audio data, then the device <NUM> only sends the transcription <NUM>.

In some implementations, the user <NUM> may provide feedback through the device <NUM>. The feedback may include an indication that the user wishes to continue to receive audio data with future messages or an indication that the user wishes to not receive audio data with future messages. For example, the user <NUM> may open the message that includes the data structure <NUM> on the device <NUM>. The device <NUM> may display an option that the user <NUM> can select to continue receiving audio data, if the audio data is available, and an option that the user <NUM> can select to no longer receive audio data. Upon selection, the device <NUM> may transmit the response to the device <NUM>. The device <NUM> may update the settings for user <NUM> automatically, or may present the information to the user <NUM> and the user <NUM> manually change the settings for user <NUM>. In another example, the user may open a message that only includes the transcription <NUM>. The device <NUM> may display an option that the user <NUM> can select to begin receiving audio data, if the audio data is available, and an option that the user <NUM> can select to not receive audio data with future messages. Similarly, upon selection, the device <NUM> may transmit the response to the device <NUM>. The device <NUM> may update the settings for user <NUM> automatically, or may present the information to the user <NUM> and the user <NUM> manually change the settings for user <NUM>.

In some implementations, the some or all of the actions performed by the device <NUM> are performed by a server. The device <NUM> receives the audio data <NUM> from the user <NUM> when the user <NUM> speaks the utterance <NUM>. The device <NUM> provides the audio data <NUM> to a server that processes the audio data <NUM> using a similar process as the one performed by the device <NUM>. The server may provide the transcription <NUM>, timing data <NUM>, classification data, and other data to the device <NUM> so that the user <NUM> may provide feedback regarding the transcription <NUM> and the timing data <NUM>. The device <NUM> may then provide the feedback to the server.

<FIG> illustrates an example system <NUM> combining audio data and a transcription of the audio data into a data structure. The system <NUM> may be implemented on a computing device such as the device <NUM> in <FIG>. The system <NUM> includes an audio subsystem <NUM> with a microphone <NUM> to receive incoming audio when a user speaks an utterance. The audio subsystem <NUM> converts audio received through the microphone <NUM> to a digital signal using the analog-to-digital converter <NUM>. The audio subsystem <NUM> also includes buffers <NUM>. The buffers <NUM> may store the digitized audio, e.g., in preparation for further processing by the system <NUM>. In some implementations, the system <NUM> is implemented with different devices. The audio subsystem <NUM> may be located on a client device, e.g., a mobile phone, and the modules located on server <NUM> that may include one or more computing devices. The contacts <NUM> may be located on the client device or server <NUM> or both.

In some implementations, the audio subsystem <NUM> may include an input port such as an audio jack. The input port may be connected to, and receive audio from, an external device such as an external microphone, and be connected to, and provide audio to, the audio subsystem <NUM>. In some implementations, the audio subsystem <NUM> may include functionality to receive audio data wirelessly. For example, the audio subsystem may include functionality, either implemented in hardware or software, to receive audio data from a short range radio, e.g., Bluetooth. The audio data received through the input port or through the wireless connection may correspond to an utterance spoken by a user.

The system <NUM> provides the audio data processed by the audio subsystem <NUM> to the speech recognizer <NUM>. The speech recognizer <NUM> is configured to identify the terms in the audio data. The speech recognizer <NUM> may user various techniques and models to identify the terms in the audio data. For example, the speech recognizer <NUM> may use one or more of an acoustic model, a language model, hidden Markov models, or neural networks. Each of these may be trained using data provided by the user and using user feedback provided during the speech recognition process and the process of generating the timing data <NUM>, both of which are described above.

During or after the speech recognition process, the speech recognizer <NUM> may use the clock <NUM> to identify the beginning points in the audio data where each term begins. The speech recognizer <NUM> may set the beginning of the audio data to time zero and the beginning of each word or term in the audio data is associated with an elapsed time from the beginning of the audio data to the beginning of the term. For example, with the audio data that corresponds to "send a message to Alice I'm running late," the term "message" may be paired with a time period that indicates an elapsed time from the beginning of the audio data to the beginning of "message" and an elapsed time from the beginning of the audio data to the beginning of "to.

In some implementations, the speech recognizer <NUM> may provide the identified terms to the user interface generator <NUM>. The user interface generator <NUM> may generated an interface that includes the identified terms. The interface may include the selectable options to play the audio data that corresponds to each of the identified terms. Using the above example, the user may select to play the audio data corresponding to "Alice. " Upon receiving the selection, the system <NUM> plays the audio data that corresponds to the beginning of "Alice" to the beginning of "I'm. " The user may provide feedback if some of the audio data does not correspond to the proper term. For example, the user interface generator may provide an audio editing graph or chart of the audio data versus time where the user can select the portion that corresponds to a particular term. This may be helpful when the audio data that the system identified as corresponding to "running" actually corresponds to only "run. " The user may then manually extend the corresponding audio portion to capture the "ing" portion. When the user provides feedback in this manner or in any other feedback mechanism, the speech recognizer may user the feedback to train the models.

In some implementations, the speech recognizer <NUM> may be configured to recognize only one or more languages. The languages may be based on a setting selected by the user in the system. For example, the speech recognizer <NUM> may be configured to only recognize English. In this instance, when a user speaks Spanish, the speech recognizer still attempts to identify English words and sounds that correspond to the Spanish utterance. A user may speak "text Bob se me hace tarde" ("text Bob I'm running late") and the speech recognizer may transcribe "text Bob send acetone. " If the speech recognizer is unsuccessful at matching the Spanish portion of the utterance to "send acetone" transcription, then user may use the audio chart to match the audio data that corresponds to "se me" to the "send" transcription and the audio data that corresponds to "hace tarde" to the "acetone" transcription.

The speech recognizer <NUM> provides the transcription to the transcription term classifier <NUM>. The transcription term classifier <NUM> classifies each word or group of words as a voice command trigger term, an object of a voice command trigger term, or a recipient. In some implementations, the transcription term classifier <NUM> may be unable to identify a voice command trigger term. In this case, the system <NUM> may display an error to the user can request that the user speak the utterance again or speak an utterance with a different command. As describe above as related to <FIG>, some voice command trigger terms may not require an object or a recipient. In some implementations, the transcription term classifier <NUM> may access a list of voice command trigger terms that are stored either locally on the system or stored remotely to assist in identifying voice command trigger terms. The list of voice command trigger terms includes a list of voice command trigger terms for which the system is able to perform an action. In some implementations, the transcription term classifier <NUM> may access a contacts list that is stored either locally on the system or remotely to assist in identifying recipients. In some instances, the transcription term classifier <NUM> identifies the voice command trigger term and the recipient and there are still terms remaining in the transcription. In this case, the transcription term classifier <NUM> may classify the remaining terms as the object of the voice command trigger term. This may be helpful when the object was spoken in another language. Continuing with the "text Bob se me hace tarde" utterance example where the transcription was "text Bob send acetone. " The transcription term classifier <NUM> may classify the "send acetone" portion as the object after classifying "text" as the voice command trigger term and "Bob" as the recipient.

The speech recognizer <NUM> provides the transcription and the audio data to the language identifier <NUM>. In some implementations, the speech recognizer <NUM> may provide confidence scores for each of the transcribed terms. The language identifier <NUM> may compare the transcription, the audio data, and the confidence scores to determine a language or languages of the utterance. Low confidence scores may indicate the presence of a language other than the language used by the speech reconsider <NUM>. The language identifier <NUM> may receive a list of possible languages that the user inputs through the user interface. For example, a user may indicate that that the user speaks in English and Spanish, then the language identifier <NUM> may label portions of the transcription as either English or Spanish. In some implementations, the user may indicate to the system contacts who are likely to receive message in languages other than the primary language of the speech recognizer <NUM>. For example, a user may indicate that the contact Bob is likely to receive messages in Spanish. The language identifier <NUM> may use this information and the confidence scores to identify the "send acetone" portion of the above example as Spanish.

The audio slicer <NUM> receives data from the language identifier <NUM>, the transcription term classifier <NUM> and the speech recognizer <NUM>. The language identifier <NUM> provides data indicating the languages identifies in the audio data. The transcription term classifier <NUM> provides data indicating the voice command trigger term, the object of the voice command trigger term, and the recipient. The speech recognizer provides the transcription, the audio data, and the timing data. The audio slicer <NUM> isolates the object of the voice command trigger term by removing the portions of the audio data that do not correspond to the object of the voice command trigger term. The audio slicer <NUM> isolates the object using the timing data to identify the portions of the audio data that do not correspond to the object of the voice command trigger term.

The audio slicer <NUM> determines whether to isolate the object of the voice command trigger term based on a number of factors. One of those factors is the comparison of the voice command trigger term to the group of voice command trigger terms <NUM>. If the voice command trigger term matches one in the group of voice command trigger terms <NUM>, then the audio slicer isolates the audio data of the object of the voice command trigger term.

Another factor may be based on input received from the user interface. The audio slicer <NUM> may provide data to the user interface generator <NUM> to display information related to isolating the audio data of the object of the voice command trigger term. For example, the user interface generator <NUM> may display a prompt asking the user whether the user wants to send audio corresponding to "send acetone. " The user interface may include an option to play the audio data corresponding to "send acetone. " In this instance, the audio data may isolate the audio data of the object of the voice command trigger term on a trial basis and pass the isolated audio data to the next stage if the user requests.

According to the invention, another factor is based on the languages identified by the language identifier <NUM>. A user may request that the audio slicer <NUM> isolate the audio data of the object of the voice command trigger term if the user speaks the object of the voice command trigger term in a different language than the other portions of the utterance, such as the voice command trigger term. For example, when a user speaks "text Bob se me hace tarde" and the language identifier <NUM> identifies the languages as Spanish and English, the audio slicer <NUM> may isolate the audio data of the object of the voice command trigger term in response to a setting inputted by the user to isolate the audio data of the object of the voice command trigger term with the object is in a different language than the trigger term or when the object is in a particular language, such as Spanish.

Another factor may be based on the recipient. A user may request that the audio slicer <NUM> isolate the audio data of the object of the voice command trigger term if the recipient is identified as one to receive audio data of the object. For example, the user may provide, through a user interface, instructions to provide the recipient Bob with the audio data of the object. Then if the audio slicer <NUM> receives a transcription with the recipient identified as Bob, the audio slicer <NUM> isolates the object of the voice command trigger term and provides the audio data to the next stage.

In some implementations, the audio slicer <NUM> may isolate the audio data of the object of the voice command trigger term based on both the identified languages of the audio data and the recipient. For example, a user may provide, through a user interface, instructions to provide the recipient Bob with the audio data of the object, if the object is in a particular language, such as Spanish. Using the same example, the audio slicer would isolate "se me hace tarde" because the recipient is Bob and "se me hace tarde" is Spanish.

In some implementations, the audio slicer <NUM> may allow the user to listen to the audio data of the object of the voice command trigger term before sending. The audio slicer <NUM> may provide the transcription of the object of the voice command trigger term and the audio data of the object of the voice command trigger term to the user interface generator <NUM>. The user interface generator <NUM> may provide an interface that allows the user to select the transcription of the object to hear the corresponding audio data. The interface may also provide the user the option of sending the audio data of the object to the recipient that may also be provided on the user interface.

The audio slicer <NUM> provides the transcription of the object of the voice command trigger term, the audio data of the object of the voice command trigger term, the recipient, and the voice command trigger term to the data structure generator <NUM>. The data structure generator <NUM> generates a data structure, according to the voice command trigger term, that is ready to send to the recipient and includes the audio data and the transcription of the object of the voice command trigger term. The data structure generator <NUM> accesses the contacts list <NUM> to identify a contact number or address of the recipient. Following the same example, the data structure generator <NUM>, by following the instructions corresponding to the "text" voice command trigger term, generates a data structure that includes the transcription and audio data of "se me hace tarde" and identifies the contact information for the recipient Bob in the contacts list <NUM>. The data structure generator <NUM> provides the data structure to the portion of the system that sends the data structure to Bob's device.

In some implementations, the speech recognizer <NUM>, clock <NUM>, language identifier <NUM>, transcription term classifier <NUM>, audio slicer <NUM>, voice command trigger terms <NUM>, and data structure generator <NUM> are located on a server <NUM>, which may include one or more computing devices. The audio subsystem <NUM> and contacts <NUM> are located on a user device. In some implementations, the contacts <NUM> may be located on both the user device and the server <NUM>. In some implementations, the user interface generator <NUM> is located on the user device. In this instance the server <NUM> provides data for display on the user device to the user interface generator <NUM> which then generates a user interface for the user device. The user device and the server <NUM> communicate over a network, for example, the internet.

<FIG> illustrates an example process <NUM> for combining audio data and a transcription of the audio data into a data structure. In general, the process <NUM> generates a data structure that includes a transcription of an utterance and audio data of the utterance and transmits the data structure to a recipient. The process <NUM> will be described as being performed by a computer system comprising at one or more computers, for example, the devices <NUM>, system <NUM>, or server <NUM> as shown in <FIG> and <FIG>, respectively.

The system receives audio data that corresponds to an utterance (<NUM>). For example, the system may receive audio data from a user speaking "send a message to Alice that the check is in the mail. " The system generates a transcription of the utterance (<NUM>). In some implementations, while or after the system generates the transcription of the utterance, the system generates timing data for each term of the transcription. The timing data may indicate the elapsed time from the beginning of the utterance to the beginning of the each term. For example, the timing data for "message" would be the time from the beginning of the utterance to the beginning of "message.

The system classifies a first portion of the transcription as a voice command trigger term and a second portion of the transcription as an object of the voice command trigger term (<NUM>). In some implementations, the system classifies a third portion of the transcription as the recipient. Following the same example, the system classifies "send a message to" as the voice command trigger term. The system also classifies "Alice" as the recipient. In some implementations, the system may classify "that" as part of the voice command trigger term, such that the voice command trigger term is "send a message to. " In this instance, the system classifies the object of the voice command trigger term as "the check is in the mail. " As illustrated in this example, the voice command trigger term is a command to send a message, and the object of the voice command trigger term is the message.

The system determines that the voice command trigger term matches a voice command trigger term for which a result of processing is to include both a transcription of an object of the voice command trigger term and audio data of the object of the voice command trigger term in a generated data structure, and that the object of the voice trigger term was spoken in a different language than the trigger term (<NUM>). For example, the system may access a group of voice command trigger terms that when processed cause the system to send both the audio data and the transcription of the object of the voice command trigger. Following the above example, if the group includes the voice command trigger term, "send a message to," then the system identifies a match.

The system isolates the audio data of the object of the voice command trigger term (<NUM>). In some implementations, the system isolates the audio data using the timing data. For example, the system removes the audio data from before "the check" and after "mail" by matching the timing data of "the check" and "mail" to the audio data. In some implementations, the system identifies the language of the utterance or of a portion of the utterance. Based on the language, the system may isolate the audio data of the object of the voice command trigger term. For example, the system may isolate the audio data if a portion of the utterance was spoken in Spanish.

The system generates a data structure that includes the transcription of the object of the voice command trigger term and the audio data of the object of the voice command trigger term (<NUM>). The system may generate the data structure based on the voice command trigger term. For example, with a voice command trigger term of "send a message to," the data structure may include the transcription and audio data of "the check is in the mail. " The system may then send the data structure to the recipient. In some implementations, the system may generate the data structure based on the language of the utterance or of a portion of the utterance. For example, the system may generate the data structure that includes the transcription and audio data of the object of the voice command trigger term based on the object being spoken in Spanish.

In some implementations, the system may generate a user interface that allows the user to instruct the system to send both the transcription and the audio data of the object of the voice command trigger term to the recipient. In this instance, the system may respond the instruction by isolating the voice command trigger term or generating the data structure.

<FIG> shows an example of a computing device <NUM> and a mobile computing device <NUM> that can be used to implement the techniques described here. The mobile computing device <NUM> is intended to represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smart-phones, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to be limiting.

The computing device <NUM> includes a processor <NUM>, a memory <NUM>, a storage device <NUM>, a high-speed interface <NUM> connecting to the memory <NUM> and multiple high-speed expansion ports <NUM>, and a low-speed interface <NUM> connecting to a low-speed expansion port <NUM> and the storage device <NUM>. Each of the processor <NUM>, the memory <NUM>, the storage device <NUM>, the high-speed interface <NUM>, the high-speed expansion ports <NUM>, and the low-speed interface <NUM>, are interconnected using various busses, and may be mounted on a common motherboard or in other manners as appropriate. The processor <NUM> can process instructions for execution within the computing device <NUM>, including instructions stored in the memory <NUM> or on the storage device <NUM> to display graphical information for a GUI on an external input/output device, such as a display <NUM> coupled to the high-speed interface <NUM>. Also, multiple computing devices may be connected, with each device providing portions of the necessary operations (e.g., as a server bank, a group of blade servers, or a multi-processor system).

The high-speed interface <NUM> manages bandwidth-intensive operations for the computing device <NUM>, while the low-speed interface <NUM> manages lower bandwidth-intensive operations. Such allocation of functions is an example only. In some implementations, the high-speed interface <NUM> is coupled to the memory <NUM>, the display <NUM> (e.g., through a graphics processor or accelerator), and to the high-speed expansion ports <NUM>, which may accept various expansion cards. In the implementation, the low-speed interface <NUM> is coupled to the storage device <NUM> and the low-speed expansion port <NUM>. The low-speed expansion port <NUM>, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet) may be coupled to one or more input/output devices, such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router, e.g., through a network adapter.

Alternatively, components from the computing device <NUM> may be combined with other components in a mobile device, such as a mobile computing device <NUM>.

The memory <NUM> stores information within the mobile computing device <NUM>. An expansion memory <NUM> may also be provided and connected to the mobile computing device <NUM> through an expansion interface <NUM>, which may include, for example, a SIMM (Single In Line Memory Module) card interface. The expansion memory <NUM> may provide extra storage space for the mobile computing device <NUM>, or may also store applications or other information for the mobile computing device <NUM>. Specifically, the expansion memory <NUM> may include instructions to carry out or supplement the processes described above, and may include secure information also. Thus, for example, the expansion memory <NUM> may be provide as a security module for the mobile computing device <NUM>, and may be programmed with instructions that permit secure use of the mobile computing device <NUM>.

The memory may include, for example, flash memory and/or NVRAM memory (non-volatile random access memory), as discussed below. In some implementations, instructions are stored in an information carrier. The instructions can also be stored by one or more storage devices, such as one or more computer- or machine-readable mediums (for example, the memory <NUM>, the expansion memory <NUM>, or memory on the processor <NUM>). In some implementations, the instructions can be received in a propagated signal, for example, over the transceiver <NUM> or the external interface <NUM>.

The mobile computing device <NUM> may communicate wirelessly through the communication interface <NUM>, which may include digital signal processing circuitry where necessary. The communication interface <NUM> may provide for communications under various modes or protocols, such as GSM voice calls (Global System for Mobile communications), SMS (Short Message Service), EMS (Enhanced Messaging Service), or MMS messaging (Multimedia Messaging Service), CDMA (code division multiple access), TDMA (time division multiple access), PDC (Personal Digital Cellular), WCDMA (Wideband Code Division Multiple Access), CDMA2000, or GPRS (General Packet Radio Service), among others. Such communication may occur, for example, through the transceiver <NUM> using a radio-frequency. In addition, short-range communication may occur, such as using a Bluetooth, WiFi, or other such transceiver. In addition, a GPS (Global Positioning System) receiver module <NUM> may provide additional navigation- and location-related wireless data to the mobile computing device <NUM>, which may be used as appropriate by applications running on the mobile computing device <NUM>.

Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

Claim 1:
A computer-implemented audio slicing method comprising:
receiving (<NUM>) audio data that corresponds to an utterance;
generating (<NUM>) a transcription of the utterance;
classifying (<NUM>) a first portion of the transcription as a voice command trigger term and a second portion of the transcription as an object of the voice command trigger term;
identifying a language of the voice command trigger term and a language of the object of the voice command trigger term;
determining (<NUM>) whether to isolate the object of the voice command trigger term based on two factors, the first factor being that the voice command trigger term matches a voice command trigger term for which a result of processing is to include both a transcription of an object of the voice command trigger term and audio data of the object of the voice command trigger term in a generated data structure and the second factor being that the object of the voice command trigger term was spoken in a different language than other portions of the utterance;
based on the result of the determining (<NUM>), isolating (<NUM>), from the audio data corresponding to the utterance, the audio data of the object of the voice command trigger term;
generating (<NUM>) a data structure that includes the transcription of the object of the voice command trigger term and the audio data of the object of the voice command trigger term; and
transmitting the data structure to a recipient.