Patent Description:
Mobile computing devices, such as mobile phones, have available auxiliary devices which can offer additional functionality, see, for example, <CIT> or <CIT>. For example, some earbuds offer a translation experience where a user can touch one earbud, speak in a first language, and the spoken phrase is translated into a second language. The user may hear the translated phrase, for example, from speakers in the mobile computing device. Similarly, phrases spoken into the mobile phone in the second language may be translated to the first language and output through the earbuds. This exchange is performed serially, such that each participant must wait for the other's speech to be completed, translated, and output through speakers of the mobile computing device or auxiliary device before speaking a response. The serial nature of the exchange adds significant delays to conversational patterns, and introduces uncomfortable social cues. The interaction flow requires both parties to pay careful attention to which person is speaking, whether the translation is being output on each device, and as a result is unintuitive.

The proposed solution relates to a mobile computing device of claim <NUM>, a system of claim <NUM>, a method of claim <NUM>, and a non-transitory computer-readable storage medium of claim <NUM>.

The present disclosure provides for a more natural, conversational exchange between a user and a foreign language speaker using translation features of mobile devices and auxiliary devices. In particular, microphones of both the mobile device and the auxiliary device are always listening for speech input. The mobile and auxiliary device may also determine whether the speech input is from the user or the foreign language speaker. The mobile device and auxiliary device can automatically determine when a spoken phrase is complete and ready for translation, send the spoken phrase for translation, and immediately begin listening again. In this regard, acknowledgements or other social cues, such as "OK," "Yes," "Oh no," etc. may be captured, translated, and output throughout a conversation in response to the other user's translated speech, even if the other user is still speaking.

The present disclosure provides for an improved translation experience between a first user and a second user using a mobile computing device and an auxiliary device, such as a pair of earbuds. The first user may be, for example, a foreign language speaker, and the second user may be the owner of the mobile computing device and auxiliary device. Microphones on both the mobile device and the auxiliary device simultaneously capture input from the first user and the second user, respectively, rather than alternating between the mobile device and the auxiliary device. Each device may determine when to endpoint, or send a block of speech for translation, for example based on pauses in the speech. Each device may accordingly send the received speech up to the endpoint for translation and output, such that it is provided in a natural flow of communication. Listening by the microphones automatically resumes immediately after endpointing, and therefore speech will not be lost.

<FIG> is a pictorial diagram of an example system in use. A first user <NUM> provides speech input <NUM>. The speech input <NUM> may be in a language foreign to a second user <NUM>. The speech input <NUM> is received at a mobile computing device <NUM> of the second user <NUM>. The second user <NUM> also provides second speech input <NUM> in a second language. The second speech input <NUM> may be received by both the mobile device <NUM> and by an auxiliary device <NUM> worn by the second user <NUM>. According to some examples, the auxiliary device <NUM> may be intended to primarily receive the second speech input <NUM>, while the mobile device <NUM> is intended to primarily receive the first speech input <NUM>. However, even in this example, the mobile device <NUM> may also pick up some of the second speech input <NUM> as a side effect, and may filter out the second speech input <NUM>. For example, the mobile device <NUM> may determine which input is received from the first user <NUM> and which input is received from the second user <NUM>, and translate the speech from the first user <NUM>. The auxiliary device <NUM> may also determine that received input is from the second user <NUM>, and send that input to the mobile device for translation. Each of the mobile device <NUM> and the auxiliary device <NUM> remains listening throughout an exchange between the first user <NUM> and the second user <NUM>, and provides translated output at times relevant to one another.

In the example shown, the auxiliary device <NUM> is a pair of wireless earbuds. However, it should be understood that the auxiliary device <NUM> may be any of a number of different types of auxiliary devices. For example, the auxiliary device <NUM> may be a pair of wired earbuds, a headset, a head-mounted display, a smart watch, a mobile assistant, etc..

The mobile computing device <NUM> may be, for example, a mobile phone, tablet, laptop, gaming system, or any other type of mobile computing device. In some examples, the mobile computing device <NUM> may be coupled to a network, such as a cellular network, wireless Internet network, etc. Translations capabilities may be stored on the mobile computing device <NUM>, or accessed from a remote source by the mobile computing device <NUM>. For example, the mobile device <NUM> may interface with a cloud computing environment in which the speech translations from a first language to a second language are performed and provided back to the mobile device <NUM>.

<FIG> is pictorial diagram of the example system including auxiliary device <NUM> and mobile device <NUM>, and illustrates an exchange of signals between the mobile device <NUM> and the auxiliary device <NUM>.

In this example, the auxiliary device <NUM> is illustrated as a pair of earbuds, which may include, for example, speaker portion <NUM> adjacent an inner ear-engaging surface <NUM>, and input portion <NUM> adjacent an outer surface. In some examples, a user may enter input by pressing the input portion <NUM> while speaking, or by tapping the input portion <NUM> prior to speaking. In other examples, manual input by a user is not required, and the user may simply begin speaking. The user's speech may be received by a microphone in the earbuds (not shown) or in the mobile device <NUM>. The user may hear translated speech from another person through the speaker portion <NUM>.

The auxiliary device <NUM> is wirelessly coupled to the mobile device <NUM>. The wireless connections between the devices may include, for example, a short range pairing connection, such as Bluetooth. Other types of wireless connections are also possible.

In some examples, such as shown in <FIG>, different wireless protocols may be used for transmission of information from the earbuds <NUM> to the mobile device <NUM>, than for transmission of information in a reverse path from the mobile device <NUM> to the earbuds <NUM>. Wireless pairing between mobile and auxiliary devices may typically use an HFP/SCO channel, which provides for lower quality two-way communication. The present example provides for replacing the HFP/SCO channel with two concurrent channels. In some examples, the concurrent channels may be of different types. A first channel <NUM>, used for transmitting information from the mobile device <NUM> to the auxiliary device <NUM>, may be a high quality one-way audio channel, such as A2DP. A second channel <NUM>, used for transmitting information from the auxiliary device <NUM> to the mobile device <NUM>, may be, for example, an RFComm channel. This allows for the mobile device <NUM> to process two streams of input audio at once. In this regard, microphones on both the mobile device <NUM> and auxiliary device <NUM> may be used concurrently, with audio transmitted simultaneously in both directions. It should be understood that A2DP and RFComm are merely examples of the different types of channels that can be used. Moreover, in some examples the channels <NUM>, <NUM> may be independent channels of a same type, each providing one-way communication.

<FIG> provides an example block diagram of the auxiliary device <NUM> and the mobile device <NUM>. Each device includes one or more processors <NUM>, <NUM>, memory <NUM>, <NUM>, and other components typically present in mobile computing devices and auxiliary devices. While a number of components are shown, it should be understood that such components are merely non-limiting examples, and that other components may additionally or alternatively be included.

As mentioned above, the auxiliary device <NUM> can be any of various types of devices, such as earbuds, head-mounted device, smart watch, etc. The mobile device <NUM> can also take a variety of forms, such as smart phone, tablet, laptop, game console, etc..

The one or more processors <NUM>, <NUM> may be any conventional processors, such as commercially available microprocessors. Alternatively, the one or more processors may be a dedicated device such as an application specific integrated circuit (ASIC) or other hardware-based processor. Although <FIG> functionally illustrates the processor, memory, and other elements of auxiliary device <NUM> and mobile computing device <NUM> as being within the same respective blocks, it will be understood by those of ordinary skill in the art that the processor or memory may actually include multiple processors or memories that may or may not be stored within the same physical housing. Similarly, the memory may be a hard drive or other storage media located in a housing different from that of the auxiliary device <NUM> or mobile computing device <NUM>. Accordingly, references to a processor or computing device will be understood to include references to a collection of processors or computing devices or memories that may or may not operate in parallel.

Memory <NUM> may store information that is accessible by the processors <NUM>, including instructions <NUM> that may be executed by the processors <NUM>, and data <NUM>. The memory <NUM> may be of a type of memory operative to store information accessible by the processors <NUM>, including a non-transitory computer-readable medium, or other medium that stores data that may be read with the aid of an electronic device, such as a hard-drive, memory card, read-only memory ("ROM"), random access memory ("RAM"), optical disks, as well as other write-capable and read-only memories. The subject matter disclosed herein may include different combinations of the foregoing, whereby different portions of the instructions <NUM> and data <NUM> are stored on different types of media.

Data <NUM> may be retrieved, stored or modified by processors <NUM> in accordance with the instructions <NUM>. For instance, although the present disclosure is not limited by a particular data structure, the data <NUM> may be stored in computer registers, in a relational database as a table having a plurality of different fields and records, XML documents, or flat files. The data <NUM> may also be formatted in a computer-readable format such as, but not limited to, binary values, ASCII or Unicode. By further way of example only, the data <NUM> may be stored as bitmaps comprised of pixels that are stored in compressed or uncompressed, or various image formats (e.g., JPEG), vector-based formats (e.g., SVG) or computer instructions for drawing graphics. Moreover, the data <NUM> may comprise information sufficient to identify the relevant information, such as numbers, descriptive text, proprietary codes, pointers, references to data stored in other memories (including other network locations) or information that is used by a function to calculate the relevant data.

The instructions <NUM> may be executed to facilitate translations performed by a mobile computing device. For example, the instructions <NUM> provide for listening for and receiving user speech, for example, through microphone <NUM>. The microphone <NUM> may be beamformed, such that it is directed to receive audio coming from a direction of the user's mouth. In this regard, the auxiliary device <NUM> may recognize received speech as being that of the user, as opposed to a foreign language speaker that is not wearing the auxiliary device <NUM>.

The instructions <NUM> may further provide for detecting an endpoint in the received speech. For example, the endpoint may be automatically determined based on a pause in speech, key words, intonation, inflection, or any of a combination of these or other factors. Once the endpoint is detected, the auxiliary device <NUM> may buffer the received speech while immediately resuming listening. In other examples, alternatively or additionally to buffering, the auxiliary device <NUM> may transmit the received speech to the mobile device <NUM> for translation. For example, the auxiliary device <NUM> may transmit the speech via an RFComm or other communication link, as discussed above in connection with <FIG>. In some examples, all endpointing may be performed by the mobile device <NUM>. In such examples, the auxiliary device <NUM> may stream all audio to the mobile device <NUM>, which handles all endpointing, voice recognition, and translation, either offline or online.

While the auxiliary device <NUM> is executing the instructions <NUM>, the mobile device <NUM> may also be executing instructions <NUM> stored in memory <NUM> along with data <NUM>. For example, similar to the auxiliary device <NUM>, the mobile device <NUM> may also include memory <NUM> storing data <NUM> and instructions <NUM> executable by the one or more processors <NUM>. The memory <NUM> may be any of a variety of types, and the data <NUM> may be any of a variety of formats, similar to the memory <NUM> and data <NUM> of the auxiliary device <NUM>. While the auxiliary device <NUM> is listening for and receiving speech from the user wearing the auxiliary device <NUM>, the mobile device <NUM> may be listening for and receiving speech as well through microphone <NUM>. The microphone <NUM> may not be beamformed, and may receive audio input from both the foreign language speaker (e.g., first user <NUM> of <FIG>) and the user of the auxiliary device <NUM> (e.g., second user <NUM> of <FIG>). Accordingly, the instructions <NUM> are configured for determining which user is providing speech input, such that some speech input (e.g., from the foreign language speaker) is translated for output through the auxiliary device <NUM> and other speech input (e.g., from the user) is filtered.

Any of a variety of voice recognition techniques may be used. As one embodiment, the mobile device <NUM> is configured to cross reference a volume level between the auxiliary device microphone <NUM> and the mobile device microphone <NUM> or is configured to cross reference waveforms of the second input with waveforms of the first input. If the sound received through microphone <NUM> is quiet and the sounds received through the microphone <NUM> is loud, then it may be determined that the foreign language speaker is providing speech input. Conversely, if the sounds received through both microphones <NUM>, <NUM> is loud, then it may be determined that the owner/user is speaking. As another example technique, a voice recognition unit may be used. The voice recognition unit may be trained to recognize a voice of the user/owner of the auxiliary device <NUM> and mobile device <NUM>. Accordingly, if the voice recognition unit detects the owner's voice, it ignores it. Similarly, the voice recognition unit may be trained to detect a language primarily spoken by the owner/user, and may filter out speech detected in that language. As yet another example technique, audio echo cancellation techniques may be used. For example, the mobile device <NUM> may listen to both microphone <NUM>, <NUM>, detect overlapping audio, and, recognize that the overlapping audio belongs to the owner. The overlapping audio may be detected by identifying similar waveforms or patterns of sound input, or detecting similar plosives or transient attacks. In some examples, any combination of the foregoing or other techniques may be used.

When the detected speech is from the foreign language speaker, the instructions <NUM> may further provide for continued listening until an endpoint is detected. As mentioned above, the endpoint may be detected based on a pause, keyword, inflection, or other factor. The received speech from the foreign language speaker is buffered, and the microphone <NUM> resumes listening.

The mobile device <NUM> is configured to perform translations of both foreign language speaker input received through the microphone <NUM>, as well as owner input received through communication from the auxiliary device <NUM>. Such translations are performed on the mobile device <NUM> itself, or (in embodiments not encompassed by the claims) may be performed using one or more remote computing devices, such as the cloud. For example, the mobile device <NUM> may upload speech for translation to a remote computing network which performs the translation, and receive a response including translated speech from the remote computing network. Translations of speech from the foreign language speaker may be provided to the auxiliary device <NUM> for output through output <NUM>. Translations of speech from the owner may be output through output <NUM> of mobile device <NUM>. The outputs <NUM>, <NUM> may each include, for example, one or more speakers adapted to provide audible output. In some examples, the outputs <NUM>, <NUM> may also include one or more other types, such as displays, tactile feedback, etc..

It should be understood that the auxiliary device <NUM> and mobile device <NUM> may each include other components which are not shown, such charging input for the battery, signals processing components, etc. Such components may also be utilized in execution of the instructions <NUM>, <NUM>.

<FIG> illustrates a method <NUM> executed by the mobile device <NUM>. In block <NUM>, the mobile device listens for and receives voice input. While the mobile device is intended to capture the voice of a secondary user, such as a foreign language speaker (e.g., user <NUM> of <FIG>), it may also capture voice input of a primary user (e.g., user <NUM> of <FIG>). The device may continually listen for and receive input without manual interaction, such as pressing buttons, etc..

In block <NUM>, the mobile device determines whether the received voice input is from the mobile device owner or the foreign language speaker. For example, the mobile device may use voice recognition, language recognition, etc. The the mobile device is configured to cross reference a volume of sound received at the mobile device with a volume of sound received at the auxiliary device and relayed to the mobile device, alternatively the mobile device is configured to cross reference waveforms of the second input with waveforms of the first input.

If in block <NUM> it is determined that the input is from the mobile device owner, the mobile device is configured to ignore the input. Accordingly, the method returns to block <NUM> to keep listening for input from the foreign language speaker. If, however, the input is determined to be from the foreign language speaker, the method proceeds to block <NUM>.

In block <NUM>, the mobile device determines whether an endpoint in speech is detected. For example, if there is a pause in the speech input for a predetermined period of time, such as half a second, one second, two seconds, etc., the mobile device may determine that an endpoint in speech has been reached. Other examples of detecting endpoints may include detecting changes in intonation or inflection, or detecting keywords. Detecting the endpoint helps to ensure proper translation of complete phrases. For example, translations of each individual word are typically inaccurate. By way of example only, while in English adjectives are typically spoken before their associated nouns, in Spanish the adjectives are spoken after the noun. By endpointing the speech after a complete statement, phrases may be translated as a whole and the words rearranged as appropriate for the translated language.

If no endpoint is detected, the mobile device keeps listening in block <NUM> and waiting for an endpoint. If an endpoint is detected, however, the mobile device buffers the received input in block <NUM>. It should be understood that the buffering may be performed while the input is received, prior to detection of the endpoint.

In block <NUM>, the mobile device translates the buffered voice input up until the endpoint. The translation may be performed at the mobile device, or through remote computing devices.

In block <NUM>, the translated input is provided to the auxiliary device for output. For example, the translated input may be provided through one of the two communication channels described in <FIG>.

Though not shown, the mobile device also receives speech from the auxiliary device. For example, the auxiliary device receives speech from the owner as described below in connection with <FIG>, and provides the received speech to the mobile device for translation. Once translated, the mobile device is configured to output the translated speech for the foreign language speaker. For example, the mobile device may emit an audible translation of the speech through a speaker for the foreign language speaker to hear, may provide a text translation of the speech on a display, etc..

<FIG> illustrates a method <NUM> performed by the auxiliary device for receiving translation. The method <NUM> may be performed by the auxiliary device contemporaneously with execution of the method <NUM> by the mobile device.

In block <NUM>, the auxiliary device listens for and receives voice input from the owner. Where a microphone in the auxiliary device is beamformed, it may determine that any input received is from the owner. In other examples, voice recognition techniques such as those described above may be used.

In block <NUM>, the auxiliary device determines whether an endpoint was reached. The endpoint may be automatically detected based on patterns in the speech. In other examples, the endpoint may be manually input by the owner, such as by pressing a button. If no endpoint is detected, the device continues listening in block <NUM> until an endpoint is detected. The received speech may be buffered at the auxiliary device during and/or after receipt. In some alternatives, rather than endpointing being performed by the auxiliary device, the auxiliary device streams the received audio continuously to the mobile device. The mobile device may run two voice recognizers simultaneously in order to detect voice and endpoint accordingly.

In block <NUM>, the received speech from the owner is transmitted to the mobile device for translation and output. For example, the speech may be sent through a second communication channel, as discussed above in connection with <FIG>. This speech may be sent concurrently with speech received (block <NUM>) from the mobile device for output (block <NUM>) by the auxiliary device.

By keeping microphones on both the auxiliary device and the mobile device continually open, speech from both the owner and the foreign language speaker is continually received. In this regard, the owner and foreign language speaker may have a more natural conversation, including interjections, affirmations, acknowledgements, etc., and without awkward pauses while waiting for translation. By automatic endpointing, phrases or other blocks of speech may be detected and translated without requiring manual input from a user. Moreover, voice recognition techniques may be used to determine which user is providing the input and thus how it should be translated. Accordingly, less user manipulation of devices is required. Rather, the users may converse naturally, and the auxiliary and mobile devices may automatically provide assistance, providing for a near real-time translation.

Claim 1:
A mobile computing device, comprising:
a microphone (<NUM>);
a communication interface configured to wirelessly communicate with an auxiliary device (<NUM>);
an output source (<NUM>); and
one or more processors (<NUM>) configured to:
receive, through the microphone (<NUM>), first voice input (<NUM>) from a first user (<NUM>);
detect an endpoint in the first voice input (<NUM>);
translate the received first voice input (<NUM>) up to the endpoint;
listen for subsequent input while translating; and
transmit the translated first voice input (<NUM>) to the auxiliary device (<NUM>) for output;
wherein the one or more processors (<NUM>) are further configured to:
distinguish first voice input (<NUM>) from the first user (<NUM>) from second voice input (<NUM>) from a second user (<NUM>);
receive, from the auxiliary device (<NUM>), the second voice input (<NUM>);
cross reference, for determining which of the first user (<NUM>) or the second user (<NUM>) is providing input:
(a) a volume level of the second voice input (<NUM>) with a volume level of the first voice input (<NUM>), or
(b) waveforms of the second voice input (<NUM>) with waveforms of the first voice input (<NUM>); and
filter out and thus ignore the second voice input (<NUM>) for translation, if it is determined that the second voice input (<NUM>) is provided by the second user (<NUM>).