Techniques for dynamic auditory phrase completion

Embodiments of the present disclosure set forth a computer-implemented method comprising detecting an initial phrase portion included in a first auditory signal generated by a user, identifying, based on the initial phrase portion, a supplemental phrase portion that complements the initial phrase portion to form a complete phrase, and providing a command signal that drives an output device to generate an audio output corresponding to the supplemental phrase portion.

BACKGROUND

Field of the Various Embodiments

Embodiments disclosed herein relate to digital assistants and, in particular, to techniques for dynamic auditory phrase completion.

Description of the Related Art

In many settings, a speaker may repeat phrases to multiple listeners. For example, a salesperson may introduce herself, explain features of a product, and provide pricing information to multitudes of potential customers throughout a day or work week. Oftentimes, repeating the same speech multiple times to different audiences strains the speaker physically and mentally. For example, a speaker at a convention may repeat certain dialogue hundreds of times a day, straining the vocal cords of the speaker. In addition, the speaker must focus to ensure that she provides accurate information each time she repeats information to a listener, which occupies the mental load of the speaker and causes the speaker to fatigue.

In prior art audio delivery systems, a user may record her speech and replay the speech for various listeners to avoid repetition and strain on the user. One drawback of these systems is that such pre-recorded audio cannot adapt to individual conversations and settings. Such audio delivery systems are thus deemed as impersonal or non-responsive to the interests of the listener and are ineffective in sustaining the attention of the listener.

In light of the above, more effective techniques for delivering information to a listener through an audio delivery system.

SUMMARY

Embodiments of the present disclosure set forth a computer-implemented method comprising detecting an initial phrase portion included in a first auditory signal generated by a user, identifying, based on the initial phrase portion, a supplemental phrase portion that complements the initial phrase portion to form a complete phrase, and providing a command signal that drives an output device to generate an audio output corresponding to the supplemental phrase portion.

Further embodiments provide, among other things, non-transitory computer-readable storage media storing instructions for implementing the method set forth above, as well as a system configured to implement the method set forth above.

At least one technological advantage of the disclosed approach relative to the prior art is that by processing the dialogue of a user and completing phrases on behalf of a user in real time, the phrase processing system can complete individual phrases for a user, thus relieving physical and mental strains placed on a user when speaking repeated phrases. Further, by storing personalized phrases that a user repeats, the phrase a processing system can offer a broader range of phrases to assist the user, while also enabling the system to easily adapt to individual conversations. Further, the phrase processing system enables the user to tailor dialogue that is presented to different listeners, thus providing more natural-sounding audio delivery to listeners. These technical advantages provide one or more technological advancements over prior art approaches.

DETAILED DESCRIPTION

Overview

Embodiments disclosed herein include a phrase processing system that includes a phrase processing application that continually analyzes portions of speech made by a user and provides an audio output that completes phrases on behalf of the user. A processing unit included in the phrase processing system operates to receive an input speech signal corresponding to a portion of a phrase spoken by a user. The processing unit identifies an initial phrase portion and analyzes the actions performed by a user or a condition of a user in order to determine whether the user triggered the phrase processing application to complete the phrase. The processing unit searches internal or external data sources and uses information from the data sources to generate a supplemental phrase portion that complements the initial phrase portion. The processing unit drives an output device to emit soundwaves corresponding to the supplemental phrase portion.

The phrase processing system may be implemented in various forms of audiovisual-based systems, such as wearable audio devices, earpieces, mobile devices, personal computers, personal speaker systems, and so forth. The phrase processing system may perform its processing functions using a dedicated processing device and/or a separate computing device, such as a mobile computing device of a user or a cloud computing system. The phrase processing system may detect speech from a user using any number of auditory sensors, which may be attached to or integrated with other system components or disposed separately. The phrase processing system also acquires physiological data associated with the user using any type of sensor.

System Overview

FIG.1illustrates a phrase processing system according to one or more embodiments. As shown, phrase processing system100includes computing device110, network150, external data store152, sensor(s)172, input device(s)174, and output device(s)176. Computing device110includes memory120, processing unit140, network interface142, and input/output devices interface144. Memory120includes user interface122, voice agent124, database126, and phrase processing application130. Phrase processing application130includes voice recognition module132, trigger processing module134, phrase building module136, and voice agent controller138.

Computing device110includes processing unit140and memory120. In various embodiments, computing device110may be a device that includes one or more processing units140, such as a system-on-a-chip (SoC). In some embodiments, computing device110may be a wearable device, such as hearing aids, headphones, portable speakers, and/or other devices that include processing unit140. In other embodiments, computing device110may be a computing device, such as a tablet computer, desktop computer, mobile phone, media player, and so forth. In some embodiments, computing device110may be a head unit included in a vehicle system or at-home entertainment system. Generally, computing device110can be configured to coordinate the overall operation of phrase processing system100. The embodiments disclosed herein contemplate any technically-feasible system100configured to implement the functionality of phrase processing system100via computing device110.

For example, a wearable device could include at least one microphone as input device174, at least one speaker as output device176, and a microprocessor-based digital signal processor (DSP) as processing unit140that produces auditory signals that drive the at least one speaker to emit soundwaves. In some embodiments, phrase processing system100may be included in a digital voice assistant that includes one or more microphones, one or more loudspeakers, and one or more processing units. In some embodiments, various components of phrase processing system100may be contained within, or implemented by, different kinds of wearable devices and/or non-wearable devices. For example, one or more of computing device110, sensor(s)172, input device(s)174, and/or output device(s)176could be disposed within a hat, scarf, shirt collar, jacket, hood, etc. Similarly, processing unit140could provide user interface122via input device(s)174and/or output device(s)176that are included in a separate mobile or wearable device, such as a smartphone, tablet, wristwatch, arm band, etc. The separate mobile or wearable device could include an associated microprocessor and/or a digital signal processor that could also be used to provide additional processing power to augment the capabilities of the computing device110.

Processing unit140may include a central processing unit (CPU), a digital signal processing unit (DSP), a microprocessor, an application-specific integrated circuit (ASIC), a neural processing unit (NPU), a graphics processing unit (GPU), a field-programmable gate array (FPGA), and so forth. Processing unit140generally comprises a programmable processor that executes program instructions to manipulate input data. In some embodiments, processing unit140may include any number of processing cores, memories, and other modules for facilitating program execution. For example, processing unit140could receive an input (e.g., phrase portion162) from user160via input device(s)174and/or sensor data via sensor(s)172and drive output device(s)176to provide phrase portion162(2), where phrase portion162(2) complements phrase portion162(1) to form a complete phrase.

In some embodiments, processing unit140can be configured to execute phrase processing application130in order to analyze input auditory signals from input device(s)174and/or acquired sensor data from sensor(s)172and generate at least an auditory output. In such instances, phrase processing application130may generate an output auditory signal that acts as a supplement to the input auditory signal. For example, user160could speak a first portion (e.g., phrase portion162(1)) of a phrase that computing device110receives via input device174. Processing unit140may execute phrase processing application130and voice agent124to provide phrase portion162(2) via output device(s)176.

In some embodiments, processing unit140could execute phrase processing application130to initially determine whether user160triggered phrase processing application130to provide phrase portion162(2). In such instances, processing unit140may execute phrase processing application130to provide phrase portion162(2) upon identifying a triggering event associated with user160.

Memory120includes a memory module, or collection of memory modules. Memory120may include a variety of computer-readable media selected for their size, relative performance, or other capabilities: volatile and/or non-volatile media, removable and/or non-removable media, etc. Memory120may include cache, random access memory (RAM), storage, etc. Memory120may include one or more discrete memory modules, such as dynamic RAM (DRAM) dual inline memory modules (DIMMs). Of course, various memory chips, bandwidths, and form factors may alternately be selected.

Non-volatile memory included in memory120generally stores application programs including phrase processing application130, and data (e.g., data stored in database126) for processing by processing unit140. In various embodiments, memory120may include non-volatile memory, such as optical drives, magnetic drives, flash drives, or other storage. In some embodiments, separate data stores, such as external data store152included in network150(“cloud storage”) may supplement memory120. Phrase processing application130within memory120can be executed by processing unit140to implement the overall functionality of computing device110and, thus, to coordinate the operation of transparent sound management system100as a whole.

In various embodiments, memory120may include one or more modules for performing various functions or techniques described herein. In some embodiments, one or more of the modules and/or applications included in memory120may be implemented locally on computing device110, and/or may be implemented via a cloud-based architecture. For example, any of the modules and/or applications included in memory120could be executed on a remote device (e.g., smartphone, a server system, a cloud computing platform, etc.) that communicates with computing device110via network interface142or I/O devices interface144.

In various embodiments, phrase processing application (PPA)130may implement one or more modules132-138to process phrase portion162(1) via input device(s)174and/or sensor data via sensor(s)172in order to determine whether user160intends for phrase processing application130to complete a given phrase and complete the given phrase on behalf of user160. In such instances, phrase processing application130causes voice agent124to drive output device(s)176to emit soundwaves corresponding to phrase portion162(2). For example, user160could speak a phrase portion162(1) that corresponds to an initial section of a given phrase, and phrase processing application130could respond by generating phrase portion162(2) that corresponds to the remaining, supplemental portion of the given phrase.

Voice recognition module132performs various natural language processing (NLP) techniques, sentiment analysis, and/or speech analysis in order to identify phrases spoken by user160. In various embodiments, voice recognition module132determines a semantic meaning of speech made by user160in order to determine whether the provided speech is a portion of a phrase. In some embodiments, voice recognition module132performs sentiment analysis to determine an intended meaning of a phrase and/or determine whether the speaker has made an aberration in a repeated phrase.

In some embodiments, voice recognition module132could employ various statistical methods, machine-learning (ML) methods, state machines, and/or various other data structures in order to identify aberrations and/or deviations of phrases spoken by user160. Additionally or alternatively, user160may train the ML models by providing feedback to certain identified words or speech portions. In some embodiments, a combination of training data (e.g., previous phrases, physiological metrics, etc.) may train the ML models.

Trigger processing module134determines whether a triggering event occurred that indicates that phrase processing application130is to provide phrase portion162(2). In some embodiments, trigger processing module134may identify a specific action taken by user160. For example, user160could speak a specific trigger word or trigger phrase, or could take a pause from speaking. In another example, user160could press a button or make a specific gesture. In such instances, trigger processing module134may analyze data acquired via input device(s)174(e.g., auditory signals acquired via microphones) or sensor(s)172(e.g., image data) in order to identify a triggering event.

In some embodiments, trigger processing module134may identify a condition corresponding to a triggering event. For example, trigger processing module134may receive an alert from voice recognition module132that identifies an aberration in the speech of user160. Trigger processing module134could respond to the identified aberration as a triggering condition to help user160in completing a given phrase and/or subsequent phrases made by user160.

Phrase building module136generates a phrase portion162(2) that complements an input phrase portion162(1). In various embodiments, phrase portions162(1),162(2) may combine to form a complete phrase. In some embodiments, phrase building module136responds to a triggering event identified by trigger processing module134by analyzing phrase portion162(1) in order to identify phrase portion162(2).

In some embodiments, phrase building module136may refer to one or more external data sources that maintain one or more external data stores152. For example, user160could initially set a preferred knowledge base or search engine for phrase processing application130to query. Phrase building module136could use one or more words included in phrase portion162(1) as search terms to identify an applicable statement or data source. For example, phrase building module136could send a query to an online encyclopedia knowledge base that includes external data store152in order to retrieve a definition for a certain term. Phrase building module136could then generate phrase portion162(2) to include a text string and/or audio clip based on the information provided by external data store152.

In some embodiments, phrase building module136may scan one or more data sources included in local database126. For example, computing device110could maintain a phrase completion table in memory120that maintains a personalized list of complete phrases (stored as either complete phrases, or mappings between phrase portions) that are associated with user160. In some embodiments, phrase building module136may initially search a phrase completion table for personalized information about a given word or phrase before searching external knowledge bases at external data stores152.

Voice agent controller138generates an output that causes voice agent124to provide phrase portion162(2) as an audio output via output device(s)176. In various embodiments, voice agent controller138may provide an audio clip for voice agent124to reproduce. For example, phrase processing application130could store a phrase spoken by user160as an audio clip in database126. When phrase building generates a phrase portion162(2), voice agent controller138could retrieve the applicable audio clip364from database126and provide at least a portion of audio clip364to voice agent124to produce via output device176. In various embodiments, voice agent controller138may cause voice agent124to synthesize phrase portion162(2). For example, voice agent controller138could generate commands for voice agent124perform text-to-speech techniques to provide audio corresponding to a text string. In some embodiments, voice agent controller138may synthesize phrase portion162(2) using a synthesized voice that sounds similar to the voice of user160.

User interface122enables user160to provide input(s) about specific data, such as input phrases, preferred external data store(s)152, audio clips to store, voices that voice agent124is to use, and so forth. In some embodiments, user interface122may take any feasible form for providing the functions described herein, such as one or more buttons, toggles, sliders, dials, knobs, etc., or as a graphical user interface (GUI).

In various embodiments, user interface122may be provided through any component of phrase processing system100. In one embodiment, user interface122may be provided by a separate computing device that is communicatively coupled with computing device110, such as through an application running on a user's mobile or wearable computing device. In another example, user interface122may receive verbal commands for user selections. In this case, computing device110may perform speech recognition on the received verbal commands and/or compare the verbal commands against commands stored in memory120. After verifying the received verbal commands, computing device110could then execute the commanded function for phrase processing system100(e.g., storing a specific phrase and/or audio clip).

Voice agent124synthesizes one or more phrases that are to be generated as an auditory signal. For example, voice agent124could synthesize a phrase that is included in a command signal from voice agent controller138signal. Voice agent124could then generate an output signal to drive one or more loudspeakers included in output device176to emit soundwaves corresponding to the synthesized phrase.

Database (DB)126may store values and other data retrieved by processing unit140to coordinate the operation of phrase processing system100. In various embodiments, in operation, processing unit140may be configured to store values in database126and/or retrieve values stored in database126. For example, database126could store sensor data, audio content (e.g., audio clips, previous speech portions, etc.), a phrase completion database, and/or one or more data stores that act as a source of truth. For example, database126may download a knowledge base that corresponds to an external data store (e.g., a downloadable portion of map data, a portion of Wikipedia®, etc.). In such instances, one or more modules may refer to internal information stored in database126in lieu of communication with external data store152.

In some embodiments, computing device110may communicate with other devices, such as sensor(s)172, input device(s)174, and/or output device(s)176, using input/output (I/O) devices interface144. In such instances, I/O devices interface144may include any number of different I/O adapters or interfaces used to provide the functions described herein. For example, I/O devices interface144could include wired and/or wireless connections, and may use various formats or protocols. In another example, computing device110, through I/O devices interface144, could receive auditory signals from input device(s)174, may detect physiological data, visual data, and so forth using sensor(s)172, and may provide output signals to output device(s)176to produce outputs in various types (e.g., visual indication, soundwaves, haptic sensations, etc.).

In some embodiments, computing device110may communicate with other devices, such as external data store152, using network interface142and network150. In some embodiments, other types of networked computing devices (not shown) may connect to computing device110via network interface142. Examples of networked computing devices include a server, a desktop computer, a mobile computing device, such as a smartphone or tablet computer, and/or a worn device, such as a wristwatch or headphones or a head-mounted display device. In some embodiments, the networked computing devices may be used as sensor(s)172, input device(s)174, and/or output device(s)176.

Network150includes a plurality of network communications systems, such as routers and switches, configured to facilitate data communication between computing device110and external data store152. Persons skilled in the art will recognize that many technically-feasible techniques exist for building network150, including technologies practiced in deploying an Internet communications network. For example, network150may include a wide-area network (WAN), a local-area network (LAN), and/or a wireless (Wi-Fi) network, among others.

External data store(s)152include various libraries that provide various types of information. For example, external data stores152may include backends for search engines, dictionaries, online encyclopedias, mapping data, fact-checking websites, news websites, and so forth. In various embodiments, phrase processing application130may search multiple external data stores152and use information provided by one or more external data store(s)152to identify a complete phrase and/or identify a phrase portion corresponding to phrase portion162(1).

Sensor(s)172include one or more devices that collect data associated with objects in an environment. In various embodiments, sensor(s)172may include groups of sensors that acquire different sensor data. For example, sensor(s)172could include a reference sensor, such as a microphone and/or a visual sensor (e.g., camera, thermal imager, linear position sensor, etc.), which could acquire auditory data, visual data, physiological data, and so forth.

In various embodiments, sensor(s)172and/or input device(s)174may include audio sensors, such as a microphone and/or a microphone array that acquires sound data. In various embodiments, the microphone may be directional (e.g., user-facing microphone, beamforming microphone array, etc.) and acquire auditory data from a specific person, such as user160. Such sound data may be processed by phrase processing application130using various audio processing techniques. The audio sensors may be a plurality of microphones or other transducers or sensors capable of converting sound waves into an electrical signal. The audio sensors may include an array of sensors that includes sensors of a single type, or a variety of different sensors. Sensor(s)172may be worn by a user, disposed separately at a fixed location, or movable. Sensor(s)172may be disposed in any feasible manner in the environment. In some embodiments, sensor(s)172may be oriented toward user160relative to output device(s)176, which may be generally disposed outward, while sensor(s)172are also user-oriented

Sensor(s)172may include one or more devices that perform measurements and/or acquire data related to certain subjects in an environment. In various embodiments, sensor(s)172may generate sensor data that is related to user160. For example, sensor(s)172could collect biometric data related to user160(e.g., visible perspiration, muscle movement, breathing rate, pupil size, eye saccades, temporary change in skin color, etc.). and/or user160when speaking (e.g., heart rate, brain activity, skin conductance, blood oxygenation, galvanic skin response, blood-pressure level, average blood glucose concentration, etc.). Further, sensor(s)172could include a user-facing camera that records the face of user160as image data. Similarly, sensor(s)172could include a facial electromyography (fEMG) sensor that measures specific muscle contractions and associated activities (e.g., a raised eyebrow, clenched jaw, etc.), of user160. Phrase processing application130could then analyze the image data in order to determine the facial expression of user160.

In another example, sensor(s)172could include sensors that acquire biological and/or physiological signals of user160when speaking (e.g., perspiration, heart rate, heart-rate variability (HRV), blood flow, blood-oxygen levels, breathing rate, galvanic skin response (GSR), sounds created by a user, behaviors of a user, etc.). Additionally, sensor(s)172could include a pupil sensor (e.g., a camera focused on the eyes of user160) that acquires image data about at least one pupil of user160. Phrase processing application130could then perform various pupillometry techniques to detect eye parameters (e.g., fluctuations in the pupil diameter, eye gaze direction, eye lid position, eye saccades, etc.) as physiological data.

Input device(s)174are devices capable of receiving one or more inputs. In various embodiments, input device(s)174may include one or more audio input devices, such as a microphone, a set of microphones, and/or a microphone array. Additionally or alternatively, input device(s)174may include other devices capable of receiving input, such as a keyboard, a mouse, a touch-sensitive screen, and/or other input devices for providing input data to computing device110. For example, input from user160may include gestures, such as various movements or orientations of the hands, arms, eyes, or other parts of the body that are received via a camera. In various embodiments, user160may trigger phrase processing application130to complete a phrase in lieu of phrase processing application130automatically attempting to provide phrase portion162(2).

Output device(s)176include devices capable of providing output, such as a display screen, loudspeakers, haptic output devices, and the like. For example, output device176could be headphones, ear buds, a speaker system (e.g., one or more loudspeakers, amplifier, etc.), or any other device that generates an acoustic field. In another example, output device176could include haptic output devices, such as ultrasound transducers, air vortex generators, air bladders, and/or any type of device configured to generate haptic output. In various embodiments, various input device(s)174and/or output device(s)176can be incorporated into computing device110, or may be external to computing device110.

In various embodiments, output device(s)176may be implemented using any number of different conventional form factors, such as discrete loudspeaker devices, around-the-ear (circumaural), on-ear (supraaural), or in-ear headphones, hearing aids, wired or wireless headsets and/or personal speakers, body-worn (head, shoulder, arm, etc.) speaker devices, body-worn close-range directional speakers or speaker arrays, body-worn ultrasonic speaker arrays, and so forth. In some embodiments, output device(s)176include other forms of outputs, such as display devices that provide visual outputs. In some embodiments, output device(s)176may be worn by user160, or disposed separately at a fixed location, or movable. As discussed above, output device(s)176may be disposed inward of the sensor(s)172and oriented inward toward a particular region or user160.

FIG.2illustrates an example phrase completion table included in the phrase processing system ofFIG.1, according to one or more embodiments. As shown, phrase completion table200includes column210for stored initial phrase portions and column250for supplemental phrase portions. Phrase completion table200includes a set of stored initial phrase portion entries212-218and supplemental phrase portion entries252-258.

For a given entry (e.g., stored initial phrase portion212and supplemental phrase portion252), phrase processing application130may store a phrase spoken by user160. In some embodiments, phrase processing application130may store a phrase as a single entry. In other embodiments, phrase processing application130may store a mapping between a stored initial phrase portion212and a supplemental phrase portion252.

For the first entry (“entry1”), phrase processing application130may record user160speaking one or more phrase portions (which may include multiple sentences). In such instances, user160may perform an action that triggers phrase processing application130to store the phrase in phrase completion table200. When user160subsequently speaks stored initial phrase portion212, phrase processing application130may respond by retrieving a portion of the recorded speech, previously spoken by user160, that corresponds to supplemental phrase portion252.

For the second entry (“entry2”), phrase processing application130may store initial phrase portion214that includes a trigger phrase (“generally defined as”). In some embodiments, phrase processing application130may refer to entry2when user160speaks a different trigger phrase (e.g., “can be defined as”). In such instances, phrase building module136may retrieve supplemental phrase portion256as corresponding to a stored initial phrase portion214that uses a different trigger phrase.

For the third entry (“entry3”), phrase completion table200may store a complete phrase instead of as two distinct phrase portions216,256. In such instances, phrase building module136may generate supplemental phrase portion256by removing a portion of the complete phrase (corresponding to stored initial phrase portion216) from the complete phrase.

For the fourth entry (“entry4”), phrase completion table200may store initial phrase portion218that includes a trigger phrase for a specific data store152. In such instances, phrase processing application130may refer to the specific external data store152for supplemental phrase portion258. For example, upon identifying the trigger term (“directions”), phrase building module136could search a map repository. Phrase building module136may then update supplemental phrase portion258with information retrieved from the online library. In some embodiments, phrase building module136may update supplemental phrase portion258based on the current position of user160.

Dynamic Auditory Phrase Completion

FIG.3illustrates a technique of the phrase processing system100ofFIG.1phrase processing and dynamic auditory phrase completion to a portion of speech made by a user, according to one or more embodiments. As shown, phrase processing system300includes input device174, sensor(s)172, database126, external data store(s)152, voice agent124, output device176, and phrase processing application130. Phrase processing application130includes voice recognition module132, trigger processing module134, phrase building module136, and voice agent controller138. Database126includes phrase completion table200, local data store copy362, and audio clip364

In operation, input device174receives a speech portion312made by user160and transmits speech portion312as input speech signal322to voice recognition module132. Voice recognition module132parses and analyzes input speech signal322to identify any phrases included in input speech signal322. Upon identifying a phrase, voice recognition module132generates initial phrase portion332and transmits initial phrase portion332to trigger processing module134and/or phrase building module136.

In various embodiments, trigger processing module134determines whether user160performed an action indicating that phrase processing system300is to complete a phrase. In some embodiments, trigger processing module134may process speech input speech signal322and/or other acquired audio via input device174. For example, input speech signal322could include a personalized key phrase or keyword that was previously stored in phrase completion table200(e.g., “my assistant will tell you about”, etc.). In another example, input speech signal322could include a general trigger phrase (e.g., “defined”, “it is commonly known”, etc.). Trigger processing module134could respond to identifying the trigger phrase by determining that user160is triggering phrase processing system300to complete a phrase. In such instances, phrase processing application130could dynamically determine words and/or phrases said before or after the trigger phrase and perform searches for the identified keywords For example, when input speech signal322include “cognitive dissonance is defined as,” phrase processing system300could respond to identifying “is defined as” by selecting “cognitive dissonance as a key phrase that is included in initial phrase portion332. Trigger processing module134could then generate a command for phrase building module136to complete the phrase that includes initial phrase portion332.

In some embodiments, trigger processing module134may process the acquired audio and identify a pause and/or aberration in the speech of user160. For example, trigger processing module134could identify a pause of over three seconds in a phrase that voice recognition module132identifies as a partial phrase. Trigger processing module134could respond by identifying the pause as a trigger. In another example, voice recognition module132could identify an aberration in phrases spoken by user160, where the aberration could indicate fatigue by the user (e.g., softer speech indicating hoarseness, slurred speech, stilted speech indicating metal fatigue, etc.). Trigger processing module134could respond to the identified aberration as a triggering condition to help user160in completing a given phrase and/or subsequent phrases made by user160.

In some embodiments, trigger processing module134may receive a trigger signal directly from input device174. For example, input device174could be an actuator or other manually-triggered mechanism, such as a button, key fob, finger ring, etc. that transmits a trigger signal in response to a manual input from user160. In such instances, trigger processing module134could identify a triggering event upon receiving the trigger signal from input device174.

Additionally or alternatively, trigger processing module134may receive other sensor data from sensor(s)172. In some embodiments, sensor172could include a user-face camera and/or facial electromyography (fEMG) sensor that acquires sensor data associated with gestures and/or facial expressions of user160. In such instances, trigger processing module134could compare biometric values associated with the sensor data to one or more threshold values to identify a triggering event. For example, an fEMG sensor included in sensor172could provide biometric data indicating that user160clenched her jaw. Trigger processing module134identifies a triggering event upon determining that the clenched jaw exceeded a threshold value. In another example, trigger processing module134could identify a triggering event upon detecting, via image data or other biometric data, that user160raised at least one eyebrow. In some embodiments, trigger processing module134could identify a triggering event as a combination of multiple actions, such as a pause in the speech of user160, along with user160blinking rapidly.

Phrase building module136responds to trigger processing module134identifying a triggering event by generating supplemental phrase portion342. In various embodiments, initial phrase portion332and supplemental phrase portion342combine to form a complete phrase. In some embodiments, phrase building module136responds to trigger processing module134identifying a triggering event by analyzing initial phrase portion332and identifying supplemental phrase portion342.

In some embodiments, phrase building module136may refer to one or more external data sources that maintain one or more external data stores152. For example, user160could initially set a preferred knowledge base or search engine for phrase processing application130to query. Upon receiving initial phrase portion332, phrase building module136could use one or more words included in initial phrase portion332as search terms to identify an applicable statement or data set (e.g., one or more text strings in a website) that is applicable to initial phrase portion332. For example, phrase building module136could send a query to a dictionary knowledge base that includes external data store152in order to retrieve a definition for a certain term. Phrase building module136could then generate supplemental phrase portion342to include a text string and/or audio clip for the definition. In some embodiments, phrase building module136may store initial phrase portion332and the generated supplemental phrase portion342as an entry in phrase completion table200.

In some embodiments, phrase building module136may scan one or more data sources included in local database126. For example, phrase processing application130could download a portion of a knowledge base and search the downloaded local copy of the knowledge base (e.g., local data store copy362) in order to speed searches and enable offline operability. In another example, phrase processing application130could maintain phrase completion table200that maintains a personalized list of complete phrases (stored as either complete phrases or mappings between initial phrase portion332and supplemental phrase portion342) that are associated with user160. In some embodiments, phrase building module136may initially search phrase completion table200before searching external knowledge bases at external data stores152. Phrase building module136may use initial phrase portion332to identify an applicable entry in phrase completion table200and retrieve the supplemental phrase portion342from the applicable entry.

In some embodiments, phrase building module136may generate supplemental phrase portion342based on initial phrase portion332. For example, phrase completion table200may store a complete phrase as an entry. Phrase building module136may generate supplemental phrase portion342by removing initial phrase portion from the complete phrase. For example, phrase completion table200may store an entry that includes the complete phrase, “I am an engineer in the West Coast team for the Phoenix Project.” Phrase building module136could receive initial phrase portion332corresponding to speech portion312and remove initial phrase portion332from the complete phrase to generate supplemental phrase portion342(“in the West Coast team for the Phoenix Project.”).

Phrase building module136transmits supplemental phrase portion342to voice agent controller138. Voice agent controller138generates an output that causes voice agent124to provide supplemental phrase portion352as an audio output via output device176. In various embodiments, voice agent controller138may provide an audio clip364for voice agent124to reproduce. For example, phrase processing application130could store a phrase spoken by user160as audio clip364in database126and associate audio clip364with a given phrase. When phrase building generates a supplemental phrase portion342that is associated with the given phrase, voice agent controller138may retrieve the applicable audio clip364from database126and provide at least a portion of audio clip364to voice agent124to produce via output device176. In some embodiments, one or more audio clips364may be stored in network storage. In such instances, voice agent controller138may connect to external data store152to obtain the remotely-stored audio clip364.

In various embodiments, voice agent controller138may cause voice agent to synthesize supplemental phrase portion342. For example, voice agent controller138could generate commands for voice agent124perform text-to-speech techniques to provide an audio signal corresponding to a text string included in supplemental phrase portion342. In some embodiments, voice agent controller138may synthesize supplemental phrase portion342using a synthesized voice similar to user160. For example voice agent124could sample speech made by user160and generate a personalized voice corresponding to user160. In such instances, voice agent124could respond to commands from voice agent controller138by synthesizing words in a manner (e.g., tone, speed, volume, etc.) that simulates user160. In other embodiments, voice agent124may synthesize supplemental phrase portion342using a generic text-to-speech voice. In such instances, user160may select a specific voice from a set of generic voices. Additionally or alternatively, voice agent controller138may combine an audio clip364with a synthesized speech portion in order to provide a complete supplemental phrase portion342.

Voice agent124provides output device176with an output signal that is based on supplemental phrase portion342. For example, voice agent124could generate an audio output signal that drives one or more loudspeakers included in output device176to emit soundwaves corresponding to speech portion352. In some embodiments, voice agent124may include one or more other output signals. For example, voice agent124may include one or more video output signals that drives a display device to display an image or video corresponding to speech portion352(e.g., illustrating a soundwave corresponding to the audio output signal, illustrating an avatar mouthing the words in speech portion352, etc.).

In some embodiments, output device176may be directional and provide an output to a specific target. For example, output device176may include one or more directional speakers that generate a steerable beam. In such instances, output device176may form a beam towards a target listener such that the listener hears speech portion352in addition to the speech portion312that was spoken by user160.

In various embodiments, phrase processing application130may continually process speech made by user160. For example, phrase processing application130could identify initial phrase portion332from input speech signal322without receiving a trigger from user160to provide supplemental phrase portion. Phrase processing application130could then listen for a subsequent phrase made by user160and, upon identifying a triggering event, provide a supplemental phrase portion for that subsequent phrase.

FIG.4is a flow diagram of methods steps to dynamically complete a phrase spoken by a user, according to one or more embodiments. Although the method steps are described with respect to the systems ofFIGS.1-3, persons skilled in the art will understand that any system configured to perform the method steps, in any order, falls within the scope of the various embodiments. In some embodiments, phrase processing system100may continually execute method400on captured audio in real-time.

Method400begins at step402, where phrase processing application130processes speech made by user160. In various embodiments, phrase processing application receives an input speech signal322from input device174. Input speech signal322corresponds to a speech portion312that was spoken by user160. In some embodiments, voice recognition module132included in phrase processing application130may perform various natural language processing techniques to identify words that are included in input speech signal322.

At step404, phrase processing application130determines whether a triggering event occurred, where the triggering event indicates that phrase processing application130is to complete a phrase. In various embodiments, trigger processing module134included in phrase processing application130determines whether user160has performed an action or is in a condition to trigger the phrase processing application130to complete a phrase corresponding to initial phrase portion332. In some embodiments, trigger processing module134may analyze input speech signal322and/or initial phrase portion332in order to identify a component (e.g., a trigger phrase, pause, aberration, etc.) as a triggering event. Additionally or alternatively, trigger processing module may analyze sensor data, such as image and/or biometric data, to determine whether user160has performed a specific action (e.g., muscle contraction, eye gaze direction, hand movement, etc.) or is in a specific condition (e.g., received alert, biometric values beyond an acceptable range, etc.) that indicates that phrase processing application130is to complete a phrase on behalf of user160.

Upon determining that a triggering event has occurred, phrase processing application130proceeds to step406; otherwise, phrase processing application130determines that a triggering event has not occurred and returns to step402to process more speech that was spoken by user160.

As step406, phrase processing application130identifies an initial phrase portion from the processed speech. In various embodiments, voice recognition module132parses and analyzes input speech signal322to identify phrases included in input speech signal322. Upon identifying a phrase, voice recognition module132may generate initial phrase portion332and transmits initial phrase portion332to trigger processing module134and/or phrase building module136. In some embodiments, phrase processing application130may perform step406before step404. In such instances, voice recognition module132may identify phrase portion332independent of trigger processing module134determining whether a triggering event occurred.

At step408, phrase processing application130determines a supplemental phrase portion342that complements initial phrase portion332. In various embodiments, phrase building module136included in phrase processing application130may refer to one or more data sources (e.g., internal database126and/or external data store152) in order to generate a supplemental phrase portion342that complements initial phrase portion332.

In some embodiments, database126may store a phrase completion table200that maintains a personalized list of complete phrases (stored as either complete phrases or mappings between initial phrase portion332and supplemental phrase portion342) that are associated with user160. In such instances, phrase building module136may use initial phrase portion332to identify an applicable entry in phrase completion table200and retrieve the supplemental phrase portion342from the applicable entry.

Additionally or alternatively, phrase building module136could use one or more words included in initial phrase portion332in a query to external data store152to identify an applicable data set (e.g., one or more text strings in a website) that is applicable to initial phrase portion332. Phrase building module136could then generate supplemental phrase portion342to include information derived from the applicable data set. In some embodiments, phrase building module136may store initial phrase portion332and the generated supplemental phrase portion342as an entry in phrase completion table200.

At step410, phrase processing application130determines whether computing device110is storing an audio clip364corresponding to supplemental phrase portion342. In various embodiments, voice agent controller138included in phrase processing application130may determine whether database126is storing an audio clip364that corresponds to supplemental phrase portion342. For example, phrase processing application130could store a phrase spoken by user160as audio clip364in database126and associate audio clip364with a given phrase. When phrase building generates a supplemental phrase portion342that is associated with the given phrase, voice agent controller138may retrieve the applicable audio clip364from database126. When voice agent controller138determines that database126includes a stored audio clip364for supplemental phrase portion342, phrase processing application130proceeds to step412; otherwise, voice agent controller138determines that database126has no stored audio clip corresponding to supplemental phrase portion342and proceeds to step414.

At step412, phrase processing application130initiates playback of the stored audio clip364. In some embodiments, voice agent controller138provides at least a portion of audio clip364to voice agent124to produce via output device176. For example, voice agent124could generate an audio output signal that includes audio clip364. The audio output signal drives one or more loudspeakers included in output device176to emit soundwaves to reproduce audio clip364.

Returning to step410, upon voice agent controller138determining that database126has no stored audio clip corresponding to supplemental phrase portion342, voice agent controller138proceeds to step414, where phrase processing application130causes supplemental phrase portion342to be synthesized as an audio signal. For example, voice agent controller138could generate commands for voice agent124perform text-to-speech techniques to provide an audio signal corresponding to a text string included in supplemental phrase portion342. In some embodiments, voice agent controller138may synthesize supplemental phrase portion342using a synthesized voice that sounds similar to the voice of user160. In other embodiments, voice agent124may synthesize supplemental phrase portion342using a generic text-to-speech voice.

At step416, phrase processing application130initiates playback of the synthesized audio signal. In various embodiments, voice agent124could provide the synthesized audio signal to output device176. One or more speakers included in output device176may then emit soundwaves based on the synthesized audio signal.

In sum, a phrase processing application receives an input speech signal corresponding to a portion of a phrase spoken by a user. A voice recognition module included in the phrase processing application identifies an initial phrase portion included in the input speech signal. A trigger processing module included in the phrase processing application analyzes the speech signal and other sensor data to determine whether the user has performed an action or is in a condition to trigger the phrase processing application to complete a phrase, corresponding to the initial phrase portion, on behalf of the user.

Upon determining that a triggering event has occurred, a phrase building module within the phrase building application generates a supplemental phrase portion that complements the initial phrase portion to form a complete phrase. In some embodiments, the phrase building module refers to an internal phrase completion table that includes user-specific entries of common phrases spoken by the user. In some embodiments, the phrase building module refers to external data stores to identify phrases applicable to the initial phrase portion. The phrase building application generates the supplemental phrase portion based on the information received from the applicable data source. A voice agent controller included in the phrase processing application causes a voice agent to provide the supplemental phrase portion and drive an output device to emit soundwaves corresponding to the supplemental phrase portion.

At least one technological advantage of the disclosed approach relative to the prior art is that by processing the dialogue of a user and completing phrases on behalf of a user in real time, the phrase processing system can complete individual phrases for a user, thus relieving physical and mental strains placed on a user when speaking repeated phrases. In particular, by providing personalized phrase portions that supplement phrases spoken by a user, the phrase processing system frees the user from using her voice constantly and relieves a mental load associated with speaking accurately. Further, by storing personalized phrases that a user repeats, the phrase processing system can offer a broader range of phrases to assist the user, while also enabling the system to easily adapt to individual conversations.

1. In various embodiments, a computer-implemented method comprising detecting an initial phrase portion included in a first auditory signal generated by a user, identifying, based on the initial phrase portion, a supplemental phrase portion that complements the initial phrase portion to form a complete phrase, and providing a command signal that drives an output device to generate an audio output corresponding to the supplemental phrase portion.

2. The computer-implemented method of clause 1, further comprising determining whether a triggering event occurred that is associated with the user generating the initial phrase portion, wherein the supplemental phrase portion is identified upon determining that the triggering event occurred.

3. The computer-implemented method of clause 1 or 2, where the initial phrase portion is detected prior to determining that the triggering event occurred.

4. The computer-implemented method of any of clauses 1-3, where the triggering event comprises at least one of a key phrase included in the first auditory signal, or a manual input provided via an input device.

5. The computer-implemented method of any of clauses 1-4, where determining whether the triggering event occurred comprises detecting, in the first auditory signal, at least one of a pause in speech that is detected in the first auditory signal, or an aberration in speech that is detected in the first auditory signal.

6. The computer-implemented method of any of clauses 1-5, further comprising prior to receiving the first auditory signal, recording a second auditory signal that includes the complete phrase, and storing (i) a first audio clip that corresponds to the initial phrase portion, and (ii) a second audio clip that corresponds to the supplemental phrase portion.

7. The computer-implemented method of any of clauses 1-6, further comprising receiving a user input associated with the second auditory signal, wherein receiving the user input triggers storage of the first audio clip and the second audio clip.

8. The computer-implemented method of any of clauses 1-7, further comprising retrieving the second audio clip that corresponds to the supplemental phrase portion, where the output device generates the audio output of the first audio clip.

9. The computer-implemented method of any of clauses 1-8, further comprising in response to detecting the initial phrase portion, sending a query to an external data store, where the query is based on the initial phrase portion, and receiving, from the external data store, a first data set, wherein the supplemental phrase portion is associated with at least a portion of the first data set.

10. The computer-implemented method of any of clauses 1-9, further comprising prior to receiving the first auditory signal, receiving an input that includes the complete phrase, parsing the complete phrase into the initial phrase portion and the supplemental phrase portion, and storing, in a phrase completion table, a mapping between the initial phrase portion and the supplemental phrase portion.

11. In various embodiment, a system that completes a complete phrase that is partially spoken by a user, the system comprising at least one microphone that acquires a first auditory signal of a user, and a computing device that detects an initial phrase portion included in the first auditory signal, identifies, based on the initial phrase portion, a supplemental phrase portion that complements the initial phrase portion to form the complete phrase, and provides a command signal that drives an output device to generate an audio output corresponding to the supplemental phrase portion.

12. The system of clause 11, where the computing device further determines whether a triggering event occurred that is associated with the user generating the initial phrase portion, wherein the supplemental phrase portion is identified upon determining that the triggering event occurred.

13. The system of clause 11 or 12, where determining whether the triggering event occurred comprises detecting at least one of a key phrase included in the first auditory signal, a pause in speech in the first auditory signal, an aberration in the first auditory signal, or a manual input provided via an input device.

14. The system of any of clauses 11-13, further comprising at least one or a visual sensor or a facial electromyography sensor that acquires biometric data associated with the user, where the triggering event comprises a biometric value above a threshold value, and the biometric value is included in biometric data that includes at least one of eye gaze direction, muscle contraction, or facial movement.

15. The system of any of clauses 11-14, further comprising a voice agent that synthesizes an audio output signal of a voice speaking the supplemental phrase portion, wherein the computing device provides the command signal to the voice agent to synthesize the audio output signal, and the voice agent drives the output device to generate an audio output of the audio output signal.

16. The system any of clauses 11-15, where the output device comprises at least one speaker that generates a steerable beam that provides the audio output to a target listener.

17. The system any of clauses 11-16, where the computing device includes a memory that stores a phrase completion table, and the phrase completion table includes at least a mapping between the initial phrase portion and the supplemental phrase portion.

18. In various embodiments, one or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of detecting an initial phrase portion included in a first auditory signal generated by a user, identifying, based on the initial phrase portion, a supplemental phrase portion that complements the initial phrase portion to form the complete phrase, and providing a command signal that drives an output device to generate an audio output corresponding to the supplemental phrase portion.

19. The one or more non-transitory computer-readable media of clause 18, further comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform the step of determining whether a triggering event occurred that is associated with the user generating the initial phrase portion, wherein the supplemental phrase portion is identified upon determining that the triggering event occurred.

20. The one or more non-transitory computer-readable media of clause 18 or 19, further comprising instructions that, when executed by the one or more processors, cause the one or more processors to perform the steps of prior to receiving the first auditory signal, receiving an input that includes the complete phrase, parsing the complete phrase into the initial phrase portion and the supplemental phrase portion, and storing, in a phrase completion table, a mapping between the initial phrase portion and the supplemental phrase portion.