Patent ID: 12243531

DETAILED DESCRIPTION

In the following description of examples, reference is made to the accompanying drawings which form a part hereof, and in which it is shown by way of illustration specific examples that can be practiced. It is to be understood that other examples can be used and structural changes can be made without departing from the scope of the disclosed examples.

Example Wearable System

FIG.1illustrates an example wearable head device100configured to be worn on the head of a user. Wearable head device100may be part of a broader wearable system that comprises one or more components, such as a head device (e.g., wearable head device100), a handheld controller (e.g., handheld controller200described below), and/or an auxiliary unit (e.g., auxiliary unit300described below). In some examples, wearable head device100can be used for virtual reality, augmented reality, or mixed reality systems or applications. Wearable head device100can comprise one or more displays, such as displays110A and110B (which may comprise left and right transmissive displays, and associated components for coupling light from the displays to the user's eyes, such as orthogonal pupil expansion (OPE) grating sets112A/112B and exit pupil expansion (EPE) grating sets114A/114B); left and right acoustic structures, such as speakers120A and120B (which may be mounted on temple arms122A and122B, and positioned adjacent to the user's left and right ears, respectively); one or more sensors such as infrared sensors, accelerometers, GPS units, inertial measurement units (IMUs, e.g. IMU126), acoustic sensors (e.g., microphones150); orthogonal coil electromagnetic receivers (e.g., receiver127shown mounted to the left temple arm122A); left and right cameras (e.g., depth (time-of-flight) cameras130A and130B) oriented away from the user; and left and right eye cameras oriented toward the user (e.g., for detecting the user's eye movements)(e.g., eye cameras128A and128B). However, wearable head device100can incorporate any suitable display technology, and any suitable number, type, or combination of sensors or other components without departing from the scope of the invention. In some examples, wearable head device100may incorporate one or more microphones150configured to detect audio signals generated by the user's voice; such microphones may be positioned adjacent to the user's mouth. In some examples, wearable head device100may incorporate networking features (e.g., Wi-Fi capability) to communicate with other devices and systems, including other wearable systems. Wearable head device100may further include components such as a battery, a processor, a memory, a storage unit, or various input devices (e.g., buttons, touchpads); or may be coupled to a handheld controller (e.g., handheld controller200) or an auxiliary unit (e.g., auxiliary unit300) that comprises one or more such components. In some examples, sensors may be configured to output a set of coordinates of the head-mounted unit relative to the user's environment, and may provide input to a processor performing a Simultaneous Localization and Mapping (SLAM) procedure and/or a visual odometry algorithm. In some examples, wearable head device100may be coupled to a handheld controller200, and/or an auxiliary unit300, as described further below.

FIG.2illustrates an example mobile handheld controller component200of an example wearable system. In some examples, handheld controller200may be in wired or wireless communication with wearable head device100and/or auxiliary unit300described below. In some examples, handheld controller200includes a handle portion220to be held by a user, and one or more buttons240disposed along a top surface210. In some examples, handheld controller200may be configured for use as an optical tracking target; for example, a sensor (e.g., a camera or other optical sensor) of wearable head device100can be configured to detect a position and/or orientation of handheld controller200—which may, by extension, indicate a position and/or orientation of the hand of a user holding handheld controller200. In some examples, handheld controller200may include a processor, a memory, a storage unit, a display, or one or more input devices, such as described above. In some examples, handheld controller200includes one or more sensors (e.g., any of the sensors or tracking components described above with respect to wearable head device100). In some examples, sensors can detect a position or orientation of handheld controller200relative to wearable head device100or to another component of a wearable system. In some examples, sensors may be positioned in handle portion220of handheld controller200, and/or may be mechanically coupled to the handheld controller. Handheld controller200can be configured to provide one or more output signals, corresponding, for example, to a pressed state of the buttons240; or a position, orientation, and/or motion of the handheld controller200(e.g., via an IMU). Such output signals may be used as input to a processor of wearable head device100, to auxiliary unit300, or to another component of a wearable system. In some examples, handheld controller200can include one or more microphones to detect sounds (e.g., a user's speech, environmental sounds), and in some cases provide a signal corresponding to the detected sound to a processor (e.g., a processor of wearable head device100).

FIG.3illustrates an example auxiliary unit300of an example wearable system. In some examples, auxiliary unit300may be in wired or wireless communication with wearable head device100and/or handheld controller200. The auxiliary unit300can include a battery to provide energy to operate one or more components of a wearable system, such as wearable head device100and/or handheld controller200(including displays, sensors, acoustic structures, processors, microphones, and/or other components of wearable head device100or handheld controller200). In some examples, auxiliary unit300may include a processor, a memory, a storage unit, a display, one or more input devices, and/or one or more sensors, such as described above. In some examples, auxiliary unit300includes a clip310for attaching the auxiliary unit to a user (e.g., a belt worn by the user). An advantage of using auxiliary unit300to house one or more components of a wearable system is that doing so may allow large or heavy components to be carried on a user's waist, chest, or back—which are relatively well suited to support large and heavy objects—rather than mounted to the user's head (e.g., if housed in wearable head device100) or carried by the user's hand (e.g., if housed in handheld controller200). This may be particularly advantageous for relatively heavy or bulky components, such as batteries.

FIG.4shows an example functional block diagram that may correspond to an example wearable system400, such as may include example wearable head device100, handheld controller200, and auxiliary unit300described above. In some examples, the wearable system400could be used for virtual reality, augmented reality, or mixed reality applications. As shown inFIG.4, wearable system400can include example handheld controller400B, referred to here as a “totem” (and which may correspond to handheld controller200described above); the handheld controller400B can include a totem-to-headgear six degree of freedom (6DOF) totem subsystem404A. Wearable system400can also include example headgear device400A (which may correspond to wearable head device100described above); the headgear device400A includes a totem-to-headgear 6DOF headgear subsystem404B. In the example, the 6DOF totem subsystem404A and the 6DOF headgear subsystem404B cooperate to determine six coordinates (e.g., offsets in three translation directions and rotation along three axes) of the handheld controller400B relative to the headgear device400A. The six degrees of freedom may be expressed relative to a coordinate system of the headgear device400A. The three translation offsets may be expressed as X, Y, and Z offsets in such a coordinate system, as a translation matrix, or as some other representation. The rotation degrees of freedom may be expressed as sequence of yaw, pitch and roll rotations; as vectors; as a rotation matrix; as a quaternion; or as some other representation. In some examples, one or more depth cameras444(and/or one or more non-depth cameras) included in the headgear device400A; and/or one or more optical targets (e.g., buttons240of handheld controller200as described above, or dedicated optical targets included in the handheld controller) can be used for 6DOF tracking. In some examples, the handheld controller400B can include a camera, as described above; and the headgear device400A can include an optical target for optical tracking in conjunction with the camera. In some examples, the headgear device400A and the handheld controller400B each include a set of three orthogonally oriented solenoids which are used to wirelessly send and receive three distinguishable signals. By measuring the relative magnitude of the three distinguishable signals received in each of the coils used for receiving, the 6DOF of the handheld controller400B relative to the headgear device400A may be determined. In some examples, 6DOF totem subsystem404A can include an Inertial Measurement Unit (IMU) that is useful to provide improved accuracy and/or more timely information on rapid movements of the handheld controller400B.

In some examples involving augmented reality or mixed reality applications, it may be desirable to transform coordinates from a local coordinate space (e.g., a coordinate space fixed relative to headgear device400A) to an inertial coordinate space, or to an environmental coordinate space. For instance, such transformations may be necessary for a display of headgear device400A to present a virtual object at an expected position and orientation relative to the real environment (e.g., a virtual person sitting in a real chair, facing forward, regardless of the position and orientation of headgear device400A), rather than at a fixed position and orientation on the display (e.g., at the same position in the display of headgear device400A). This can maintain an illusion that the virtual object exists in the real environment (and does not, for example, appear positioned unnaturally in the real environment as the headgear device400A shifts and rotates). In some examples, a compensatory transformation between coordinate spaces can be determined by processing imagery from the depth cameras444(e.g., using a Simultaneous Localization and Mapping (SLAM) and/or visual odometry procedure) in order to determine the transformation of the headgear device400A relative to an inertial or environmental coordinate system. In the example shown inFIG.4, the depth cameras444can be coupled to a SLAM/visual odometry block406and can provide imagery to block406. The SLAM/visual odometry block406implementation can include a processor configured to process this imagery and determine a position and orientation of the user's head, which can then be used to identify a transformation between a head coordinate space and a real coordinate space. Similarly, in some examples, an additional source of information on the user's head pose and location is obtained from an IMU409of headgear device400A. Information from the IMU409can be integrated with information from the SLAM/visual odometry block406to provide improved accuracy and/or more timely information on rapid adjustments of the user's head pose and position.

In some examples, the depth cameras444can supply 3D imagery to a hand gesture tracker411, which may be implemented in a processor of headgear device400A. The hand gesture tracker411can identify a user's hand gestures, for example by matching 3D imagery received from the depth cameras444to stored patterns representing hand gestures. Other suitable techniques of identifying a user's hand gestures will be apparent.

In some examples, one or more processors416may be configured to receive data from headgear subsystem404B, the IMU409, the SLAM/visual odometry block406, depth cameras444, microphones450; and/or the hand gesture tracker411. The processor416can also send and receive control signals from the 6DOF totem system404A. The processor416may be coupled to the 6DOF totem system404A wirelessly, such as in examples where the handheld controller400B is untethered. Processor416may further communicate with additional components, such as an audio-visual content memory418, a Graphical Processing Unit (GPU)420, and/or a Digital Signal Processor (DSP) audio spatializer422. The DSP audio spatializer422may be coupled to a Head Related Transfer Function (HRTF) memory425. The GPU420can include a left channel output coupled to the left source of imagewise modulated light424and a right channel output coupled to the right source of imagewise modulated light426. GPU420can output stereoscopic image data to the sources of imagewise modulated light424,426. The DSP audio spatializer422can output audio to a left speaker412and/or a right speaker414. The DSP audio spatializer422can receive input from processor419indicating a direction vector from a user to a virtual sound source (which may be moved by the user, e.g., via the handheld controller400B). Based on the direction vector, the DSP audio spatializer422can determine a corresponding HRTF (e.g., by accessing a HRTF, or by interpolating multiple HRTFs). The DSP audio spatializer422can then apply the determined HRTF to an audio signal, such as an audio signal corresponding to a virtual sound generated by a virtual object. This can enhance the believability and realism of the virtual sound, by incorporating the relative position and orientation of the user relative to the virtual sound in the mixed reality environment—that is, by presenting a virtual sound that matches a user's expectations of what that virtual sound would sound like if it were a real sound in a real environment.

In some examples, such as shown inFIG.4, one or more of processor416, GPU420, DSP audio spatializer422, HRTF memory425, and audio/visual content memory418may be included in an auxiliary unit400C (which may correspond to auxiliary unit300described above). The auxiliary unit400C may include a battery427to power its components and/or to supply power to headgear device400A and/or handheld controller400B. Including such components in an auxiliary unit, which can be mounted to a user's waist, can limit the size and weight of headgear device400A, which can in turn reduce fatigue of a user's head and neck.

WhileFIG.4presents elements corresponding to various components of an example wearable system400, various other suitable arrangements of these components will become apparent to those skilled in the art. For example, elements presented inFIG.4as being associated with auxiliary unit400C could instead be associated with headgear device400A or handheld controller400B. Furthermore, some wearable systems may forgo entirely a handheld controller400B or auxiliary unit400C. Such changes and modifications are to be understood as being included within the scope of the disclosed examples.

Speech Processing Engines

Speech recognition systems in general include a speech processing engine that can accept an input audio signal corresponding to human speech (a source signal); process and analyze the input audio signal; and produce, as a result of the analysis, an output corresponding to the human speech. In the case of automatic speech recognition (ASR) systems, for example, the output of a speech processing engine may be a text transcription of the human speech. In the case of natural language processing systems, the output may be one or more commands or instructions indicated by the human speech; or some representation (e.g., a logical expression or a data structure) of the semantic meaning of the human speech. Other types of speech processing systems (e.g., automatic translation systems), including those that do not necessarily “recognize” speech, are contemplated and are within the scope of the disclosure.

Speech recognition systems are found in a diverse array of products and applications: conventional telephone systems; automated voice messaging systems; voice assistants (including standalone and smartphone-based voice assistants); vehicles and aircraft; desktop and document processing software; data entry; home appliances; medical devices; language translation software; closed captioning systems; and others. An advantage of speech recognition systems is that they may allow users to provide input to a computer system using natural spoken language, such as presented to one or more microphones, instead of conventional computer input devices such as keyboards or touch panels; accordingly, speech recognition systems may be particularly useful in environments where conventional input devices (e.g., keyboards) may be unavailable or impractical. Further, by permitting users to provide intuitive voice-based input, speech processing engines can heighten feelings of immersion. As such, speech recognition can be a natural fit for wearable systems, and in particular, for virtual reality, augmented reality, and/or mixed reality applications of wearable systems, in which user immersion is a primary goal; and in which it may be desirable to limit the use of conventional computer input devices, whose presence may detract from feelings of immersion.

Typically, the output of any speech processing engine does not correspond to the source human speech with perfect certainty; because of the many variables that can affect the audio signals provided as input, even sophisticated speech processing engines do not consistently produce perfect text output for all speakers. For example, the reliability of speech processing engines may be highly dependent on the quality of the input audio signal. Where input audio signals are recorded in ideal conditions—for example, in acoustically controlled environments, with a single human speaker enunciating clearly and directly into a microphone from a close distance—the source speech can be more readily determined from the audio signal. In real-world applications, however, input audio signals may deviate from ideal conditions, such that determining the source human speech becomes more difficult. For example, input audio signals may include significant ambient noise, or speech from multiple speakers, in addition to the user; for instance, speech from other people, pets, or electronic devices (e.g., televisions) can be mixed in with the user's speech in the input signal. In addition, even the user's speech may include not only speech intended for the speech processing engine (input speech), but also speech directed at other listeners (such as other people, pets, or other devices). By isolating the input speech from the broader input audio signal, the fidelity of the input processed by the speech processing engine can be improved; and the accuracy of the speech processing engine's output can be improved accordingly.

Identifying and Segmenting Input Speech

The present disclosure is directed to systems and methods for improving the accuracy of a speech processing system by removing, from raw speech signals, portions of those signals that are not directed by the user to the speech processing system. As described herein, such non-input portions can be identified (e.g., classified) based on audio characteristics of the speech signals themselves (e.g., sudden changes in the speech's vocabulary, semantics, or grammar); and/or by using input from sensors associated with wearable devices (e.g., head-mounted devices such as described above with respect toFIG.1). Such non-input portions may be especially prominent in mobile applications of speech processing, in household usage of speech processing systems, or in applications of speech processing in uncontrolled environments, such as outdoor environments where other voices or ambient noise may be present. Wearable systems are frequently intended for use in such applications, and may therefore be especially susceptible to undirected speech. For example, where some wearable systems are intended for use in uncontrolled environments, a high potential can exist for environmental noise (or speech of other humans) to be recorded along with the target human speech. Sensors of wearable systems (such as described above with respect toFIGS.1-4) are well suited to solving this problem, as described herein. However, in some examples, as described herein, directivity can be determined based solely on a speech signal, even without the benefit of sensor input.

FIG.5illustrates an example system500, according to some embodiments, in which a speech processing engine550produces a text output552(such as described above) based on a raw speech signal510provided as input. In some examples, raw speech signal510can be can be provided as detected by one or more microphones, but in some examples can be provided from a data file (e.g., an audio waveform file), from an audio stream (e.g., provided via a network), or from any other suitable source. In system500, improved accuracy of text output552can be achieved by presenting, as input to speech processing engine550, a “directed” speech signal540that includes only those portions of raw input speech signal510that are determined to constitute input speech directed to speech processing engine550(as opposed to, for example, extraneous speech such as described above). Directed speech signal540can be determined at stage530from the raw input speech signal510and/or from sensor data520, which can correspond to data from sensors such as described above with respect to example wearable head device100inFIG.1.

At stage530, raw speech signal510can be divided into individual speech segments; then, for each segment, a probability can be determined that the segment corresponds to input speech that was intended as input for the speech processing engine550. In some cases, probabilistic modelling or machine learning techniques can indicate this probability for each segment of the raw speech signal510. Directed speech signal540can then be generated by filtering, from raw speech signal510, the segments of raw speech signal510that do not meet a threshold probability of corresponding to input speech (rather than to non-input speech). (As used herein, input speech can include input audio that is provided by a particular user and that is also directed by the user toward a speech recognition system.).

FIGS.6A-6Dillustrate examples of a raw speech signal, a segmented version of the speech signal, a probabilistic model of the raw speech signal (though in some embodiments machine learning techniques may be used), and a directed speech signal generated from the raw speech signal, respectively.FIG.6Ashows an example audio waveform600(which may correspond to raw speech signal510), expressed as an amplitude (e.g., of voltage) V(t) as a function of time, such as might be detected by one or more microphones and/or represented in a waveform audio file. In the example, the waveform600corresponds to a user speaking the example sequence, “What's the weather . . . not now, Charlie . . . tomorrow.” In the example, the speech sequence includes at least one portion (“What's the weather”) intended as a query to the speech processing engine (e.g., speech processing engine550); at least one portion (“not now, Charlie”) intended not as input to speech processing engine, but to another listener (presumably, Charlie); and at least one portion (“tomorrow”) that could reasonably belong, semantically, either to the speech recognition input portion (“What's the weather . . . tomorrow”) or to the non-input portion (“not now, Charlie . . . tomorrow”). In addition, raw speech signal510includes non-verbal noise in between spoken word portions. If raw speech signal510were applied directly as input to speech processing engine550, the system might produce unexpected results, as the presence of non-input speech (“not now, Charlie,” and possibly “tomorrow”) could interfere with the system's ability to meaningfully respond to the input speech (“What's the weather,” possibly with the qualifier “tomorrow”). Higher quality results can be achieved by, in advance of providing input to speech processing engine550, filtering raw speech signal600to generate a directed audio signal that includes speech directed at speech processing engine550(e.g., “What's the weather . . . tomorrow”) to the exclusion of non-input speech not directed at speech processing engine550(e.g., “not now, Charlie”). (As used herein, non-input speech can include input audio that is not provided by a particular user and/or that is not directed toward a speech processing system.)

A segmentation process can divide a raw speech signal into individual segments of audio that can be individually evaluated as corresponding to input speech or non-input speech.FIG.6Billustrates an example segmentation of raw speech signal600into segments of audio. Segments can include phonemes, words, phrases, sentences, utterances, or combinations of any of the above. For each segment, example system500can determine whether the segment corresponds to input speech or non-input speech, with the results of the determination used to determine whether the segment should be included or excluded from directed speech signal540. As shown inFIG.6B, a segment of signal600can be expressed as a region of signal600that lies between two points in time (e.g., along the invariant t axis). For example, in the figure, a first segment601(e.g., corresponding to “What's the weather”) lies between points t0and t1; a second segment602(e.g., corresponding to non-speech, such as background noise) lies between points t1and t2; a third segment603(e.g., corresponding to “not now”) lies between points t2and t3; a fourth segment604(e.g., corresponding to “Charlie”) lies between points t3and t4; a fifth segment605(e.g., corresponding to non-speech, such as background noise) lies between points t4and t5; a sixth segment606(e.g., corresponding to “tomorrow”) lies between points t5and t6; and a seventh segment607(e.g., corresponding to non-speech, such as background noise) lies between points t6and t7.

The boundaries of such segments can be determined according to one or more suitable techniques. For example, various techniques known in the art can be used to determine boundaries of spoken words or phrases. According to some such techniques, boundaries between segments can be determined based on, for example, periods of relative silence (indicating gaps between “chunks” of speech); changes in pitch or intonation (which may indicate the start or end of a word, phrase, or idea); changes in the cadence of speech (which can indicate the start or end or a word, phrase, or idea, or a transition from one word, phrase, or idea to another); breathing patterns (which can indicate the speaker is about to begin a new word, phrase, or idea); and so on. In some examples, statistical analysis of a speech signal can be useful to identify segment boundaries; for example, portions of the speech signal that represent statistical outliers in the signal (e.g., portions of the speech signal comprising frequency components not commonly found elsewhere in the signal) can signify the start or end of a word, phrase, or idea. Various machine learning techniques can also be used to identify segment boundaries.

In some examples, sensor data520can be used to segment a speech signal (e.g., the raw speech signal510), by indicating potential separation points where a user may be likely to change the target of their speech (e.g., transitioning from speaking to a speech processing engine to speaking to another person in the room). For instance, sensor data may indicate when a user turns their head, changes the focus of their eye gaze, or moves to a different location in the room. Sudden changes in such sensor data can be used to indicate boundaries between speech segments.

The lengths (e.g., average time, or number of syllables) of speech segments may vary. In some examples, segments may generally be on the order of several words, such as may make up a spoken phrase. In some examples, segments may be longer (e.g., constituting one or more full sentences or utterances), or shorter (e.g., constituting individual words, or even individual syllables). As described herein, speech can be included or excluded from directed speech signal540on a per-segment basis, such that for each segment, either the entire segment is included, or the entire segment is excluded. Utilizing longer segments can increase the risk that a single segment will include both input speech and non-input speech, which can cause undesirable results: excluding such a segment from directed speech signal540would result in failing to present the user's input speech to speech processing engine550, while including it would present non-input speech to speech processing engine550—an opposite goal of generating directed speech signal540. While using shorter segments can reduce this problem, it presents a possible tradeoff in the computational overhead (and accompanying latency) required to process additional segments for a single speech signal. A desirable balance of segment size may be to group, to the extent possible, single related words or thoughts in a single segment, such that the entire segment is, or is not, directed to speech processing engine550. For example, in example signal600, “What's the weather” and “not now” each constitute a single chunk of speech that rises or falls together, and may thus be beneficial to group as a single segment. However, segments may be arbitrarily large or arbitrarily small (including segments as small as a single digital audio sample), and the present disclosure is not limited to any particular segmentation size.

In some examples, segmentation may be performed on a prerecorded speech signal, where the entire speech signal is captured before it is segmented. Segmentation may be comparatively more accurate and/or efficient in such examples, as knowledge of the entire speech signal can be used to generate more meaningful speech segments; that is, which portions of the speech signal should be segmented together can be easier to determine when the entire signal is known. However, in some examples, “live” speech may be segmented as it is being detected. Techniques for segmenting prerecorded speech signals may also be used to segment live speech signals (for example, by applying such techniques to buffered chunks of live speech). In some cases, segmentation decisions on live speech may need to be periodically revisited as new speech clarifies the intention of previous speech. Additionally, portions of speech can be flagged for manual review, where they can later be evaluated and corrected manually.

FIG.6Cdemonstrates an example probability model610corresponding to speech signal600. In the example, probability model610can express, as a function of time t, a probability p(t) that the segment of the corresponding audio signal600at time t is user speech directed at speech processing engine550. (Alternatively, in some examples, p(t) can describe the probability that the segment is not user speech directed at the speech processing engine.) For instance, in the example, at a time tk1that falls between t0and t1, p(tk1) is equal to 0.9, indicating that the portion of speech signal600at time tk1(V(tk1), e.g., “weather”) has a 90% probability of being user speech directed to speech processing engine550. Similarly, at a time tk2that falls between t3and t4, p(tk2) is equal to 0.1, indicating that the portion of speech signal600at time tk2(V(tk2), e.g., “Charlie”) has a 10% probability of being user speech directed to speech processing engine550.

As shown in the figure, probability p(t) can be determined on a per-segment basis, such that for a segment that begins at time t0and ends at time t1, p(t) remains constant between p(t0) and p(t1) (that is, the entire segment will have the same probability value). Accordingly, in probability model610, segment601(“What's the weather”) has a corresponding probability value611of 0.9; segment603(“not now”) has a corresponding probability value613of 0.3; segment604(“Charlie”) has a corresponding probability value614of 0.1; and segment606(“tomorrow”) has a corresponding probability value616of 0.6. In the figure, the remaining segments (i.e., segments602,605, and607, which may correspond to background noise or other non-speech audio) have corresponding probability values (i.e.,612,615, and617, respectively) of zero.

Classifying Input Speech

Determining a probability value for a speech segment can be referred to as “classifying” the speech segment, and a module or process for performing this determination (e.g.,562,568,574) can be referred to as a “classifier.”FIGS.7A,7B, and7Cillustrate example classifiers of example system500for determining a probability value for a segment of a speech signal (e.g., segments610of speech signal600described above). This determination can be performed using the speech signal itself (e.g., as shown inFIG.7A); using sensor data associated with the user (e.g., as shown inFIG.7B); or using some combination of the speech signal and the sensor data (e.g., as shown inFIG.7C).

In the example shown inFIG.7A, speech segment516, statistical data512for the speech signal, and/or a speech data repository527are used by classifier562to determine a probability value566with which the speech segment516corresponds to input speech (e.g., user speech directed at a speech recognition system). At stage563, speech segment516can be parameterized/characterized according to one or more parameters, such as by using statistical data512of the speech signal. This can facilitate classifying the speech segment based on speech data repository527. Speech data repository527may be stored in a database. A Fourier transform of a time-based speech segment516can be performed in order to provide a spectral representation of the speech segment (e.g., a function of frequency indicating the relative prevalence of various frequency parameters in the speech segment516). In some cases, speech segment516can be compared against statistical data512to determine a degree to which speech segment516deviates from the larger speech signal of which it is a part. For instance, this can indicate levels of (or changes in) volume or component frequencies of the speech segment that can be used at stage564to characterize the speech segment. In some examples, aspects of the speaker—for example, the speaker's age, sex, and/or native language—can be used as parameters to characterize the speech segment516. Other ways in which speech segment516can be parameterized, with such parameters used to characterize the speech segment at stage564, will be apparent to those skilled in the art. As examples, speech segment516can be preprocessed with pre-emphasis, spectral analysis, loudness analysis, DCT/MFCC/LPC/MQ analysis, Mel filter bank filtering, noise reduction, band-pass filtering of the signal to the most useful speech range (e.g., 85-8000 Hz), and dynamic range compression. The remaining signal can then be parameterized into a set of time-invariant features (e.g., speaker identification/biometrics, gender identification, mean fundamental frequency, mean loudness) and time-varying feature vectors (e.g., formant center frequencies and bandwidths, fundamental frequency, DCT/MFCC/LPC/MQ coefficients, phoneme identification, consonant identification, pitch contour, loudness contour).

At stage564of the example, a probability value566is determined that speech segment516corresponds to input speech. Probability value566can be determined using speech data repository527. For example, a database including speech data repository527can identify, for elements of speech in the database, whether those elements correspond to input speech. Various types of data may be represented in speech data repository527. In some examples, speech data repository527can include a set of audio waveforms corresponding to speech segments; and can indicate, for each waveform, whether the corresponding speech segment belongs to input speech. In some examples, instead of or in addition to audio waveforms, speech data repository527can include audio parameters that correspond to the speech segments. Speech segment516can be compared with the speech segments of speech data repository527—for example, by comparing an audio waveform of speech segment516with audio waveforms of speech data repository527, or by comparing parameters of speech segment516(such as may be characterized at stage563) with analogous parameters of speech data repository527. Based on such comparisons, probability566can be determined for speech segment516. (Methods for creating the data in speech data repository527are described below.)

Techniques for determining probability566will be familiar to those skilled in the art. For instance, in some examples, nearest neighbor interpolation can be used at stage564to compare speech segment516to similar speech segments in an N-dimensional space (in which the N dimensions can comprise, for example, audio parameters and/or audio waveform data described above); and to determine probability value566based on the relative distances between speech segment516and its neighbors in the N-dimensional space. As another example, support vector machines can be used at stage564to determine, based on speech data repository527, a basis for classifying a speech segment as either an input speech segment or a non-input speech segment; and for classifying speech segment516(e.g., determining a probability value566that the speech segment is input speech) according to that basis. Other suitable techniques for analyzing speech segment516and/or speech data repository527, comparing speech segment516to speech data repository527, and/or classifying speech segment516based on speech data repository527in order to determine probability566will be apparent; the disclosure is not limited to any particular technique or combination of techniques.

In some examples, machine learning techniques can be used, alone or in combination with other techniques described herein, to determine probability value566. For example, a neural network could be trained on speech data repository527, and applied to speech segment516to determine probability value566for speech segment516. As another example, a genetic algorithm can be used to determine a function, based on speech data repository527, for determining the probability566for speech segment516. Other suitable machine learning techniques, which will be familiar to those skilled in the art, will be apparent; the disclosure is not limited to any particular technique or combination of techniques.

In some examples, the probability value566for speech segment516may be influenced by other speech segments of the same speech signal. For instance, users may be unlikely to provide input in short bursts, surrounded by non-input speech (or vice versa); instead, users may be more likely to provide speech recognition input in largely contiguous sequences. That is, all other factors equal, a speech segment516is more likely to be an input speech segment if the segments that come immediately before or after it are also input speech segments; and vice versa. In such examples, probabilistic techniques (e.g., Bayesian networks, hidden Markov models) can be used at stage564, alone or in combination with other techniques described herein, to determine probability566. Various probabilistic techniques can be suitable for this purpose, and the disclosure is not limited to any particular technique or combination of techniques.

In some examples, speech data repository527can be generated by recording a set of speech signals of various speech sources, and identifying, for each portion of each speech signal, a speech target of that portion. For instance, a user could be observed interacting with a group of people, with a speech recognition system present in the same room, as the user's speech (and/or other audio) is recorded. The observer can identify, for each region of the recorded speech, whether that region of speech was directed from the user (and not some other source) as input to the speech recognition system, or to some other target. This information can be apparent to the observer by observing the context in which the user is speaking—commonly, it is easy and intuitive for humans (unlike machines) to determine, based on an observation of a user, whether the user is speaking to a speech recognition system, or to something else. This process can be repeated for multiple users, and in some cases for non-human speakers (e.g., pets, TV speakers, appliances), until a sufficiently large and diverse set of speech data (e.g., audio waveform data, and/or parameters associated with the speech as described above) is generated. From this speech data, individual speech segments can be determined; these speech segments can be associated with the observer's determination of whether or not the corresponding speech is directed by the user to a speech recognition system.

In the example shown inFIG.7A, as described above, probability value566is determined based on the user's own speech as detected by one or more microphones. Accordingly, the predictive value of this system with respect to probability value566—that is, the degree to which the example ofFIG.7Aenables probability value566to be determined more accurately than otherwise—is limited by the degree of correlation between the audio characteristics of a speech signal, and whether the speech signal is input speech. The greater the degree of correlation, the more useful the speech signal will be in determining which portions of the signal are input speech. While there may be at least some such correlation between the speech audio and the intended target, correlation may also exist between the intended target of the speech, and sensor data associated with the speaker, such as sensor data520; accordingly, the overall predictive value of the system can be improved by incorporating sensor data520, alone or in addition to raw speech signal510, such as described below with respect toFIGS.7B and7C.

FIG.7Billustrates an example portion of example system500, in which sensor data520is used by classifier568to determine a probability value572with which the speech segment516is input speech. In some examples, as described above, sensor data520can correspond to data from sensors such as described above with respect to example wearable head device100inFIG.1. As described above, such a wearable system can include one or more sensors that can provide input about the user and/or the environment of the wearable system. For instance, wearable head device100can include a camera (e.g., camera444described inFIG.4) to output visual signals corresponding to the environment; in some examples, the camera can be a forward-facing camera on a head-mounted unit that shows what is currently in front of the user of the wearable system. In some examples, wearable head device100can include a LIDAR unit, a radar unit, and/or acoustic sensors, which can output signals corresponding to the physical geometry (e.g., walls, physical objects) of the user's environment. In some examples, wearable head device100can include a GPS unit, which can indicate geographic coordinates corresponding to the wearable system's current location. In some examples, wearable head device100can include an accelerometer, a gyroscope; and/or an inertial measurement unit (IMU) to indicate an orientation of the wearable head device100. In some examples, wearable head device100can include environmental sensors, such as temperature or pressure sensors. In some examples, wearable head device100can include biometric sensors, such as iris cameras; fingerprint sensors; eye tracking sensors (e.g., electrooculography (EOG) sensors) to measure a user's eye movements or eye gaze; or sensors to measure a user's vital signs. In examples where wearable head device100includes a head-mounted unit, such orientation can correspond to an orientation of the user's head (and, by extension, the user's mouth and a direction of the user's speech). Other suitable sensors can be included and can provide sensor data520. Moreover, in some examples, sensors other than those of a wearable system can be utilized as appropriate. For instance, sensors associated with one or more microphones of a speech recognition system (e.g., GPS, IMU) could be used to in conjunction with sensors of a wearable system to determine a relative distance and orientation between the user and the speech recognition system.

In the example shown inFIG.7B, stage569can parameterize/characterize speech segment516according to one or more parameters, such as described above with respect to stage563, with respect to aspects of sensor data520. This can facilitate classifying the speech segment based on sensor data520. For instance, stage569can perform a Fourier transform of signals of sensor data520(e.g., signals describing a user's position or orientation (e.g., from GPS, acoustic, radar, or IMU sensors) as a function of time elapsed during the speech segment) in order to determine a spectral representation of those signals. As examples, speech segment516can be characterized according to the user's eye movements (e.g., from EOG sensors), eye gaze targets (e.g., from cameras or EOG sensors), and/or visual targets (e.g., from RGB cameras or LIDAR units). In some examples, sensor data520can be compared to a broader range of sensor data (e.g., sensor data captured over a period of several minutes prior to the start of the speech signal) to determine the degree to which sensor data520deviates from the broader range of sensor data. Other ways in which sensor data520can be parameterized, with such parameters used to characterize the speech segment at stage564, will be apparent to those skilled in the art. As described above with respect to speech segment516, speech segment564can be preprocessed with pre-emphasis, spectral analysis, loudness analysis, DCT/MFCC/LPC/MQ analysis, Mel filter bank filtering, noise reduction, band-pass filtering of the signal to the most useful speech range (e.g., 85-8000 Hz), and dynamic range compression. The remaining signal can then be parameterized into a set of time-invariant features (e.g., speaker identification/biometrics, gender identification, mean fundamental frequency, mean loudness) and time-varying feature vectors (e.g., formant center frequencies and bandwidths, fundamental frequency, DCT/MFCC/LPC/MQ coefficients, phoneme identification, consonant identification, pitch contour, loudness contour).

At stage570of the example, a probability value572is determined that speech segment516corresponds to input speech. In some approaches, probability value572can be determined using a sensor data repository528, which can include a database identifying, for elements of speech in the database, whether those elements correspond to input speech. In some examples, sensor data repository528can include data sets representing sensor measurements (e.g., sequences of a user's head position, orientation, and/or eye gaze over time) corresponding to speech segments; and can indicate, for each data set, whether the corresponding speech segment belongs to input speech. In some examples, instead of or in addition to sensor data sets, sensor data repository528can include parameters that correspond to the speech segments. Speech segment516can be compared with sensor data repository528—for example, by comparing raw sensor data520with corresponding signals of sensor data repository528, or by comparing parameters of speech segment516(such as may be characterized at stage569) with analogous parameters of sensor data repository528. Based on such comparisons, probability572can be determined for speech segment516.

Techniques for determining probability572will be familiar to those skilled in the art. For example, the techniques described above with respect to determining probability value566—e.g., nearest neighbor interpolation, support vector machines, neural networks, genetic algorithms, probabilistic techniques such as Bayesian networks or Markov networks, or any combination of the above—can be applied to sensor data repository528and sensor data520in an analogous fashion. Other techniques will be apparent, and the disclosure is not limited to any particular technique or combination of techniques.

In some examples, sensor data repository528need not be accessed directly by classifier568in order to classify speech segment516at stage570. For example, stage570can apply one or more rules to determine, based on sensor data520, a probability value572with which speech segment516corresponds to input speech. For instance, it can be determined at stage570, based on sensor data520(e.g., data from position and orientation sensors), that the user is facing the microphone (or turned to face the microphone shortly before uttering speech segment516); and it can then be determined from this information that speech segment516is likely to be input speech. Conversely, it can be determined at stage570that the user is facing away from the speech processing engine microphone (or recently turned to face away from the microphone), and that speech segment516is unlikely to be input speech. This is because humans generally tend to face the object to which their speech is directed, whether that object is a person or a device. Similarly, it can be determined at stage570, based on sensor data520(e.g., data from cameras or EOG sensors), that the user is looking at the microphone (or recently shifted their eye gaze toward the microphone), and that speech segment516is likely to be input speech. Conversely, it can be determined that the user is not looking at the microphone, and that the speech segment is unlikely to be input speech. As another example, if sensor data520(e.g., camera data) indicates that the user is looking directly at another person while uttering speech segment516, it can be determined that speech segment516is unlikely to be input speech (i.e., that the speech is instead directed at the person the user is looking at). Rules for determining how to classify a probability value572based on sensor data can be determined using machine learning techniques familiar to those skilled in the art, such as neural networks or genetic algorithms, using sensor data repository528as a training set.

In some examples, sensor data repository528can be generated similarly to speech data repository527as described above. For instance, data of sensor data repository528can be generated by recording a set of speech signals of various speech sources, with accompanying sensor data generated at the same time as the speech signals; and identifying, for each portion of each speech signal, a speech target of that portion. For instance, a user could be observed interacting with a group of people, with a speech recognition system present in the same room, as the user's speech is recorded. The observer can identify, for each region of the recorded speech, whether that region of speech was directed as input from the user to the speech recognition system, or to some other target. From this speech and/or sensor data, individual speech segments can be determined; these speech segments, and their accompanying sensor data, can be associated with the observer's determination of whether or not the corresponding speech is directed by the user to a speech recognition system.

Sensor data520can also be used at stage570to identify whether or not microphone input belongs to a particular user. For example, the amplitude of a user's speech, as detected by one or more microphones, can be expected to fall within a predictable range that falls off as a function of the distance between the microphone and the user, and that changes as a function of the relative orientation of the user with respect to the microphone (e.g., falls off as the user faces away from the microphone). (In some cases, this range can be determined experimentally for a particular user.) If sensor data520(e.g., GPS data, camera data, acoustic data, radar data) indicates that the user is a particular distance from the microphone, a range of expected amplitudes of that user's speech for that particular distance can be determined. Microphone input that falls outside of that amplitude range can be rejected as belonging to a source other than the user. Likewise, other speech characteristics (e.g., high frequency content) can be predicted based on the user's position, orientation, or other sensor data520; and microphone input that is inconsistent with that sensor data can be rejected. Similarly, microphone input that changes significantly (e.g., in volume or frequency characteristics) while the user's position and orientation remain constant (or vice versa) can be rejected. And conversely, microphone input that is consistent with predicted characteristics of a user's speech, based on sensor data, can reinforce that the microphone input belongs to that user. Other techniques of identifying a source of microphone input, based on sensor data, will be apparent to those skilled in the art.

InFIG.7B, as described above, probability value572is determined based on the user's own speech as detected by one or more microphones. As with the example shown inFIG.7Aand probability value566, the predictive value of this system with respect to probability value572is limited by the degree of correlation between the intended target of a speech signal, and the accompanying sensor data produced alongside the speech signal. The greater the correlation, the more useful the sensor data will be in determining which portions of the signal are input speech. Such a correlation reflects that sensor data (such as from sensors of a wearable system, like those described above) can provide many of the same body language cues that humans use to interpret and contextualize others' speech. For example, humans are accustomed to determining a speaker's intended speech target using the speaker's position (e.g., the speaker's movement, and distance from the listener); orientation (e.g., to whom the speaker is facing); eye gaze (e.g., who the speaker is making eye contact with); gesticulation (e.g., hand and arm movements, facial expressions); and so forth. Many of these body language cues also apply even when the speaker is addressing a device, such as a microphone-enabled speech recognition system. Sensor data can correspond to this body language, such as by providing data indicating the speaker's position, orientation, eye patterns, movement, and so on. Accordingly, using sensor data such as described above can provide valuable information as to the intended target of the corresponding speech.

In some examples, the predictive value of the system can be improved by utilizing both speech data (e.g., as described with respect toFIG.7A) and sensor data (e.g., as described above with respect toFIG.7B) that corresponds to the same speech signal. For example, where a speech segment corresponds to both a speech cue (e.g., the user raises their voice) and a sensor cue (e.g., the user quickly turns their head), the two cues combined can provide strong predictive evidence that the speech segment is intended as input from the user to a speech processing engine.

FIG.7Cillustrates an example portion of example system500in which analysis data512for a speech signal (e.g., speech signal510), and sensor data520are both used by classifier574to determine a probability value578with which the speech segment516is directed by the user to a speech processing engine. Stages of the example system shown can proceed as described above with respect toFIGS.7A and7B. For instance, stage575can parameterize/characterize speech segment516based on speech characteristics determined from speech signal510and/or speech signal analysis data512, such as described above with respect to stage563ofFIG.7A; and stage575can also parameterize/characterize speech segment516based on sensor data520, such as described above with respect to stage569ofFIG.7B. At stage576, a probability value578can be determined for speech segment516based on its speech characteristics, such as described above with respect to stage564ofFIG.7A; and based further on its corresponding sensor data, such as described above with respect to stage570ofFIG.7B. This probability value determination can make use of speech and/or sensor data, such as in a speech/sensor data repository529. Speech/sensor data repository529can include a database including information relating speech data to an intended target of that speech, such as described above with respect to speech data repository527ofFIG.7A; and can further include information relating sensor data to an intended target of its corresponding speech, such as described above with respect to sensor data repository528ofFIG.7B. Further, speech/sensor data repository529can include information relating combinations of speech data and sensor data to an intended speech target. This may be useful in situations where neither the speech data nor the sensor data itself is independently predictive of an intended speech target, but the combination of the two correlates strongly to an intended speech target and has greater predictive value.

Generating a Probability Model

FIG.8is a flow chart showing a portion of example system500, illustrating an example of generating a probability model586from a raw speech signal510, according to some embodiments. InFIG.8, stage560generates a probability model586(which may correspond to probability model610, described above with respect toFIG.6C) from a raw speech signal510(which may correspond to signal600, described above with respect toFIGS.6A-6B) and sensor data520. At stage560, statistical data512for the speech signal (e.g., representing statistical analysis of speech signal510such as described above) can be generated according to techniques familiar to those skilled in the art. At stage514of stage560, speech signal510can be segmented into individual speech segments516, such as described above with respect toFIGS.6A-6D. For each speech segment516, one or more classifiers (e.g.,562,568,574described above) can be applied to generate a probability value, corresponding to the probability that the segment is input speech. In the example shown inFIG.8, three classifiers are applied: a first classifier (562) generates a first probability value566based on the speech segment516and speech data512, such as described above with respect toFIG.7A; a second classifier (568) generates a second probability value572based on the speech segment516and sensor data520, such as described above with respect toFIG.7B; and a third classifier (574) generates a third probability value578based on the speech segment516, speech data512, and sensor data520, such as described above with respect toFIG.7C. However, in some examples, only one classifier (e.g., classifier574) need be used; and in some examples, additional classifiers beyond the three described here may be utilized to generate additional respective probability values. In some cases, different classifiers can apply different metrics to determine respective probability values.

In some examples where multiple classifiers are used to determine multiple respective probability values for speech segment516—such as the example shown inFIG.8, where classifiers562,568, and574are used to generate probability values566,572, and578, respectively—it may be necessary to determine an overall probability582for speech segment516, based on the individual probability values generated by their respective classifiers. In such examples, comparison logic580can be used to mediate among the individual probability values to determine overall probability582. In some examples, comparison logic580may compute overall probability582as an average of individual probabilities (e.g.,566,572,578). In some examples, comparison logic580may compute overall probability582as a weighted average of the individual probabilities, weighted for example by the fidelity of the input data (e.g., speech data512, sensor data520). Other suitable techniques that can be employed by comparison logic580will be familiar to those skilled in the art, and the disclosure is not limited to any such technique or combination of techniques. Example techniques for combining the outputs of multiple classifiers include ensemble learning; Bayes optimal classifier, bagging (bootstrap aggregating), boosting techniques (e.g., AdaBoost); bucket of models; and stacking.

Once a probability value for a speech segment516has been determined, such as described above, the process of determining a probability value can repeat (stage584) for any remaining speech segments516. For example, speech signal600, described above with respect toFIGS.6A-6D, can be divided into seven speech segments (601through607), such as described above; if this speech signal600were provided as input510to the system shown inFIG.8, each of stages562,568, and574might be applied to each of the seven speech segments, resulting in a probability value582for each of the segments. Once a probability value has been determined for each speech segment516, the probability values can be used to generate a probability model586. As described above, probability model586can indicate a probability value for each speech segment of a speech signal. For example, inFIG.6C, probability model610indicates a probability value for each speech segment of speech signal600. Generating probability model586for a speech signal can include expressing a probability value as a function of elapsed time of the speech signal; with such a model, such as shown as model610inFIG.6C, a time t can be applied as input to the model, and the model will indicate the probability that the portion of the speech signal corresponding to time t (e.g., the portion of speech signal600after t seconds have elapsed) is directed as input to a speech processing engine. However, other suitable implementations of probability model586will be apparent and are within the scope of the disclosure.

Determining a Directed Speech Signal

FIG.9illustrates a portion of example system500, by which system500determines a directed speech signal540from raw speech signal510and/or sensor data520, such as by using probability model586described above. As shown inFIG.9, at stage530, system500can generate a directed audio signal540, which can be an input speech signal to a speech processing engine that includes speech directed by a user to the speech processing engine, while excluding speech not directed by the user to the speech processing engine. Directed audio signal540can correspond to signal620described above with respect toFIG.6D. An example of stage530generating directed audio signal540can proceed as follows with reference toFIG.9. At stage560, raw speech signal510and/or sensor data520can be used to determine, for each of one or more segments of raw speech signal510, a probability that the segment corresponds to speech directed by the user as input to a speech processing engine. An example implementation of stage560is described above with respect toFIG.8. As described above, the output of target determination stage560can be represented as probability model586, which can express, for example as a function of elapsed time, the probability that a portion of speech signal510is user speech directed at the speech processing engine. For example, model586can be a mathematical function expressing, for each time t of a raw speech signal having one or more segments, the probability that a segment of that raw speech signal corresponding to that time t is directed at the speech processing engine. As shown in the example inFIG.9, stage560can also output a passthrough signal588, which may be a buffered signal corresponding to the raw speech signal510provided to target determination stage560.

At stage590of the example inFIG.9, the raw speech signal (e.g., passthrough signal588) can be filtered based on the probabilistic model586, such that segments of the raw speech signal510that correspond, with a sufficiently high probability, to input speech can be included in directed audio signal540; and conversely, segments of raw speech signal510that do not correspond to input speech can be excluded from directed audio signal540. Stage590can employ a threshold probability value to serve as a cutoff to determine what constitutes a sufficiently high probability for an audio segment to be included in directed audio signal540. For example, as described above,FIG.6Cillustrates a probability model610that corresponds to the raw speech signal600shown inFIGS.6A and6B. As described above with respect toFIG.6C, probability model610indicates, for each of speech segments601through607of speech signal600, a probability that the speech segment corresponds to input speech. InFIG.6C, threshold value618is a value of 0.5; however, other threshold values can be used as appropriate. At stage590, speech segments with corresponding probability values that meet or exceed threshold value618(e.g., speech segments601and606) could be included in directed audio waveform540; and segments whose corresponding probability values do not meet threshold value618(e.g., speech segments602,603,604,605, and607) could be excluded from directed audio waveform540. The result would be the audio waveform620shown inFIG.6D, in which only speech segments with sufficiently high probability (“What's the weather” and “tomorrow”) are included in the waveform620, and remaining segments are excluded. Compared to providing the raw speech signal600to the speech recognition system, providing audio waveform620as input to the speech recognition system promotes accuracy and computational efficiency, because the speech recognition system does not need to waste computational resources on irrelevant speech (or other audio) that carries a risk of generating erroneous results.

Training Classifiers

FIG.10illustrates an example process1000for capturing audio and non-audio classifier training data, according to one or more examples of the disclosure. Process1000can be applied to a human test subject1012, interacting (as a user might) with a speech processing engine (e.g., as included in a device with an integrated voice assistant). One or more microphones and one or more sensors can be configured to capture audio data and non-audio data (e.g., sensor data), respectively, from test subject1012. In some embodiments, the non-audio data may be non-microphone sensor data such as, for example, inertial measurement unit data, visual data, and the like. At step1010of the process, raw audio data of the voice of test subject592can be captured via the one or more microphones. Similarly, at step1020, non-audio data of the test subject can be captured via the one or more sensors. In some cases, test subject1012can be equipped with a single device, such as a wearable head device such as described above, that can include one or more microphones and one or more sensors. These microphones and sensors can be configured to for capturing the audio data at step1010and the non-audio data at step1020, respectively. Steps1010and1020can be performed simultaneously.

At step1030, the audio captured at step1010can be segmented and tagged as either input speech or non-input speech. This may be an automated process, a manual process, or some combination thereof. For example, audio data captured at step1010can be presented to a voice-activity detector (VAD) or to a human “tagger” observing test subject1012, and the audio data can be manually separated by the tagger into individual phrases or portions thereof. The tagger can then, based on the tagger's observation of test subject1012interacting with the speech recognition engine, manually identify each phrase as input speech or non-input speech. In some cases, the tagger can annotate each phrase with various metadata (e.g., an intended recipient for each phrase, or the audio source of each phrase). Other metadata entered by the tagger can include aspects about the speaker (e.g., the speaker's age, sex, and/or native language). In some examples, the tagger can also segment and tag non-speech audio (e.g., background noise and/or speech from people other than the speaker).

Similarly, at step1040, non-audio data captured at step1020can also be segmented and tagged as either being directed to the speech processing engine, or not. In some examples, a human tagger can identify and/or isolate non-audio data (e.g., sensor data) associated with individual phrases spoken by test subject1012, described above. In some cases, the tagger can manually associate non-audio data with audio data to which it corresponds. In some examples, non-audio data can be automatically associated with each phrase, based on start and end times of segmented and classified phrases from step1030. In some examples, non-audio data can include information about a user's head pose, gaze, gestures, location relative to target recipient phrases, or any other sensor data captured.

At step1050, the audio captured at step1010, the segmented and tagged phrases from step1030(e.g., input speech and non-input speech, including background noise or non-speech audio), the non-audio data captured at step1020, and/or the segmented and tagged non-audio data from step1040can be stored in a repository for classifier training. For example, speech data repository527described above can store audio from step1010and/or phrases from step1030; sensor data repository528can store non-audio data from step1020and/or step1040; and speech/sensor data repository529can store any of the above. In some examples, the audio captured at step1010and/or the segmented and tagged phrases from step1030are stored separately from the non-audio data captured step1020, and/or the segmented and tagged non-audio data from step1040(e.g., audio data and non-audio data are stored in separate databases). The stored audio data and/or non-audio data can be used to train classifiers, such as described above.

In some embodiments, audio and/or non-audio characteristics can be extracted from the input speech, non-input speech, or non-speech (e.g., background noise) stored in the one or more databases from step1050ofFIG.10. Examples of audio characteristics can include levels of (or changes in) volume (or signal amplitude), pre-vocalization hesitation, intra utterance hesitation, disfluency (e.g., stuttering, repetition), speech rate, syntax, grammar, vocabulary, length of phrase (e.g., duration, word count), pitch (e.g., fluctuation and contour), and/or prosody. Examples of non-audio characteristics that can be extracted from non-audio data include gestures, gaze (and changes thereto), head pose (and changes thereto), and position (e.g., distance and orientation) to physical and/or virtual objects (and changes thereto). In some examples, a Fourier transform of each speech and/or non-speech segment (e.g., each audio and/or non-audio segment corresponding to input speech, non-input speech, and/or non-speech) is stored in step1050ofFIG.10(e.g., both input speech and non-input speech) and provides a spectral representation of each speech segment (e.g., a function of frequency indicating the relative prevalence of various frequency parameters in the speech segment). Other methods of extracting time, frequency, and combined time-frequency parametric representations of audio and non-audio data will be familiar to those skilled in the art. In some examples, the extracted audio and/or non-audio characteristics can be stored with the corresponding input speech, non-input speech, and/or non-speech.

In some embodiments, the segmented and annotated audio data and non-audio data captured through process1000ofFIG.10(e.g., the input speech, non-input speech, and/or non-speech with corresponding metadata) can be fed into one or more classifiers for training purposes, such as described above. By running sample classes of input speech, non-input speech, and non-speech through one or more classifiers, the one or more classifiers can be trained to recognize input speech, non-input speech, and/or non-speech. In some examples, a majority subset (e.g., 60%) of the segmented and annotated audio data and non-audio data are run through the one or more classifiers and a minority subset or remaining (e.g., 40%) segmented and annotated audio data and non-audio data are used to evaluate the one or more classifiers. Evaluation techniques will be familiar to those skilled in the art. In some embodiments, these classifiers can be further trained by enabling users to confirm or reject classifications.

As described above, one or more classifiers (e.g., naive Bayes classifiers, support vector machines, k-nearest neighbor classifiers, AdaBoost classifiers, decision trees, or artificial neural networks) to distinguish between input speech and non-input speech. These classifiers can be trained to recognize audio characteristics and non-audio characteristics associated with input speech and/or non-input speech for improved speech processing. A method to train classifiers in accordance with the disclosure can include capturing audio and/or non-audio data; extracting audio and/or non-audio characteristics of input speech and non-input speech; training one or more classifiers, for example, using machine learning techniques, and/or, in some examples, updating the classifiers for improved input speech identification (e.g., by confirming and/or rejecting classifications), as described below.

FIG.11illustrates an example environment that can be used to generate audio data and sensor data for classifier training. The figure illustrates test subject592(which may correspond to test subject1012described above) in an environment591that includes a voice target (such as a voice assistant device including a speech processing engine), and one or more “distractor” sources.593A-593H. The distractor sources are configured to present test subject592with audio or visual “distractor” stimuli, to which test subject592may respond. Audio data and non-audio data (e.g., sensor data) associated with a response of test subject592to these distractor stimuli can be detected; this audio data and non-audio data can describe the response of test subject592(as detected by microphones and sensors) to external stimuli presented from the location of the corresponding distractor source. This audio data and non-audio data can be used accordingly to train a classifier (such as described above) to distinguish input speech from non-input speech (e.g., speech directed at an external stimulus, represented by the distractor source).

Distractor sources593A-593H can be placed at varying distances from and angles to test subject592, such as shown in the figure. Distractor sources593A-593H can be presented as speakers or visuals, or as any other suitable object that can produce sound and/or visuals (e.g., human beings, animals, electronic devices, etc.). For example, distractor source593A can represent a smart home device (e.g., a speaker with an integrated “smart” voice assistant (a “smart speaker”)) and distractor source593B can represent a human; the audio data and non-audio data can reflect differences in the response of test subject592based on the apparent identity of the distractor source. Environment591can represent a controlled environment (e.g., a sound proof room, or a room in which distractor sources593A-593H produce sound in a controlled fashion) or an uncontrolled environment (e.g., in the home of test subject592or in a public place). For example, in a controlled environment, test subject592can freely interact (e.g., with little to no direction or script) with a wearable device with an integrated voice assistant (e.g., wearable head device100) to instruct the device to perform a particular operation (e.g., open an app, play music, query information, for example, from the Internet, enter information into calendar, read information from a calendar, make a phone call, send a text message, control a smart thermostat, control a smart lock, control one or more smart lights, or any other operation). Test personnel (represented by distractor sources593A-593H) can engage in conversation with test subject592. This prompts test subject592to interact with wearable device and the test personnel. In some examples, distractor sources593A-593H can be virtual sources; for example, a software application running on a wearable system can produce sound from one or more virtual sound sources represented by distractor sources593A-593H. In some examples, distractor sources593A-593H may be presented via a wearable head device worn by test subject592(e.g., via speakers and/or a display of the wearable head device), with audio data and non-audio data potentially captured by microphones and sensors of that same wearable device.

Interactions such as shown inFIG.11(e.g., spoken phrases594A-594D spoken in the environment591) can be detected and used to train one or more classifiers in accordance with this disclosure. For example, spoken phrases594A-594D can be recorded (e.g., by one or more microphones150on wearable head device100or by one or more microphones on sound source594A) in an audio file as a continuous audio stream: “Hey Magic Leap, open . . . . Mom, can I . . . . Not right now, Charlie . . . open Maps.” Similarly, non-audio data of test subject592interacting with one or more distractor sources593A-593H can be captured simultaneously with the audio data. In some examples, data from one or more sensors on a wearable system (e.g., wearable head device100inFIG.1and/or handheld controller200inFIG.2) on test subject592can be used to capture information about the head positions of test subject592(e.g., as detected by position and orientation sensors of the wearable head device), hand gestures (e.g., as detected by movements of handheld controller200or by one or more cameras130A and130B configured on wearable head device100), eye gaze (e.g., as detected by one or more cameras128A and102B configured on wearable head device100), and/or the distance of test subject592from one or more distractor sources593A-593H (e.g., as measured from the wearable head device100to one or more of distractor sources593A-593H by one or more cameras130A and130B and/or GPS, acoustic, radar, or IMU sensors).

With respect to the systems and methods described above, elements of the systems and methods can be implemented by one or more computer processors (e.g., CPUs or DSPs) as appropriate. The disclosure is not limited to any particular configuration of computer hardware, including computer processors, used to implement these elements. In some cases, multiple computer systems can be employed to implement the systems and methods described above. For example, a first computer processor (e.g., a processor of a wearable device coupled to one or more microphones) can be utilized to receive input microphone signals, and perform initial processing of those signals (e.g., signal conditioning and/or segmentation, such as described above). A second (and perhaps more computationally powerful) processor can then be utilized to perform more computationally intensive processing, such as determining probability values associated with speech segments of those signals. Another computer device, such as a cloud server, can host a speech processing engine, to which input signals are ultimately provided. Other suitable configurations will be apparent and are within the scope of the disclosure.

Although the disclosed examples have been fully described with reference to the accompanying drawings, it is to be noted that various changes and modifications will become apparent to those skilled in the art. For example, elements of one or more implementations may be combined, deleted, modified, or supplemented to form further implementations. Such changes and modifications are to be understood as being included within the scope of the disclosed examples as defined by the appended claims.