Patent Publication Number: US-2023135768-A1

Title: Determining input for speech processing engine

Description:
CROSS-REFERENCE TO RELATED APPLICATION 
     This application is a Continuation of Non-Provisional application Ser. No. 16/805,337, filed Feb. 28, 2020, which claims priority to U.S. Provisional Application No. 62/812,959, filed on Mar. 1, 2019, the contents of which are incorporated by reference herein in their entirety. 
    
    
     FIELD 
     This disclosure relates in general to systems and methods for processing speech signals, and in particular to systems and methods for processing a speech signal for presentation to a speech processing engine. 
     BACKGROUND 
     Systems for speech recognition are tasked with receiving audio input representing human speech, typically via one or more microphones, and processing the audio input to determine words, logical structures, or other outputs corresponding to that audio input. For example, automatic speech recognition (ASR) systems may generate a text output based on the human speech corresponding to an audio input signal; and natural language processing (NLP) tools may generate logical structures, or computer data, corresponding to the meaning of that human speech. While such systems may contain any number of components, at the heart of such systems is a speech processing engine, which is a component that accepts an audio signal as input, performs some recognition logic on the input, and outputs some text corresponding to that input. (While reference is made herein to speech processing engines, other forms of speech processing besides speech recognition should also be considered within the scope of the disclosure.) 
     Historically, audio input, such as detected via a microphone, was provided to speech processing engines in a structured, predictable manner. For example, a user might speak directly into a microphone of a desktop computer in response to a first prompt (e.g., “Begin Speaking Now”); immediately after pressing a first button input (e.g., a “start” or “record” button, or a microphone icon in a software interface); or after a significant period of silence. Similarly, a user might stop providing microphone input in response to a second prompt (e.g., “Stop Speaking”); immediately before pressing a second button input (e.g., a “stop” or “pause” button); or by remaining silent for a period of time. Such structured input sequences left little doubt as to when the user was providing input to a speech processing engine (e.g., between a first prompt and a second prompt, or between pressing a start button and pressing a stop button). Moreover, because such systems typically required deliberate action on the part of the user, it could generally be assumed that a user&#39;s speech input was directed to the speech processing engine, and not to some other listener (e.g., a person in an adjacent room). Accordingly, many speech processing engines of the time may not have had any particular need to identify, from microphone input, which portions of the input were directed to the speech processing engine and were intended to provide speech recognition input, and conversely, which portions were not. 
     The ways in which users provide speech recognition input has changed as speech processing engines have become more pervasive and more fully integrated into users&#39; everyday lives. For example, some automated voice assistants are now housed in or otherwise integrated with household appliances, automotive dashboards, smart phones, wearable devices, “living room” devices (e.g., devices with integrated “smart” voice assistants), and other environments far removed from the conventional desktop computer. In many cases, speech processing engines are made more broadly usable by this level of integration into everyday life. However, these systems would be made cumbersome by system prompts, button inputs, and other conventional mechanisms for demarcating microphone input to the speech processing engine. Instead, some such systems place one or more microphones in an “always on” state, in which the microphones listen for a “wake-up word” (e.g., the “name” of the device or any other predetermined word or phrase) that denotes the beginning of a speech recognition input sequence. Upon detecting the wake-up word, the speech processing engine can process the following sequence of microphone input as input to the speech processing engine. 
     While the wake-up word system replaces the need for discrete prompts or button inputs for speech processing engines, it carries a risk of false positives, such as where the wake-up word is spoken by a user without the intention of activating the speech processing engine, or is inadvertently “spoken” by a television or a music speaker in the vicinity of the speech processing engine. It can be desirable to replace the wake-up word system with a more intelligent way of determining, based on speech input, whether a user intends for that speech to provide input to a speech processing engine—that is, whether the user&#39;s speech is “input speech,” rather than “non-input speech.” This would allow users to interact more naturally with the speech processing engine (e.g., without having to invoke a dedicated wake-up word), and would encourage the use of such systems in everyday environments and situations. 
     A related problem of speech processing engines that occupy everyday spaces is that these engines may detect large amounts of ambient noise, speech not directed at the engine, or other audio signals that are not intended as input. For example, a speech processing engine in the living room will detect not only a user&#39;s deliberate speech processing engine input (e.g., “What&#39;s the weather forecast for Tuesday?”), but also sounds and speech from pets, devices (e.g., television speakers), or other people; ambient noises; or portions of the user&#39;s speech that are directed to other people in the vicinity. Processing these non-input portions of an audio signal wastes computational resources, and can compromise the accuracy of a speech processing engine which may already be limited in some uncontrolled environments (e.g., outdoors). It is desirable for a speech processing engine to identify, from microphone input, which portions of the microphone input represent input intended for the speech processing engine (input speech); and to disregard the portions (non-input speech) that do not. 
     It is further desirable to use sensor-equipped systems, including those that incorporate a wearable head-mounted unit, to improve the ability of speech processing engines to identify and disregard audio input that is not intended for the speech processing engine. Sensor data (e.g., data from individual sensors or data from multiple sensors fused together by an inertial measurement unit) can be used by speech processing engines to help identify and process only those portions of an audio input signal that are input speech, allowing the outputs of those systems to generate more accurate and more reliable results. Sensors of wearable devices, in particular, may be especially useful; for example, such sensors can indicate the position and orientation of a user; the user&#39;s eye movements and eye gaze targets; movements of the user&#39;s hands; and biometric data (e.g., vital signs such as heart rate and breathing rate). In many cases, these indications can provide a speech recognition system with the same sort of non-verbal cues (such as a user&#39;s movements and body language) that humans use to intuitively understand to whom another person is speaking. In addition, wearable systems are well suited for mobile, outdoor applications—precisely the type of applications in which many conventional speech processing engines may perform especially poorly. 
     BRIEF SUMMARY 
     Examples of the disclosure describe systems and methods for presenting a signal to a speech processing engine. According to an example method, an audio signal is received via one or more microphones. A portion of the audio signal is identified, and a probability is determined that the portion comprises speech directed by a user of the speech processing engine as input to the speech processing engine. In accordance with a determination that the probability exceeds a threshold, the portion of the audio signal is presented as input to the speech processing engine. In accordance with a determination that the probability does not exceed the threshold, the portion of the audio signal is not presented as input to the speech processing engine. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG.  1    illustrates an example wearable system according to some embodiments of the disclosure. 
         FIG.  2    illustrates an example handheld controller that can be used in conjunction with an example wearable system according to some embodiments of the disclosure. 
         FIG.  3    illustrates an example auxiliary unit that can be used in conjunction with an example wearable system according to some embodiments of the disclosure. 
         FIG.  4    illustrates an example functional block diagram for an example wearable system according to some embodiments of the disclosure. 
         FIG.  5    illustrates a flow chart of an example system for processing acoustic speech signals according to some embodiments of the disclosure. 
         FIGS.  6 A- 6 D  illustrate examples of processing acoustic speech signals according to some embodiments of the disclosure. 
         FIGS.  7 A- 7 C  illustrate flow charts of example systems for processing acoustic speech signals according to some embodiments of the disclosure. 
         FIG.  8    illustrates a flow chart showing portions of an example system for processing acoustic speech signals according to some embodiments of the disclosure. 
         FIG.  9    illustrates a flow chart showing portions of an example system for processing acoustic speech signals according to some embodiments of the disclosure. 
         FIG.  10    illustrates a user interacting with one or more recipients according to some embodiments of the disclosure. 
         FIG.  11    illustrates an example process for capturing audio and non-audio classifier training data, according to some embodiments of the disclosure. 
     
    
    
     DETAILED DESCRIPTION 
     In the following description of examples, reference is made to the accompanying drawings which form a part hereof, and in which it is shown by way of illustration specific examples that can be practiced. It is to be understood that other examples can be used and structural changes can be made without departing from the scope of the disclosed examples. 
     Example Wearable System 
       FIG.  1    illustrates an example wearable head device  100  configured to be worn on the head of a user. Wearable head device  100  may be part of a broader wearable system that comprises one or more components, such as a head device (e.g., wearable head device  100 ), a handheld controller (e.g., handheld controller  200  described below), and/or an auxiliary unit (e.g., auxiliary unit  300  described below). In some examples, wearable head device  100  can be used for virtual reality, augmented reality, or mixed reality systems or applications. Wearable head device  100  can comprise one or more displays, such as displays  110 A and  110 B (which may comprise left and right transmissive displays, and associated components for coupling light from the displays to the user&#39;s eyes, such as orthogonal pupil expansion (OPE) grating sets  112 A/ 112 B and exit pupil expansion (EPE) grating sets  114 A/ 114 B); left and right acoustic structures, such as speakers  120 A and  120 B (which may be mounted on temple arms  122 A and  122 B, and positioned adjacent to the user&#39;s left and right ears, respectively); one or more sensors such as infrared sensors, accelerometers, GPS units, inertial measurement units (IMUs, e.g. IMU  126 ), acoustic sensors (e.g., microphones  150 ); orthogonal coil electromagnetic receivers (e.g., receiver  127  shown mounted to the left temple arm  122 A); left and right cameras (e.g., depth (time-of-flight) cameras  130 A and  130 B) oriented away from the user; and left and right eye cameras oriented toward the user (e.g., for detecting the user&#39;s eye movements)(e.g., eye cameras  128 A and  128 B). However, wearable head device  100  can incorporate any suitable display technology, and any suitable number, type, or combination of sensors or other components without departing from the scope of the invention. In some examples, wearable head device  100  may incorporate one or more microphones  150  configured to detect audio signals generated by the user&#39;s voice; such microphones may be positioned adjacent to the user&#39;s mouth. In some examples, wearable head device  100  may incorporate networking features (e.g., Wi-Fi capability) to communicate with other devices and systems, including other wearable systems. Wearable head device  100  may further include components such as a battery, a processor, a memory, a storage unit, or various input devices (e.g., buttons, touchpads); or may be coupled to a handheld controller (e.g., handheld controller  200 ) or an auxiliary unit (e.g., auxiliary unit  300 ) that comprises one or more such components. In some examples, sensors may be configured to output a set of coordinates of the head-mounted unit relative to the user&#39;s environment, and may provide input to a processor performing a Simultaneous Localization and Mapping (SLAM) procedure and/or a visual odometry algorithm. In some examples, wearable head device  100  may be coupled to a handheld controller  200 , and/or an auxiliary unit  300 , as described further below. 
       FIG.  2    illustrates an example mobile handheld controller component  200  of an example wearable system. In some examples, handheld controller  200  may be in wired or wireless communication with wearable head device  100  and/or auxiliary unit  300  described below. In some examples, handheld controller  200  includes a handle portion  220  to be held by a user, and one or more buttons  240  disposed along a top surface  210 . In some examples, handheld controller  200  may be configured for use as an optical tracking target; for example, a sensor (e.g., a camera or other optical sensor) of wearable head device  100  can be configured to detect a position and/or orientation of handheld controller  200 —which may, by extension, indicate a position and/or orientation of the hand of a user holding handheld controller  200 . In some examples, handheld controller  200  may include a processor, a memory, a storage unit, a display, or one or more input devices, such as described above. In some examples, handheld controller  200  includes one or more sensors (e.g., any of the sensors or tracking components described above with respect to wearable head device  100 ). In some examples, sensors can detect a position or orientation of handheld controller  200  relative to wearable head device  100  or to another component of a wearable system. In some examples, sensors may be positioned in handle portion  220  of handheld controller  200 , and/or may be mechanically coupled to the handheld controller. Handheld controller  200  can be configured to provide one or more output signals, corresponding, for example, to a pressed state of the buttons  240 ; or a position, orientation, and/or motion of the handheld controller  200  (e.g., via an IMU). Such output signals may be used as input to a processor of wearable head device  100 , to auxiliary unit  300 , or to another component of a wearable system. In some examples, handheld controller  200  can include one or more microphones to detect sounds (e.g., a user&#39;s speech, environmental sounds), and in some cases provide a signal corresponding to the detected sound to a processor (e.g., a processor of wearable head device  100 ). 
       FIG.  3    illustrates an example auxiliary unit  300  of an example wearable system. In some examples, auxiliary unit  300  may be in wired or wireless communication with wearable head device  100  and/or handheld controller  200 . The auxiliary unit  300  can include a battery to provide energy to operate one or more components of a wearable system, such as wearable head device  100  and/or handheld controller  200  (including displays, sensors, acoustic structures, processors, microphones, and/or other components of wearable head device  100  or handheld controller  200 ). In some examples, auxiliary unit  300  may include a processor, a memory, a storage unit, a display, one or more input devices, and/or one or more sensors, such as described above. In some examples, auxiliary unit  300  includes a clip  310  for attaching the auxiliary unit to a user (e.g., a belt worn by the user). An advantage of using auxiliary unit  300  to house one or more components of a wearable system is that doing so may allow large or heavy components to be carried on a user&#39;s waist, chest, or back—which are relatively well suited to support large and heavy objects—rather than mounted to the user&#39;s head (e.g., if housed in wearable head device  100 ) or carried by the user&#39;s hand (e.g., if housed in handheld controller  200 ). This may be particularly advantageous for relatively heavy or bulky components, such as batteries. 
       FIG.  4    shows an example functional block diagram that may correspond to an example wearable system  400 , such as may include example wearable head device  100 , handheld controller  200 , and auxiliary unit  300  described above. In some examples, the wearable system  400  could be used for virtual reality, augmented reality, or mixed reality applications. As shown in  FIG.  4   , wearable system  400  can include example handheld controller  400 B, referred to here as a “totem” (and which may correspond to handheld controller  200  described above); the handheld controller  400 B can include a totem-to-headgear six degree of freedom (6DOF) totem subsystem  404 A. Wearable system  400  can also include example headgear device  400 A (which may correspond to wearable head device  100  described above); the headgear device  400 A includes a totem-to-headgear 6DOF headgear subsystem  404 B. In the example, the 6DOF totem subsystem  404 A and the 6DOF headgear subsystem  404 B cooperate to determine six coordinates (e.g., offsets in three translation directions and rotation along three axes) of the handheld controller  400 B relative to the headgear device  400 A. The six degrees of freedom may be expressed relative to a coordinate system of the headgear device  400 A. The three translation offsets may be expressed as X, Y, and Z offsets in such a coordinate system, as a translation matrix, or as some other representation. The rotation degrees of freedom may be expressed as sequence of yaw, pitch and roll rotations; as vectors; as a rotation matrix; as a quaternion; or as some other representation. In some examples, one or more depth cameras  444  (and/or one or more non-depth cameras) included in the headgear device  400 A; and/or one or more optical targets (e.g., buttons  240  of handheld controller  200  as described above, or dedicated optical targets included in the handheld controller) can be used for 6DOF tracking. In some examples, the handheld controller  400 B can include a camera, as described above; and the headgear device  400 A can include an optical target for optical tracking in conjunction with the camera. In some examples, the headgear device  400 A and the handheld controller  400 B each include a set of three orthogonally oriented solenoids which are used to wirelessly send and receive three distinguishable signals. By measuring the relative magnitude of the three distinguishable signals received in each of the coils used for receiving, the 6DOF of the handheld controller  400 B relative to the headgear device  400 A may be determined. In some examples, 6DOF totem subsystem  404 A can include an Inertial Measurement Unit (IMU) that is useful to provide improved accuracy and/or more timely information on rapid movements of the handheld controller  400 B. 
     In some examples involving augmented reality or mixed reality applications, it may be desirable to transform coordinates from a local coordinate space (e.g., a coordinate space fixed relative to headgear device  400 A) to an inertial coordinate space, or to an environmental coordinate space. For instance, such transformations may be necessary for a display of headgear device  400 A to present a virtual object at an expected position and orientation relative to the real environment (e.g., a virtual person sitting in a real chair, facing forward, regardless of the position and orientation of headgear device  400 A), rather than at a fixed position and orientation on the display (e.g., at the same position in the display of headgear device  400 A). This can maintain an illusion that the virtual object exists in the real environment (and does not, for example, appear positioned unnaturally in the real environment as the headgear device  400 A shifts and rotates). In some examples, a compensatory transformation between coordinate spaces can be determined by processing imagery from the depth cameras  444  (e.g., using a Simultaneous Localization and Mapping (SLAM) and/or visual odometry procedure) in order to determine the transformation of the headgear device  400 A relative to an inertial or environmental coordinate system. In the example shown in  FIG.  4   , the depth cameras  444  can be coupled to a SLAM/visual odometry block  406  and can provide imagery to block  406 . The SLAM/visual odometry block  406  implementation can include a processor configured to process this imagery and determine a position and orientation of the user&#39;s head, which can then be used to identify a transformation between a head coordinate space and a real coordinate space. Similarly, in some examples, an additional source of information on the user&#39;s head pose and location is obtained from an IMU  409  of headgear device  400 A. Information from the IMU  409  can be integrated with information from the SLAM/visual odometry block  406  to provide improved accuracy and/or more timely information on rapid adjustments of the user&#39;s head pose and position. 
     In some examples, the depth cameras  444  can supply 3D imagery to a hand gesture tracker  411 , which may be implemented in a processor of headgear device  400 A. The hand gesture tracker  411  can identify a user&#39;s hand gestures, for example by matching 3D imagery received from the depth cameras  444  to stored patterns representing hand gestures. Other suitable techniques of identifying a user&#39;s hand gestures will be apparent. 
     In some examples, one or more processors  416  may be configured to receive data from headgear subsystem  404 B, the IMU  409 , the SLAM/visual odometry block  406 , depth cameras  444 , microphones  450 ; and/or the hand gesture tracker  411 . The processor  416  can also send and receive control signals from the 6DOF totem system  404 A. The processor  416  may be coupled to the 6DOF totem system  404 A wirelessly, such as in examples where the handheld controller  400 B is untethered. Processor  416  may further communicate with additional components, such as an audio-visual content memory  418 , a Graphical Processing Unit (GPU)  420 , and/or a Digital Signal Processor (DSP) audio spatializer  422 . The DSP audio spatializer  422  may be coupled to a Head Related Transfer Function (HRTF) memory  425 . The GPU  420  can include a left channel output coupled to the left source of imagewise modulated light  424  and a right channel output coupled to the right source of imagewise modulated light  426 . GPU  420  can output stereoscopic image data to the sources of imagewise modulated light  424 ,  426 . The DSP audio spatializer  422  can output audio to a left speaker  412  and/or a right speaker  414 . The DSP audio spatializer  422  can receive input from processor  419  indicating a direction vector from a user to a virtual sound source (which may be moved by the user, e.g., via the handheld controller  400 B). Based on the direction vector, the DSP audio spatializer  422  can determine a corresponding HRTF (e.g., by accessing a HRTF, or by interpolating multiple HRTFs). The DSP audio spatializer  422  can then apply the determined HRTF to an audio signal, such as an audio signal corresponding to a virtual sound generated by a virtual object. This can enhance the believability and realism of the virtual sound, by incorporating the relative position and orientation of the user relative to the virtual sound in the mixed reality environment—that is, by presenting a virtual sound that matches a user&#39;s expectations of what that virtual sound would sound like if it were a real sound in a real environment. 
     In some examples, such as shown in  FIG.  4   , one or more of processor  416 , GPU  420 , DSP audio spatializer  422 , HRTF memory  425 , and audio/visual content memory  418  may be included in an auxiliary unit  400 C (which may correspond to auxiliary unit  300  described above). The auxiliary unit  400 C may include a battery  427  to power its components and/or to supply power to headgear device  400 A and/or handheld controller  400 B. Including such components in an auxiliary unit, which can be mounted to a user&#39;s waist, can limit the size and weight of headgear device  400 A, which can in turn reduce fatigue of a user&#39;s head and neck. 
     While  FIG.  4    presents elements corresponding to various components of an example wearable system  400 , various other suitable arrangements of these components will become apparent to those skilled in the art. For example, elements presented in  FIG.  4    as being associated with auxiliary unit  400 C could instead be associated with headgear device  400 A or handheld controller  400 B. Furthermore, some wearable systems may forgo entirely a handheld controller  400 B or auxiliary unit  400 C. Such changes and modifications are to be understood as being included within the scope of the disclosed examples. 
     Speech Processing Engines 
     Speech recognition systems in general include a speech processing engine that can accept an input audio signal corresponding to human speech (a source signal); process and analyze the input audio signal; and produce, as a result of the analysis, an output corresponding to the human speech. In the case of automatic speech recognition (ASR) systems, for example, the output of a speech processing engine may be a text transcription of the human speech. In the case of natural language processing systems, the output may be one or more commands or instructions indicated by the human speech; or some representation (e.g., a logical expression or a data structure) of the semantic meaning of the human speech. Other types of speech processing systems (e.g., automatic translation systems), including those that do not necessarily “recognize” speech, are contemplated and are within the scope of the disclosure. 
     Speech recognition systems are found in a diverse array of products and applications: conventional telephone systems; automated voice messaging systems; voice assistants (including standalone and smartphone-based voice assistants); vehicles and aircraft; desktop and document processing software; data entry; home appliances; medical devices; language translation software; closed captioning systems; and others. An advantage of speech recognition systems is that they may allow users to provide input to a computer system using natural spoken language, such as presented to one or more microphones, instead of conventional computer input devices such as keyboards or touch panels; accordingly, speech recognition systems may be particularly useful in environments where conventional input devices (e.g., keyboards) may be unavailable or impractical. Further, by permitting users to provide intuitive voice-based input, speech processing engines can heighten feelings of immersion. As such, speech recognition can be a natural fit for wearable systems, and in particular, for virtual reality, augmented reality, and/or mixed reality applications of wearable systems, in which user immersion is a primary goal; and in which it may be desirable to limit the use of conventional computer input devices, whose presence may detract from feelings of immersion. 
     Typically, the output of any speech processing engine does not correspond to the source human speech with perfect certainty; because of the many variables that can affect the audio signals provided as input, even sophisticated speech processing engines do not consistently produce perfect text output for all speakers. For example, the reliability of speech processing engines may be highly dependent on the quality of the input audio signal. Where input audio signals are recorded in ideal conditions—for example, in acoustically controlled environments, with a single human speaker enunciating clearly and directly into a microphone from a close distance—the source speech can be more readily determined from the audio signal. In real-world applications, however, input audio signals may deviate from ideal conditions, such that determining the source human speech becomes more difficult. For example, input audio signals may include significant ambient noise, or speech from multiple speakers, in addition to the user; for instance, speech from other people, pets, or electronic devices (e.g., televisions) can be mixed in with the user&#39;s speech in the input signal. In addition, even the user&#39;s speech may include not only speech intended for the speech processing engine (input speech), but also speech directed at other listeners (such as other people, pets, or other devices). By isolating the input speech from the broader input audio signal, the fidelity of the input processed by the speech processing engine can be improved; and the accuracy of the speech processing engine&#39;s output can be improved accordingly. 
     Identifying and Segmenting Input Speech 
     The present disclosure is directed to systems and methods for improving the accuracy of a speech processing system by removing, from raw speech signals, portions of those signals that are not directed by the user to the speech processing system. As described herein, such non-input portions can be identified (e.g., classified) based on audio characteristics of the speech signals themselves (e.g., sudden changes in the speech&#39;s vocabulary, semantics, or grammar); and/or by using input from sensors associated with wearable devices (e.g., head-mounted devices such as described above with respect to  FIG.  1   ). Such non-input portions may be especially prominent in mobile applications of speech processing, in household usage of speech processing systems, or in applications of speech processing in uncontrolled environments, such as outdoor environments where other voices or ambient noise may be present. Wearable systems are frequently intended for use in such applications, and may therefore be especially susceptible to undirected speech. For example, where some wearable systems are intended for use in uncontrolled environments, a high potential can exist for environmental noise (or speech of other humans) to be recorded along with the target human speech. Sensors of wearable systems (such as described above with respect to  FIGS.  1 - 4   ) are well suited to solving this problem, as described herein. However, in some examples, as described herein, directivity can be determined based solely on a speech signal, even without the benefit of sensor input. 
       FIG.  5    illustrates an example system  500 , according to some embodiments, in which a speech processing engine  550  produces a text output  552  (such as described above) based on a raw speech signal  510  provided as input. In some examples, raw speech signal  510  can be can be provided as detected by one or more microphones, but in some examples can be provided from a data file (e.g., an audio waveform file), from an audio stream (e.g., provided via a network), or from any other suitable source. In system  500 , improved accuracy of text output  552  can be achieved by presenting, as input to speech processing engine  550 , a “directed” speech signal  540  that includes only those portions of raw input speech signal  510  that are determined to constitute input speech directed to speech processing engine  550  (as opposed to, for example, extraneous speech such as described above). Directed speech signal  540  can be determined at stage  530  from the raw input speech signal  510  and/or from sensor data  520 , which can correspond to data from sensors such as described above with respect to example wearable head device  100  in  FIG.  1   . 
     At stage  530 , raw speech signal  510  can be divided into individual speech segments; then, for each segment, a probability can be determined that the segment corresponds to input speech that was intended as input for the speech processing engine  550 . In some cases, probabilistic modelling or machine learning techniques can indicate this probability for each segment of the raw speech signal  510 . Directed speech signal  540  can then be generated by filtering, from raw speech signal  510 , the segments of raw speech signal  510  that do not meet a threshold probability of corresponding to input speech (rather than to non-input speech). (As used herein, input speech can include input audio that is provided by a particular user and that is also directed by the user toward a speech recognition system.) 
       FIGS.  6 A- 6 D  illustrate examples of a raw speech signal, a segmented version of the speech signal, a probabilistic model of the raw speech signal (though in some embodiments machine learning techniques may be used), and a directed speech signal generated from the raw speech signal, respectively.  FIG.  6 A  shows an example audio waveform  600  (which may correspond to raw speech signal  510 ), expressed as an amplitude (e.g., of voltage) V(t) as a function of time, such as might be detected by one or more microphones and/or represented in a waveform audio file. In the example, the waveform  600  corresponds to a user speaking the example sequence, “What&#39;s the weather . . . not now, Charlie . . . tomorrow.” In the example, the speech sequence includes at least one portion (“What&#39;s the weather”) intended as a query to the speech processing engine (e.g., speech processing engine  550 ); at least one portion (“not now, Charlie”) intended not as input to speech processing engine, but to another listener (presumably, Charlie); and at least one portion (“tomorrow”) that could reasonably belong, semantically, either to the speech recognition input portion (“What&#39;s the weather . . . tomorrow”) or to the non-input portion (“not now, Charlie . . . tomorrow”). In addition, raw speech signal  510  includes non-verbal noise in between spoken word portions. If raw speech signal  510  were applied directly as input to speech processing engine  550 , the system might produce unexpected results, as the presence of non-input speech (“not now, Charlie,” and possibly “tomorrow”) could interfere with the system&#39;s ability to meaningfully respond to the input speech (“What&#39;s the weather,” possibly with the qualifier “tomorrow”). Higher quality results can be achieved by, in advance of providing input to speech processing engine  550 , filtering raw speech signal  600  to generate a directed audio signal that includes speech directed at speech processing engine  550  (e.g., “What&#39;s the weather . . . tomorrow”) to the exclusion of non-input speech not directed at speech processing engine  550  (e.g., “not now, Charlie”). (As used herein, non-input speech can include input audio that is not provided by a particular user and/or that is not directed toward a speech processing system.) 
     A segmentation process can divide a raw speech signal into individual segments of audio that can be individually evaluated as corresponding to input speech or non-input speech.  FIG.  6 B  illustrates an example segmentation of raw speech signal  600  into segments of audio. Segments can include phonemes, words, phrases, sentences, utterances, or combinations of any of the above. For each segment, example system  500  can determine whether the segment corresponds to input speech or non-input speech, with the results of the determination used to determine whether the segment should be included or excluded from directed speech signal  540 . As shown in  FIG.  6 B , a segment of signal  600  can be expressed as a region of signal  600  that lies between two points in time (e.g., along the invariant t axis). For example, in the figure, a first segment  601  (e.g., corresponding to “What&#39;s the weather”) lies between points t 0  and t 1 ; a second segment  602  (e.g., corresponding to non-speech, such as background noise) lies between points t 1  and t 2 ; a third segment  603  (e.g., corresponding to “not now”) lies between points t 2  and t 3 ; a fourth segment  604  (e.g., corresponding to “Charlie”) lies between points t 3  and t 4 ; a fifth segment  605  (e.g., corresponding to non-speech, such as background noise) lies between points t 4  and t 5 ; a sixth segment  606  (e.g., corresponding to “tomorrow”) lies between points t 5  and t 6 ; and a seventh segment  607  (e.g., corresponding to non-speech, such as background noise) lies between points t 6  and t 7 . 
     The boundaries of such segments can be determined according to one or more suitable techniques. For example, various techniques known in the art can be used to determine boundaries of spoken words or phrases. According to some such techniques, boundaries between segments can be determined based on, for example, periods of relative silence (indicating gaps between “chunks” of speech); changes in pitch or intonation (which may indicate the start or end of a word, phrase, or idea); changes in the cadence of speech (which can indicate the start or end or a word, phrase, or idea, or a transition from one word, phrase, or idea to another); breathing patterns (which can indicate the speaker is about to begin a new word, phrase, or idea); and so on. In some examples, statistical analysis of a speech signal can be useful to identify segment boundaries; for example, portions of the speech signal that represent statistical outliers in the signal (e.g., portions of the speech signal comprising frequency components not commonly found elsewhere in the signal) can signify the start or end of a word, phrase, or idea. Various machine learning techniques can also be used to identify segment boundaries. 
     In some examples, sensor data  520  can be used to segment a speech signal (e.g., the raw speech signal  510 ), by indicating potential separation points where a user may be likely to change the target of their speech (e.g., transitioning from speaking to a speech processing engine to speaking to another person in the room). For instance, sensor data may indicate when a user turns their head, changes the focus of their eye gaze, or moves to a different location in the room. Sudden changes in such sensor data can be used to indicate boundaries between speech segments. 
     The lengths (e.g., average time, or number of syllables) of speech segments may vary. In some examples, segments may generally be on the order of several words, such as may make up a spoken phrase. In some examples, segments may be longer (e.g., constituting one or more full sentences or utterances), or shorter (e.g., constituting individual words, or even individual syllables). As described herein, speech can be included or excluded from directed speech signal  540  on a per-segment basis, such that for each segment, either the entire segment is included, or the entire segment is excluded. Utilizing longer segments can increase the risk that a single segment will include both input speech and non-input speech, which can cause undesirable results: excluding such a segment from directed speech signal  540  would result in failing to present the user&#39;s input speech to speech processing engine  550 , while including it would present non-input speech to speech processing engine  550 —an opposite goal of generating directed speech signal  540 . While using shorter segments can reduce this problem, it presents a possible tradeoff in the computational overhead (and accompanying latency) required to process additional segments for a single speech signal. A desirable balance of segment size may be to group, to the extent possible, single related words or thoughts in a single segment, such that the entire segment is, or is not, directed to speech processing engine  550 . For example, in example signal  600 , “What&#39;s the weather” and “not now” each constitute a single chunk of speech that rises or falls together, and may thus be beneficial to group as a single segment. However, segments may be arbitrarily large or arbitrarily small (including segments as small as a single digital audio sample), and the present disclosure is not limited to any particular segmentation size. 
     In some examples, segmentation may be performed on a prerecorded speech signal, where the entire speech signal is captured before it is segmented. Segmentation may be comparatively more accurate and/or efficient in such examples, as knowledge of the entire speech signal can be used to generate more meaningful speech segments; that is, which portions of the speech signal should be segmented together can be easier to determine when the entire signal is known. However, in some examples, “live” speech may be segmented as it is being detected. Techniques for segmenting prerecorded speech signals may also be used to segment live speech signals (for example, by applying such techniques to buffered chunks of live speech). In some cases, segmentation decisions on live speech may need to be periodically revisited as new speech clarifies the intention of previous speech. Additionally, portions of speech can be flagged for manual review, where they can later be evaluated and corrected manually. 
       FIG.  6 C  demonstrates an example probability model  610  corresponding to speech signal  600 . In the example, probability model  610  can express, as a function of time t, a probability p(t) that the segment of the corresponding audio signal  600  at time t is user speech directed at speech processing engine  550 . (Alternatively, in some examples, p(t) can describe the probability that the segment is not user speech directed at the speech processing engine.) For instance, in the example, at a time t k1  that falls between t 0  and t 1 , p(t k1 ) is equal to 0.9, indicating that the portion of speech signal  600  at time t k1  (V(t k1 ), e.g., “weather”) has a 90% probability of being user speech directed to speech processing engine  550 . Similarly, at a time t k2  that falls between t 3  and t 4 , p(t k2 ) is equal to 0.1, indicating that the portion of speech signal  600  at time t k2  (V(t k2 ), e.g., “Charlie”) has a 10% probability of being user speech directed to speech processing engine  550 . 
     As shown in the figure, probability p(t) can be determined on a per-segment basis, such that for a segment that begins at time t 0  and ends at time t 1 , p(t) remains constant between p(t 0 ) and p(t 1 ) (that is, the entire segment will have the same probability value). Accordingly, in probability model  610 , segment  601  (“What&#39;s the weather”) has a corresponding probability value  611  of 0.9; segment  603  (“not now”) has a corresponding probability value  613  of 0.3; segment  604  (“Charlie”) has a corresponding probability value  614  of 0.1; and segment  606  (“tomorrow”) has a corresponding probability value  616  of 0.6. In the figure, the remaining segments (i.e., segments  602 ,  605 , and  607 , which may correspond to background noise or other non-speech audio) have corresponding probability values (i.e.,  612 ,  615 , and  617 , respectively) of zero. 
     Classifying Input Speech 
     Determining a probability value for a speech segment can be referred to as “classifying” the speech segment, and a module or process for performing this determination (e.g.,  562 ,  568 ,  574 ) can be referred to as a “classifier.”  FIGS.  7 A,  7 B, and  7 C  illustrate example classifiers of example system  500  for determining a probability value for a segment of a speech signal (e.g., segments  610  of speech signal  600  described above). This determination can be performed using the speech signal itself (e.g., as shown in  FIG.  7 A ); using sensor data associated with the user (e.g., as shown in  FIG.  7 B ); or using some combination of the speech signal and the sensor data (e.g., as shown in  FIG.  7 C ). 
     In the example shown in  FIG.  7 A , speech segment  516 , statistical data  512  for the speech signal, and/or a speech data repository  527  are used by classifier  562  to determine a probability value  566  with which the speech segment  516  corresponds to input speech (e.g., user speech directed at a speech recognition system). At stage  563 , speech segment  516  can be parameterized/characterized according to one or more parameters, such as by using statistical data  512  of the speech signal. This can facilitate classifying the speech segment based on speech data repository  527 . Speech data repository  527  may be stored in a database. A Fourier transform of a time-based speech segment  516  can be performed in order to provide a spectral representation of the speech segment (e.g., a function of frequency indicating the relative prevalence of various frequency parameters in the speech segment  516 ). In some cases, speech segment  516  can be compared against statistical data  512  to determine a degree to which speech segment  516  deviates from the larger speech signal of which it is a part. For instance, this can indicate levels of (or changes in) volume or component frequencies of the speech segment that can be used at stage  564  to characterize the speech segment. In some examples, aspects of the speaker—for example, the speaker&#39;s age, sex, and/or native language—can be used as parameters to characterize the speech segment  516 . Other ways in which speech segment  516  can be parameterized, with such parameters used to characterize the speech segment at stage  564 , will be apparent to those skilled in the art. As examples, speech segment  516  can be preprocessed with pre-emphasis, spectral analysis, loudness analysis, DCT/MFCC/LPC/MQ analysis, Mel filter bank filtering, noise reduction, band-pass filtering of the signal to the most useful speech range (e.g., 85-8000 Hz), and dynamic range compression. The remaining signal can then be parameterized into a set of time-invariant features (e.g., speaker identification/biometrics, gender identification, mean fundamental frequency, mean loudness) and time-varying feature vectors (e.g., formant center frequencies and bandwidths, fundamental frequency, DCT/MFCC/LPC/MQ coefficients, phoneme identification, consonant identification, pitch contour, loudness contour). 
     At stage  564  of the example, a probability value  566  is determined that speech segment  516  corresponds to input speech. Probability value  566  can be determined using speech data repository  527 . For example, a database including speech data repository  527  can identify, for elements of speech in the database, whether those elements correspond to input speech. Various types of data may be represented in speech data repository  527 . In some examples, speech data repository  527  can include a set of audio waveforms corresponding to speech segments; and can indicate, for each waveform, whether the corresponding speech segment belongs to input speech. In some examples, instead of or in addition to audio waveforms, speech data repository  527  can include audio parameters that correspond to the speech segments. Speech segment  516  can be compared with the speech segments of speech data repository  527 —for example, by comparing an audio waveform of speech segment  516  with audio waveforms of speech data repository  527 , or by comparing parameters of speech segment  516  (such as may be characterized at stage  563 ) with analogous parameters of speech data repository  527 . Based on such comparisons, probability  566  can be determined for speech segment  516 . (Methods for creating the data in speech data repository  527  are described below.) 
     Techniques for determining probability  566  will be familiar to those skilled in the art. For instance, in some examples, nearest neighbor interpolation can be used at stage  564  to compare speech segment  516  to similar speech segments in an N-dimensional space (in which the N dimensions can comprise, for example, audio parameters and/or audio waveform data described above); and to determine probability value  566  based on the relative distances between speech segment  516  and its neighbors in the N-dimensional space. As another example, support vector machines can be used at stage  564  to determine, based on speech data repository  527 , a basis for classifying a speech segment as either an input speech segment or a non-input speech segment; and for classifying speech segment  516  (e.g., determining a probability value  566  that the speech segment is input speech) according to that basis. Other suitable techniques for analyzing speech segment  516  and/or speech data repository  527 , comparing speech segment  516  to speech data repository  527 , and/or classifying speech segment  516  based on speech data repository  527  in order to determine probability  566  will be apparent; the disclosure is not limited to any particular technique or combination of techniques. 
     In some examples, machine learning techniques can be used, alone or in combination with other techniques described herein, to determine probability value  566 . For example, a neural network could be trained on speech data repository  527 , and applied to speech segment  516  to determine probability value  566  for speech segment  516 . As another example, a genetic algorithm can be used to determine a function, based on speech data repository  527 , for determining the probability  566  for speech segment  516 . Other suitable machine learning techniques, which will be familiar to those skilled in the art, will be apparent; the disclosure is not limited to any particular technique or combination of techniques. 
     In some examples, the probability value  566  for speech segment  516  may be influenced by other speech segments of the same speech signal. For instance, users may be unlikely to provide input in short bursts, surrounded by non-input speech (or vice versa); instead, users may be more likely to provide speech recognition input in largely contiguous sequences. That is, all other factors equal, a speech segment  516  is more likely to be an input speech segment if the segments that come immediately before or after it are also input speech segments; and vice versa. In such examples, probabilistic techniques (e.g., Bayesian networks, hidden Markov models) can be used at stage  564 , alone or in combination with other techniques described herein, to determine probability  566 . Various probabilistic techniques can be suitable for this purpose, and the disclosure is not limited to any particular technique or combination of techniques. 
     In some examples, speech data repository  527  can be generated by recording a set of speech signals of various speech sources, and identifying, for each portion of each speech signal, a speech target of that portion. For instance, a user could be observed interacting with a group of people, with a speech recognition system present in the same room, as the user&#39;s speech (and/or other audio) is recorded. The observer can identify, for each region of the recorded speech, whether that region of speech was directed from the user (and not some other source) as input to the speech recognition system, or to some other target. This information can be apparent to the observer by observing the context in which the user is speaking—commonly, it is easy and intuitive for humans (unlike machines) to determine, based on an observation of a user, whether the user is speaking to a speech recognition system, or to something else. This process can be repeated for multiple users, and in some cases for non-human speakers (e.g., pets, TV speakers, appliances), until a sufficiently large and diverse set of speech data (e.g., audio waveform data, and/or parameters associated with the speech as described above) is generated. From this speech data, individual speech segments can be determined; these speech segments can be associated with the observer&#39;s determination of whether or not the corresponding speech is directed by the user to a speech recognition system. 
     In the example shown in  FIG.  7 A , as described above, probability value  566  is determined based on the user&#39;s own speech as detected by one or more microphones. Accordingly, the predictive value of this system with respect to probability value  566 —that is, the degree to which the example of  FIG.  7 A  enables probability value  566  to be determined more accurately than otherwise—is limited by the degree of correlation between the audio characteristics of a speech signal, and whether the speech signal is input speech. The greater the degree of correlation, the more useful the speech signal will be in determining which portions of the signal are input speech. While there may be at least some such correlation between the speech audio and the intended target, correlation may also exist between the intended target of the speech, and sensor data associated with the speaker, such as sensor data  520 ; accordingly, the overall predictive value of the system can be improved by incorporating sensor data  520 , alone or in addition to raw speech signal  510 , such as described below with respect to  FIGS.  7 B and  7 C . 
       FIG.  7 B  illustrates an example portion of example system  500 , in which sensor data  520  is used by classifier  568  to determine a probability value  572  with which the speech segment  516  is input speech. In some examples, as described above, sensor data  520  can correspond to data from sensors such as described above with respect to example wearable head device  100  in  FIG.  1   . As described above, such a wearable system can include one or more sensors that can provide input about the user and/or the environment of the wearable system. For instance, wearable head device  100  can include a camera (e.g., camera  444  described in  FIG.  4   ) to output visual signals corresponding to the environment; in some examples, the camera can be a forward-facing camera on a head-mounted unit that shows what is currently in front of the user of the wearable system. In some examples, wearable head device  100  can include a LIDAR unit, a radar unit, and/or acoustic sensors, which can output signals corresponding to the physical geometry (e.g., walls, physical objects) of the user&#39;s environment. In some examples, wearable head device  100  can include a GPS unit, which can indicate geographic coordinates corresponding to the wearable system&#39;s current location. In some examples, wearable head device  100  can include an accelerometer, a gyroscope; and/or an inertial measurement unit (IMU) to indicate an orientation of the wearable head device  100 . In some examples, wearable head device  100  can include environmental sensors, such as temperature or pressure sensors. In some examples, wearable head device  100  can include biometric sensors, such as iris cameras; fingerprint sensors; eye tracking sensors (e.g., electrooculography (EOG) sensors) to measure a user&#39;s eye movements or eye gaze; or sensors to measure a user&#39;s vital signs. In examples where wearable head device  100  includes a head-mounted unit, such orientation can correspond to an orientation of the user&#39;s head (and, by extension, the user&#39;s mouth and a direction of the user&#39;s speech). Other suitable sensors can be included and can provide sensor data  520 . Moreover, in some examples, sensors other than those of a wearable system can be utilized as appropriate. For instance, sensors associated with one or more microphones of a speech recognition system (e.g., GPS, IMU) could be used to in conjunction with sensors of a wearable system to determine a relative distance and orientation between the user and the speech recognition system. 
     In the example shown in  FIG.  7 B , stage  569  can parameterize/characterize speech segment  516  according to one or more parameters, such as described above with respect to stage  563 , with respect to aspects of sensor data  520 . This can facilitate classifying the speech segment based on sensor data  520 . For instance, stage  569  can perform a Fourier transform of signals of sensor data  520  (e.g., signals describing a user&#39;s position or orientation (e.g., from GPS, acoustic, radar, or IMU sensors) as a function of time elapsed during the speech segment) in order to determine a spectral representation of those signals. As examples, speech segment  516  can be characterized according to the user&#39;s eye movements (e.g., from EOG sensors), eye gaze targets (e.g., from cameras or EOG sensors), and/or visual targets (e.g., from RGB cameras or LIDAR units). In some examples, sensor data  520  can be compared to a broader range of sensor data (e.g., sensor data captured over a period of several minutes prior to the start of the speech signal) to determine the degree to which sensor data  520  deviates from the broader range of sensor data. Other ways in which sensor data  520  can be parameterized, with such parameters used to characterize the speech segment at stage  564 , will be apparent to those skilled in the art. As described above with respect to speech segment  516 , speech segment  564  can be preprocessed with pre-emphasis, spectral analysis, loudness analysis, DCT/MFCC/LPC/MQ analysis, Mel filter bank filtering, noise reduction, band-pass filtering of the signal to the most useful speech range (e.g., 85-8000 Hz), and dynamic range compression. The remaining signal can then be parameterized into a set of time-invariant features (e.g., speaker identification/biometrics, gender identification, mean fundamental frequency, mean loudness) and time-varying feature vectors (e.g., formant center frequencies and bandwidths, fundamental frequency, DCT/MFCC/LPC/MQ coefficients, phoneme identification, consonant identification, pitch contour, loudness contour). 
     At stage  570  of the example, a probability value  572  is determined that speech segment  516  corresponds to input speech. In some approaches, probability value  572  can be determined using a sensor data repository  528 , which can include a database identifying, for elements of speech in the database, whether those elements correspond to input speech. In some examples, sensor data repository  528  can include data sets representing sensor measurements (e.g., sequences of a user&#39;s head position, orientation, and/or eye gaze over time) corresponding to speech segments; and can indicate, for each data set, whether the corresponding speech segment belongs to input speech. In some examples, instead of or in addition to sensor data sets, sensor data repository  528  can include parameters that correspond to the speech segments. Speech segment  516  can be compared with sensor data repository  528 —for example, by comparing raw sensor data  520  with corresponding signals of sensor data repository  528 , or by comparing parameters of speech segment  516  (such as may be characterized at stage  569 ) with analogous parameters of sensor data repository  528 . Based on such comparisons, probability  572  can be determined for speech segment  516 . 
     Techniques for determining probability  572  will be familiar to those skilled in the art. For example, the techniques described above with respect to determining probability value  566 —e.g., nearest neighbor interpolation, support vector machines, neural networks, genetic algorithms, probabilistic techniques such as Bayesian networks or Markov networks, or any combination of the above—can be applied to sensor data repository  528  and sensor data  520  in an analogous fashion. Other techniques will be apparent, and the disclosure is not limited to any particular technique or combination of techniques. 
     In some examples, sensor data repository  528  need not be accessed directly by classifier  568  in order to classify speech segment  516  at stage  570 . For example, stage  570  can apply one or more rules to determine, based on sensor data  520 , a probability value  572  with which speech segment  516  corresponds to input speech. For instance, it can be determined at stage  570 , based on sensor data  520  (e.g., data from position and orientation sensors), that the user is facing the microphone (or turned to face the microphone shortly before uttering speech segment  516 ); and it can then be determined from this information that speech segment  516  is likely to be input speech. Conversely, it can be determined at stage  570  that the user is facing away from the speech processing engine microphone (or recently turned to face away from the microphone), and that speech segment  516  is unlikely to be input speech. This is because humans generally tend to face the object to which their speech is directed, whether that object is a person or a device. Similarly, it can be determined at stage  570 , based on sensor data  520  (e.g., data from cameras or EOG sensors), that the user is looking at the microphone (or recently shifted their eye gaze toward the microphone), and that speech segment  516  is likely to be input speech. Conversely, it can be determined that the user is not looking at the microphone, and that the speech segment is unlikely to be input speech. As another example, if sensor data  520  (e.g., camera data) indicates that the user is looking directly at another person while uttering speech segment  516 , it can be determined that speech segment  516  is unlikely to be input speech (i.e., that the speech is instead directed at the person the user is looking at). Rules for determining how to classify a probability value  572  based on sensor data can be determined using machine learning techniques familiar to those skilled in the art, such as neural networks or genetic algorithms, using sensor data repository  528  as a training set. 
     In some examples, sensor data repository  528  can be generated similarly to speech data repository  527  as described above. For instance, data of sensor data repository  528  can be generated by recording a set of speech signals of various speech sources, with accompanying sensor data generated at the same time as the speech signals; and identifying, for each portion of each speech signal, a speech target of that portion. For instance, a user could be observed interacting with a group of people, with a speech recognition system present in the same room, as the user&#39;s speech is recorded. The observer can identify, for each region of the recorded speech, whether that region of speech was directed as input from the user to the speech recognition system, or to some other target. From this speech and/or sensor data, individual speech segments can be determined; these speech segments, and their accompanying sensor data, can be associated with the observer&#39;s determination of whether or not the corresponding speech is directed by the user to a speech recognition system. 
     Sensor data  520  can also be used at stage  570  to identify whether or not microphone input belongs to a particular user. For example, the amplitude of a user&#39;s speech, as detected by one or more microphones, can be expected to fall within a predictable range that falls off as a function of the distance between the microphone and the user, and that changes as a function of the relative orientation of the user with respect to the microphone (e.g., falls off as the user faces away from the microphone). (In some cases, this range can be determined experimentally for a particular user.) If sensor data  520  (e.g., GPS data, camera data, acoustic data, radar data) indicates that the user is a particular distance from the microphone, a range of expected amplitudes of that user&#39;s speech for that particular distance can be determined. Microphone input that falls outside of that amplitude range can be rejected as belonging to a source other than the user. Likewise, other speech characteristics (e.g., high frequency content) can be predicted based on the user&#39;s position, orientation, or other sensor data  520 ; and microphone input that is inconsistent with that sensor data can be rejected. Similarly, microphone input that changes significantly (e.g., in volume or frequency characteristics) while the user&#39;s position and orientation remain constant (or vice versa) can be rejected. And conversely, microphone input that is consistent with predicted characteristics of a user&#39;s speech, based on sensor data, can reinforce that the microphone input belongs to that user. Other techniques of identifying a source of microphone input, based on sensor data, will be apparent to those skilled in the art. 
     In  FIG.  7 B , as described above, probability value  572  is determined based on the user&#39;s own speech as detected by one or more microphones. As with the example shown in  FIG.  7 A  and probability value  566 , the predictive value of this system with respect to probability value  572  is limited by the degree of correlation between the intended target of a speech signal, and the accompanying sensor data produced alongside the speech signal. The greater the correlation, the more useful the sensor data will be in determining which portions of the signal are input speech. Such a correlation reflects that sensor data (such as from sensors of a wearable system, like those described above) can provide many of the same body language cues that humans use to interpret and contextualize others&#39; speech. For example, humans are accustomed to determining a speaker&#39;s intended speech target using the speaker&#39;s position (e.g., the speaker&#39;s movement, and distance from the listener); orientation (e.g., to whom the speaker is facing); eye gaze (e.g., who the speaker is making eye contact with); gesticulation (e.g., hand and arm movements, facial expressions); and so forth. Many of these body language cues also apply even when the speaker is addressing a device, such as a microphone-enabled speech recognition system. Sensor data can correspond to this body language, such as by providing data indicating the speaker&#39;s position, orientation, eye patterns, movement, and so on. Accordingly, using sensor data such as described above can provide valuable information as to the intended target of the corresponding speech. 
     In some examples, the predictive value of the system can be improved by utilizing both speech data (e.g., as described with respect to  FIG.  7 A ) and sensor data (e.g., as described above with respect to  FIG.  7 B ) that corresponds to the same speech signal. For example, where a speech segment corresponds to both a speech cue (e.g., the user raises their voice) and a sensor cue (e.g., the user quickly turns their head), the two cues combined can provide strong predictive evidence that the speech segment is intended as input from the user to a speech processing engine. 
       FIG.  7 C  illustrates an example portion of example system  500  in which analysis data  512  for a speech signal (e.g., speech signal  510 ), and sensor data  520  are both used by classifier  574  to determine a probability value  578  with which the speech segment  516  is directed by the user to a speech processing engine. Stages of the example system shown can proceed as described above with respect to  FIGS.  7 A and  7 B . For instance, stage  575  can parameterize/characterize speech segment  516  based on speech characteristics determined from speech signal  510  and/or speech signal analysis data  512 , such as described above with respect to stage  563  of  FIG.  7 A ; and stage  575  can also parameterize/characterize speech segment  516  based on sensor data  520 , such as described above with respect to stage  569  of  FIG.  7 B . At stage  576 , a probability value  578  can be determined for speech segment  516  based on its speech characteristics, such as described above with respect to stage  564  of  FIG.  7 A ; and based further on its corresponding sensor data, such as described above with respect to stage  570  of  FIG.  7 B . This probability value determination can make use of speech and/or sensor data, such as in a speech/sensor data repository  529 . Speech/sensor data repository  529  can include a database including information relating speech data to an intended target of that speech, such as described above with respect to speech data repository  527  of  FIG.  7 A ; and can further include information relating sensor data to an intended target of its corresponding speech, such as described above with respect to sensor data repository  528  of  FIG.  7 B . Further, speech/sensor data repository  529  can include information relating combinations of speech data and sensor data to an intended speech target. This may be useful in situations where neither the speech data nor the sensor data itself is independently predictive of an intended speech target, but the combination of the two correlates strongly to an intended speech target and has greater predictive value. 
     Generating a Probability Model 
       FIG.  8    is a flow chart showing a portion of example system  500 , illustrating an example of generating a probability model  586  from a raw speech signal  510 , according to some embodiments. In  FIG.  8   , stage  560  generates a probability model  586  (which may correspond to probability model  610 , described above with respect to  FIG.  6 C ) from a raw speech signal  510  (which may correspond to signal  600 , described above with respect to  FIGS.  6 A- 6 B ) and sensor data  520 . At stage  560 , statistical data  512  for the speech signal (e.g., representing statistical analysis of speech signal  510  such as described above) can be generated according to techniques familiar to those skilled in the art. At stage  514  of stage  560 , speech signal  510  can be segmented into individual speech segments  516 , such as described above with respect to  FIGS.  6 A- 6 D . For each speech segment  516 , one or more classifiers (e.g.,  562 ,  568 ,  574  described above) can be applied to generate a probability value, corresponding to the probability that the segment is input speech. In the example shown in  FIG.  8   , three classifiers are applied: a first classifier ( 562 ) generates a first probability value  566  based on the speech segment  516  and speech data  512 , such as described above with respect to  FIG.  7 A ; a second classifier ( 568 ) generates a second probability value  572  based on the speech segment  516  and sensor data  520 , such as described above with respect to  FIG.  7 B ; and a third classifier ( 574 ) generates a third probability value  578  based on the speech segment  516 , speech data  512 , and sensor data  520 , such as described above with respect to  FIG.  7 C . However, in some examples, only one classifier (e.g., classifier  574 ) need be used; and in some examples, additional classifiers beyond the three described here may be utilized to generate additional respective probability values. In some cases, different classifiers can apply different metrics to determine respective probability values. 
     In some examples where multiple classifiers are used to determine multiple respective probability values for speech segment  516 —such as the example shown in  FIG.  8   , where classifiers  562 ,  568 , and  574  are used to generate probability values  566 ,  572 , and  578 , respectively—it may be necessary to determine an overall probability  582  for speech segment  516 , based on the individual probability values generated by their respective classifiers. In such examples, comparison logic  580  can be used to mediate among the individual probability values to determine overall probability  582 . In some examples, comparison logic  580  may compute overall probability  582  as an average of individual probabilities (e.g.,  566 ,  572 ,  578 ). In some examples, comparison logic  580  may compute overall probability  582  as a weighted average of the individual probabilities, weighted for example by the fidelity of the input data (e.g., speech data  512 , sensor data  520 ). Other suitable techniques that can be employed by comparison logic  580  will be familiar to those skilled in the art, and the disclosure is not limited to any such technique or combination of techniques. Example techniques for combining the outputs of multiple classifiers include ensemble learning; Bayes optimal classifier, bagging (bootstrap aggregating), boosting techniques (e.g., AdaBoost); bucket of models; and stacking. 
     Once a probability value for a speech segment  516  has been determined, such as described above, the process of determining a probability value can repeat (stage  584 ) for any remaining speech segments  516 . For example, speech signal  600 , described above with respect to  FIGS.  6 A- 6 D , can be divided into seven speech segments ( 601  through  607 ), such as described above; if this speech signal  600  were provided as input  510  to the system shown in  FIG.  8   , each of stages  562 ,  568 , and  574  might be applied to each of the seven speech segments, resulting in a probability value  582  for each of the segments. Once a probability value has been determined for each speech segment  516 , the probability values can be used to generate a probability model  586 . As described above, probability model  586  can indicate a probability value for each speech segment of a speech signal. For example, in  FIG.  6 C , probability model  610  indicates a probability value for each speech segment of speech signal  600 . Generating probability model  586  for a speech signal can include expressing a probability value as a function of elapsed time of the speech signal; with such a model, such as shown as model  610  in  FIG.  6 C , a time t can be applied as input to the model, and the model will indicate the probability that the portion of the speech signal corresponding to time t (e.g., the portion of speech signal  600  after t seconds have elapsed) is directed as input to a speech processing engine. However, other suitable implementations of probability model  586  will be apparent and are within the scope of the disclosure. 
     Determining a Directed Speech Signal 
       FIG.  9    illustrates a portion of example system  500 , by which system  500  determines a directed speech signal  540  from raw speech signal  510  and/or sensor data  520 , such as by using probability model  586  described above. As shown in  FIG.  9   , at stage  530 , system  500  can generate a directed audio signal  540 , which can be an input speech signal to a speech processing engine that includes speech directed by a user to the speech processing engine, while excluding speech not directed by the user to the speech processing engine. Directed audio signal  540  can correspond to signal  620  described above with respect to  FIG.  6 D . An example of stage  530  generating directed audio signal  540  can proceed as follows with reference to  FIG.  9   . At stage  560 , raw speech signal  510  and/or sensor data  520  can be used to determine, for each of one or more segments of raw speech signal  510 , a probability that the segment corresponds to speech directed by the user as input to a speech processing engine. An example implementation of stage  560  is described above with respect to  FIG.  8   . As described above, the output of target determination stage  560  can be represented as probability model  586 , which can express, for example as a function of elapsed time, the probability that a portion of speech signal  510  is user speech directed at the speech processing engine. For example, model  586  can be a mathematical function expressing, for each time t of a raw speech signal having one or more segments, the probability that a segment of that raw speech signal corresponding to that time t is directed at the speech processing engine. As shown in the example in  FIG.  9   , stage  560  can also output a passthrough signal  588 , which may be a buffered signal corresponding to the raw speech signal  510  provided to target determination stage  560 . 
     At stage  590  of the example in  FIG.  9   , the raw speech signal (e.g., passthrough signal  588 ) can be filtered based on the probabilistic model  586 , such that segments of the raw speech signal  510  that correspond, with a sufficiently high probability, to input speech can be included in directed audio signal  540 ; and conversely, segments of raw speech signal  510  that do not correspond to input speech can be excluded from directed audio signal  540 . Stage  590  can employ a threshold probability value to serve as a cutoff to determine what constitutes a sufficiently high probability for an audio segment to be included in directed audio signal  540 . For example, as described above,  FIG.  6 C  illustrates a probability model  610  that corresponds to the raw speech signal  600  shown in  FIGS.  6 A and  6 B . As described above with respect to  FIG.  6 C , probability model  610  indicates, for each of speech segments  601  through  607  of speech signal  600 , a probability that the speech segment corresponds to input speech. In  FIG.  6 C , threshold value  618  is a value of 0.5; however, other threshold values can be used as appropriate. At stage  590 , speech segments with corresponding probability values that meet or exceed threshold value  618  (e.g., speech segments  601  and  606 ) could be included in directed audio waveform  540 ; and segments whose corresponding probability values do not meet threshold value  618  (e.g., speech segments  602 ,  603 ,  604 ,  605 , and  607 ) could be excluded from directed audio waveform  540 . The result would be the audio waveform  620  shown in  FIG.  6 D , in which only speech segments with sufficiently high probability (“What&#39;s the weather” and “tomorrow”) are included in the waveform  620 , and remaining segments are excluded. Compared to providing the raw speech signal  600  to the speech recognition system, providing audio waveform  620  as input to the speech recognition system promotes accuracy and computational efficiency, because the speech recognition system does not need to waste computational resources on irrelevant speech (or other audio) that carries a risk of generating erroneous results. 
     Training Classifiers 
       FIG.  10    illustrates an example process  1000  for capturing audio and non-audio classifier training data, according to one or more examples of the disclosure. Process  1000  can be applied to a human test subject  1012 , interacting (as a user might) with a speech processing engine (e.g., as included in a device with an integrated voice assistant). One or more microphones and one or more sensors can be configured to capture audio data and non-audio data (e.g., sensor data), respectively, from test subject  1012 . In some embodiments, the non-audio data may be non-microphone sensor data such as, for example, inertial measurement unit data, visual data, and the like. At step  1010  of the process, raw audio data of the voice of test subject  592  can be captured via the one or more microphones. Similarly, at step  1020 , non-audio data of the test subject can be captured via the one or more sensors. In some cases, test subject  1012  can be equipped with a single device, such as a wearable head device such as described above, that can include one or more microphones and one or more sensors. These microphones and sensors can be configured to for capturing the audio data at step  1010  and the non-audio data at step  1020 , respectively. Steps  1010  and  1020  can be performed simultaneously. 
     At step  1030 , the audio captured at step  1010  can be segmented and tagged as either input speech or non-input speech. This may be an automated process, a manual process, or some combination thereof. For example, audio data captured at step  1010  can be presented to a voice-activity detector (VAD) or to a human “tagger” observing test subject  1012 , and the audio data can be manually separated by the tagger into individual phrases or portions thereof. The tagger can then, based on the tagger&#39;s observation of test subject  1012  interacting with the speech recognition engine, manually identify each phrase as input speech or non-input speech. In some cases, the tagger can annotate each phrase with various metadata (e.g., an intended recipient for each phrase, or the audio source of each phrase). Other metadata entered by the tagger can include aspects about the speaker (e.g., the speaker&#39;s age, sex, and/or native language). In some examples, the tagger can also segment and tag non-speech audio (e.g., background noise and/or speech from people other than the speaker). 
     Similarly, at step  1040 , non-audio data captured at step  1020  can also be segmented and tagged as either being directed to the speech processing engine, or not. In some examples, a human tagger can identify and/or isolate non-audio data (e.g., sensor data) associated with individual phrases spoken by test subject  1012 , described above. In some cases, the tagger can manually associate non-audio data with audio data to which it corresponds. In some examples, non-audio data can be automatically associated with each phrase, based on start and end times of segmented and classified phrases from step  1030 . In some examples, non-audio data can include information about a user&#39;s head pose, gaze, gestures, location relative to target recipient phrases, or any other sensor data captured. 
     At step  1050 , the audio captured at step  1010 , the segmented and tagged phrases from step  1030  (e.g., input speech and non-input speech, including background noise or non-speech audio), the non-audio data captured at step  1020 , and/or the segmented and tagged non-audio data from step  1040  can be stored in a repository for classifier training. For example, speech data repository  527  described above can store audio from step  1010  and/or phrases from step  1030 ; sensor data repository  528  can store non-audio data from step  1020  and/or step  1040 ; and speech/sensor data repository  529  can store any of the above. In some examples, the audio captured at step  1010  and/or the segmented and tagged phrases from step  1030  are stored separately from the non-audio data captured step  1020 , and/or the segmented and tagged non-audio data from step  1040  (e.g., audio data and non-audio data are stored in separate databases). The stored audio data and/or non-audio data can be used to train classifiers, such as described above. 
     In some embodiments, audio and/or non-audio characteristics can be extracted from the input speech, non-input speech, or non-speech (e.g., background noise) stored in the one or more databases from step  1050  of  FIG.  10   . Examples of audio characteristics can include levels of (or changes in) volume (or signal amplitude), pre-vocalization hesitation, intra utterance hesitation, disfluency (e.g., stuttering, repetition), speech rate, syntax, grammar, vocabulary, length of phrase (e.g., duration, word count), pitch (e.g., fluctuation and contour), and/or prosody. Examples of non-audio characteristics that can be extracted from non-audio data include gestures, gaze (and changes thereto), head pose (and changes thereto), and position (e.g., distance and orientation) to physical and/or virtual objects (and changes thereto). In some examples, a Fourier transform of each speech and/or non-speech segment (e.g., each audio and/or non-audio segment corresponding to input speech, non-input speech, and/or non-speech) is stored in step  1050  of  FIG.  10    (e.g., both input speech and non-input speech) and provides a spectral representation of each speech segment (e.g., a function of frequency indicating the relative prevalence of various frequency parameters in the speech segment). Other methods of extracting time, frequency, and combined time-frequency parametric representations of audio and non-audio data will be familiar to those skilled in the art. In some examples, the extracted audio and/or non-audio characteristics can be stored with the corresponding input speech, non-input speech, and/or non-speech. 
     In some embodiments, the segmented and annotated audio data and non-audio data captured through process  1000  of  FIG.  10    (e.g., the input speech, non-input speech, and/or non-speech with corresponding metadata) can be fed into one or more classifiers for training purposes, such as described above. By running sample classes of input speech, non-input speech, and non-speech through one or more classifiers, the one or more classifiers can be trained to recognize input speech, non-input speech, and/or non-speech. In some examples, a majority subset (e.g., 60%) of the segmented and annotated audio data and non-audio data are run through the one or more classifiers and a minority subset or remaining (e.g., 40%) segmented and annotated audio data and non-audio data are used to evaluate the one or more classifiers. Evaluation techniques will be familiar to those skilled in the art. In some embodiments, these classifiers can be further trained by enabling users to confirm or reject classifications. 
     As described above, one or more classifiers (e.g., naive Bayes classifiers, support vector machines, k-nearest neighbor classifiers, AdaBoost classifiers, decision trees, or artificial neural networks) to distinguish between input speech and non-input speech. These classifiers can be trained to recognize audio characteristics and non-audio characteristics associated with input speech and/or non-input speech for improved speech processing. A method to train classifiers in accordance with the disclosure can include capturing audio and/or non-audio data; extracting audio and/or non-audio characteristics of input speech and non-input speech; training one or more classifiers, for example, using machine learning techniques, and/or, in some examples, updating the classifiers for improved input speech identification (e.g., by confirming and/or rejecting classifications), as described below. 
       FIG.  11    illustrates an example environment that can be used to generate audio data and sensor data for classifier training. The figure illustrates test subject  592  (which may correspond to test subject  1012  described above) in an environment  591  that includes a voice target (such as a voice assistant device including a speech processing engine), and one or more “distractor” sources.  593 A- 593 H. The distractor sources are configured to present test subject  592  with audio or visual “distractor” stimuli, to which test subject  592  may respond. Audio data and non-audio data (e.g., sensor data) associated with a response of test subject  592  to these distractor stimuli can be detected; this audio data and non-audio data can describe the response of test subject  592  (as detected by microphones and sensors) to external stimuli presented from the location of the corresponding distractor source. This audio data and non-audio data can be used accordingly to train a classifier (such as described above) to distinguish input speech from non-input speech (e.g., speech directed at an external stimulus, represented by the distractor source). 
     Distractor sources  593 A- 593 H can be placed at varying distances from and angles to test subject  592 , such as shown in the figure. Distractor sources  593 A- 593 H can be presented as speakers or visuals, or as any other suitable object that can produce sound and/or visuals (e.g., human beings, animals, electronic devices, etc.). For example, distractor source  593 A can represent a smart home device (e.g., a speaker with an integrated “smart” voice assistant (a “smart speaker”)) and distractor source  593 B can represent a human; the audio data and non-audio data can reflect differences in the response of test subject  592  based on the apparent identity of the distractor source. Environment  591  can represent a controlled environment (e.g., a sound proof room, or a room in which distractor sources  593 A- 593 H produce sound in a controlled fashion) or an uncontrolled environment (e.g., in the home of test subject  592  or in a public place). For example, in a controlled environment, test subject  592  can freely interact (e.g., with little to no direction or script) with a wearable device with an integrated voice assistant (e.g., wearable head device  100 ) to instruct the device to perform a particular operation (e.g., open an app, play music, query information, for example, from the Internet, enter information into calendar, read information from a calendar, make a phone call, send a text message, control a smart thermostat, control a smart lock, control one or more smart lights, or any other operation). Test personnel (represented by distractor sources  593 A- 593 H) can engage in conversation with test subject  592 . This prompts test subject  592  to interact with wearable device and the test personnel. In some examples, distractor sources  593 A- 593 H can be virtual sources; for example, a software application running on a wearable system can produce sound from one or more virtual sound sources represented by distractor sources  593 A- 593 H. In some examples, distractor sources  593 A- 593 H may be presented via a wearable head device worn by test subject  592  (e.g., via speakers and/or a display of the wearable head device), with audio data and non-audio data potentially captured by microphones and sensors of that same wearable device. 
     Interactions such as shown in  FIG.  11    (e.g., spoken phrases  594 A- 594 D spoken in the environment  591 ) can be detected and used to train one or more classifiers in accordance with this disclosure. For example, spoken phrases  594 A- 594 D can be recorded (e.g., by one or more microphones  150  on wearable head device  100  or by one or more microphones on sound source  594 A) in an audio file as a continuous audio stream: “Hey Magic Leap, open . . . Mom, can I . . . Not right now, Charlie . . . open Maps.” Similarly, non-audio data of test subject  592  interacting with one or more distractor sources  593 A- 593 H can be captured simultaneously with the audio data. In some examples, data from one or more sensors on a wearable system (e.g., wearable head device  100  in  FIG.  1    and/or handheld controller  200  in  FIG.  2   ) on test subject  592  can be used to capture information about the head positions of test subject  592  (e.g., as detected by position and orientation sensors of the wearable head device), hand gestures (e.g., as detected by movements of handheld controller  200  or by one or more cameras  130 A and  130 B configured on wearable head device  100 ), eye gaze (e.g., as detected by one or more cameras  128 A and  102 B configured on wearable head device  100 ), and/or the distance of test subject  592  from one or more distractor sources  593 A- 593 H (e.g., as measured from the wearable head device  100  to one or more of distractor sources  593 A- 593 H by one or more cameras  130 A and  130 B and/or GPS, acoustic, radar, or IMU sensors). 
     With respect to the systems and methods described above, elements of the systems and methods can be implemented by one or more computer processors (e.g., CPUs or DSPs) as appropriate. The disclosure is not limited to any particular configuration of computer hardware, including computer processors, used to implement these elements. In some cases, multiple computer systems can be employed to implement the systems and methods described above. For example, a first computer processor (e.g., a processor of a wearable device coupled to one or more microphones) can be utilized to receive input microphone signals, and perform initial processing of those signals (e.g., signal conditioning and/or segmentation, such as described above). A second (and perhaps more computationally powerful) processor can then be utilized to perform more computationally intensive processing, such as determining probability values associated with speech segments of those signals. Another computer device, such as a cloud server, can host a speech processing engine, to which input signals are ultimately provided. Other suitable configurations will be apparent and are within the scope of the disclosure. 
     Although the disclosed examples have been fully described with reference to the accompanying drawings, it is to be noted that various changes and modifications will become apparent to those skilled in the art. For example, elements of one or more implementations may be combined, deleted, modified, or supplemented to form further implementations. Such changes and modifications are to be understood as being included within the scope of the disclosed examples as defined by the appended claims.