Patent Description:
This instant specification relates to speech detection.

As computer processors have decreased in size and expense, mobile computing devices have become increasingly widespread. Designed to be portable, many mobile computing devices are lightweight and small enough to be worn or carried in a pocket or handbag. However, the portability of modern mobile computing devices comes at a price: today's mobile computing devices often incorporate small input devices to reduce the size and weight of the device. For example, many current mobile devices include small keyboards that many people (especially those with poor dexterity) find difficult to use.

Some mobile computing devices address this problem by allowing a user to interact with the device using speech. For example, a user can place a call to someone in his contact list by simply speaking a voice command (e.g., "call") and the name of the person into the phone. However, speech can be difficult to distinguish from background noise in some environments, and it can hard to capture user speech in a manner that is natural to the user. In addition, it can be challenging to begin recording speech at the right time. For example, if recording begins after the user has started speaking the resulting recording may not include all of the user's voice command. Furthermore, a user may be notified that a spoken command was not recognized by the device after the user has spoken, which can be frustrating for users.

<CIT> describes push to talk voice buffering systems and methods. This document indicates that delays between initiating a push to talk (PTT) session and the time when an initiator is allowed to begin speaking are frustrating. This document describes that this problem is solved by buffering audio data. The method proposed by <CIT> comprises a push to talk origination action that comprises a user pushing a push to talk button mat the beginning of a sequence that comprises the cellular telephone indicating to the user that the user can begin speaking for the push to talk call by playing a chirp-like sound. At or about the time that the chirp plays, the cell phone issues a command to a vocoder to turn on and start recording audio information in memory. The cell phone buffers the audio information for transmitting the audio information at a later time.

Various aspects of the present invention are defined in the independent claims. Some preferred features are defined in the dependent claims.

This document describes systems and techniques for detecting speech. In some implementations, a mobile device can determine its distance from a user, as well as its angle relative to the user. Based on this information, the device can initiate or stop voice recording. In an illustrative example, the user may place the device in a predetermined position, e.g., next to his ear. The device may detect that it has entered this position and begin voice recording. Once the user moves the device out of this position, the device may stop recording user input. The recorded speech may be used as input to an application running on the device or running on an external device.

<FIG> is a conceptual diagram <NUM> of multisensory speech detection. The diagram <NUM> depicts a user <NUM> holding a mobile device <NUM>. The mobile device <NUM> may be a cellular telephone, PDA, laptop, or other appropriate portable computing device. In the illustrative example shown in <FIG>, the user <NUM> may want to interact with an application running on the mobile device <NUM>. For instance, the user may want to search for the address of a business using a Web-based application such as GOOGLE MAPS. Typically, the user <NUM> would use the mobile device <NUM> to type the name of the business into a search box on an appropriate website to conduct the search. However, the user <NUM> may be unwilling or unable to use the device <NUM> to type the necessary information into the website's search box.

In the illustrative example of multisensory speech detection shown in <FIG>, the user <NUM> may conduct the search by simply placing the mobile device <NUM> in a natural operating position and saying the search terms. For example, in some implementations, the device <NUM> may begin or end recording speech by identifying the orientation of the device <NUM>. The recorded speech (or text corresponding to the recorded speech) may be provided as input to a selected search application.

The letters "A," "B," and "C "in <FIG> represent different states in the illustrative example of multisensory speech detection. In State A, the user <NUM> is holding the device <NUM> in a non-operating position; that is, a position outside a predetermined set of angles or too far from the user <NUM> or, in some cases, both. For example, between uses, the user <NUM> may hold the device <NUM> at his side as shown in <FIG> or place the device in a pocket or bag. If the device <NUM> has such an orientation, the device <NUM> is probably not in use, and it is unlikely that the user <NUM> is speaking into the mobile device <NUM>. As such, the device <NUM> may be placed in a non-recording mode.

When the user <NUM> wants to use the device <NUM>, the user <NUM> may place the device <NUM> in an operating mode/position. In the illustrative example shown in the diagram <NUM>, the device <NUM> may determine when it is placed in selected operating positions, referred to as poses. State B shows the mobile device <NUM> in several example poses. For example, the left-most figure in State B illustrates a "telephone pose" <NUM>. A telephone pose can, in some implementations, correspond to the user <NUM> holding the mobile device <NUM> in a position commonly used to speak into a telephone. For example, as shown in <FIG>, the device <NUM> may be held to a side of the user's <NUM> head with the speaker of the device <NUM> held near the user's <NUM> ear. Holding the device <NUM> in this way can make it easier for the user <NUM> to hear audio emitted by the device <NUM> and speak into a microphone connected to the device <NUM>.

The middle figure shown in State B depicts the user <NUM> holding the device <NUM> in a "PDA pose" <NUM>. For example, as shown in <FIG>, PDA pose <NUM> may correspond to the user <NUM> holding the mobile device <NUM> at nearly arm's length and positioned so that the user <NUM> can see and interact with the mobile device <NUM>. For instance, in this position, the user <NUM> can press buttons on the keypad of the device <NUM> or a virtual keyboard displayed on the device's <NUM> screen. In some cases, the user <NUM> may also enter voice commands into the device <NUM> in this position.

Finally, the right-most figure shown in State B illustrates a "walkie-talkie pose" <NUM>. In some cases, a walkie-talkie pose <NUM> may comprise the user <NUM> holding the mobile device <NUM> to his face such that the device's <NUM> microphone is close the user's <NUM> mouth. This position may allow the user <NUM> to speak directly into the microphone of the device <NUM>, while also being able to hear sounds emitted by a speakerphone linked to the device <NUM>.

Although <FIG> shows three poses, others may be used. For instance, in an alternative implementation, a pose may take into account whether a mobile device is open or closed. For example, the mobile device <NUM> shown in <FIG> may be a "flip phone"; that is, a phone having a form factor that includes two or more sections (typically a lid and a base) that can fold together or apart using a hinge. For some of these devices, a pose may include whether the phone is open or closed, in addition to (or in lieu of) the orientation of the phone. For instance, if the mobile device <NUM> is a flip phone, the telephone pose <NUM> shown in <FIG> may include the device being open. Even though the current example describes a flip phone, other types or form factors (e.g., a phone that swivels or slides open) may be used.

When the device <NUM> is identified as being in a predetermined pose, the device <NUM> may begin recording auditory information such as speech from the user <NUM>. For example, State C depicts a user speaking into the device <NUM> while the device <NUM> is in the telephone pose. Because, in some implementations, the device <NUM> may begin recording auditory information when the device <NUM> is detected in the telephone pose <NUM>, the device <NUM> may begin recording just before (or as) the user <NUM> starts speaking. As such, the device <NUM> may capture the beginning of the user's speech.

When the device <NUM> leaves a pose, the device <NUM> may stop recording. For instance, in the example shown in <FIG>, after the user <NUM> finishes speaking into the device <NUM>, he may return the device <NUM> to a non-operating position by, for example, placing the device <NUM> by his side as shown at State A. When the device <NUM> leaves a pose (telephone pose <NUM> in the current example), the device <NUM> may stop recording. For example, if the device <NUM> is outside a selected set of angles and/or too far from the user <NUM>, the device <NUM> can cease its recording operations. In some cases, the information recorded by the device <NUM> up to this point can be provided to an application running on the device or on a remote device. For example, as noted above, the auditory information can be converted to text and supplied to a search application being executed by the device <NUM>.

<FIG> is a block diagram <NUM> of an example multisensory speech detection system. The block diagram <NUM> shows an illustrative mobile device <NUM>. The device <NUM> includes a screen <NUM> that, in some cases, can be used to both display output to a user and accept user input. For example, the screen <NUM> may be a touch screen that can display a keypad that can be used to enter alphanumeric characters. The device <NUM> may also include a physical keypad <NUM> that may also be used to input information into the device. In some cases the device <NUM> may include a button (not shown) on the keypad <NUM> or another part of the phone (e.g., on a side of the phone) that starts and stops a speech application running on the device <NUM>. Finally, the device <NUM> can incorporate a trackball <NUM> that, in some cases, may be used to, among other things, manipulate a pointing element displayed on a graphical user interface on the device <NUM>.

The device <NUM> may include one or more sensors that can be used to detect speech readiness, among other things. For example, the device <NUM> can include an accelerometer <NUM>. The accelerometer <NUM> may be used to determine an angle of the device. For example, the accelerometer <NUM> can determine an angle of the device <NUM> and supply this information to other device <NUM> components.

In addition to the accelerometer <NUM>, the device <NUM> may also include a proximity sensor <NUM>. In some cases, the proximity sensor <NUM> can be used to determine how far the device <NUM> is from a user. For example, the proximity sensor <NUM> may include an infrared sensor that emits a beam of infrared light and uses the reflected signal to compute the distance to an object. In alternative implementations, other types of sensors may be used. For example, the sensor may be capacitive, photoelectric, or inductive, among other kinds of sensors.

The device can also include a camera <NUM>. Signals from the camera <NUM> can be processed to derive additional information about the pose of the device <NUM>. For example, if the camera <NUM> points toward the user, the camera <NUM> can determine the proximity of the user. In some cases, the camera <NUM> can determine the angle of the user using features having a known angle such as the horizon, vehicles, pedestrians, etc. For example, if the camera <NUM> is pointing at a general scene that does not include a user, the camera <NUM> can determine its orientation in the scene in an absolute coordinate system. However, if the camera <NUM> can see the user, the camera <NUM> can determine its orientation with respect to the user. If the camera <NUM> can see both the general scene and the user, the camera <NUM> can determine both its orientation with respect to the user and the scene and, in addition, can determine where the user is in the scene.

The device may also include a central processing unit <NUM> that executes instructions stored in memory <NUM>. The processor <NUM> may comprise multiple processors responsible for coordinating interactions among other device components and communications over an I/O interface <NUM>. The device <NUM> may communicate with a remote computing device <NUM> through the Internet <NUM>. Some or all of the processing performed by the gesture classifier <NUM>, pose identifier <NUM>, speech detector <NUM>, speaker identifier <NUM> and speech endpointer <NUM> can be performed by the remote computing device <NUM>.

A microphone <NUM> may capture auditory input and provide the input to both a speech detector <NUM> and a speaker identifier <NUM>. In some implementations, the speech detector <NUM> may determine if a user is speaking into the device <NUM>. For example, the speech detector <NUM> can determine whether the auditory input captured by the microphone <NUM> is above a threshold value. If the input is above the threshold value, the speech detector <NUM> may pass a value to another device <NUM> component indicating that the speech has been detected. In some cases, the device <NUM> may store this value in memory <NUM> (e. g, RAM or a hard drive) for future use.

In some cases, a speech detector <NUM> can determine when a user is speaking. For example, the speech detector <NUM> can determine whether captured audio signals include speech or consist entirely of background noise. In some cases, the speech detector <NUM> may assume that the initially detected audio is noise. Audio signals at a specified magnitude (e.g., <NUM> dB) above the initially detected audio signal may be considered speech.

If the device includes a camera <NUM> the camera <NUM> may also provide visual signals to the speech detector <NUM> that can be used to determine if the user is speaking. For example, if the user's lips are visible to the camera, the motion of the lips may be an indication of speech activity, as may be correlation of that motion with the acoustic signal. A lack of motion in the user's lips can, in some cases, be evidence that the detected acoustic energy came from another speaker or sound source.

The speaker identifier <NUM>, in some cases, may be able to determine the identity of the person speaking into the device <NUM>. For example, the device <NUM> may store auditory profiles (e.g., speech signals) of one or more users. The auditory information supplied by the microphone <NUM> may be compared to the profiles; a match may indicate that an associated user is speaking into the device <NUM>. Data indicative of the match may be provided to other device <NUM> components, stored in memory, or both. In some implementations, identification of a speaker can be used to confirm that the speech is not background noise, but is intended to be recorded.

The speaker identifier <NUM> can also use biometric information obtained by the camera <NUM> to identify the speaker. For example, biometric information captured by the camera can include (but is not limited to) face appearance, lip motion, ear shape, or hand print. The camera may supply this information to the speaker identifier <NUM>. The speaker identifier <NUM> can use any or all of the information provided by the camera <NUM> in combination with (or without) acoustic information to deduce the speaker's identity.

The device <NUM> may also include a gesture classifier <NUM>. The gesture classifier <NUM> may be used to classify movement of the device <NUM>. In some cases, the accelerometer <NUM> can supply movement information to the gesture classifier <NUM> that the gesture classifier <NUM> may separate into different classifications. For example, the gesture classifier <NUM> can classify movement of the phone into groups such as "shake" and "flip. " In addition, the gesture classifier <NUM> may also classify motion related to gestures such as "to mouth," "from mouth," "facing user," "to ear," and "from ear.

A pose identifier <NUM> included in the device <NUM> may infer/detect different poses of the device <NUM>. The pose identifier <NUM> may use data provided by the proximity sensor <NUM> and the gesture classifier <NUM> to identify poses. For example, the pose identifier <NUM> may determine how far the device <NUM> is from an object (e.g., a person) using information provided by the proximity sensor <NUM>. This information, combined with a gesture classification provided by the gesture classifier <NUM> can be used by the posture identifier <NUM> to determine which pose (if any) the device <NUM> has been placed in. In one example, if the gesture classifier <NUM> transmits a "to ear" classification to the pose identifier <NUM> and the proximity sensor <NUM> indicates that the device is being held close to the user, the pose identifier <NUM> may determine that the device <NUM> is in telephone pose. A camera <NUM> can also be used to provide evidence about movement. For example, the optical flow detected by the camera <NUM> may provide evidence of movement.

The device may also include a speech endpointer <NUM>. The speech endpointer <NUM>, in some implementations, can combine outputs from the pose identifier <NUM>, speaker identifier <NUM>, and speech detector <NUM>, to determine, inter alia, whether a user is speaking into the device, beginning to speak into the device, or has stopped speaking into the device. For example, the pose identifier <NUM> may transmit information to the endpointer <NUM> indicating that the device is not in an operating position. Inputs from the speech detector <NUM> and speaker identifier <NUM> may indicate that the user is not currently speaking. The combination of these inputs may indicate to the endpointer <NUM> that the user has stopped speaking.

<FIG> and <FIG> are flow charts of example processes <NUM> and <NUM>, respectively, for multisensory speech detection. The processes <NUM> and <NUM> may be performed, for example, by a system such as the system shown in <FIG> and, for clarity of presentation, the description that follows uses that system as the basis of an example for describing the processes. However, another system, or combination of systems, may be used to perform the processes <NUM> and <NUM>.

<FIG> illustrates an example process <NUM> of multisensory speech detection. The process <NUM> begins at step <NUM> where it is determined whether a record button has been pressed. For example, as noted above, the mobile devices <NUM> includes a button that allows a user to initiate or end speech recording by pressing the button. If a button press is detected at step <NUM> the process <NUM> starts recording speech and display a start of input (SOI) confirmation that recording has started at step <NUM>. For example, the device <NUM> may execute a recording program stored in memory when the button is pressed. In addition, the device <NUM> displays a message on the screen indicating that recording has begun. In some implementations, the device <NUM> may vibrate or play a tone, in addition to, or in lieu of, display an on-screen confirmation.

However, if a record button press is not detected at step <NUM>, the process <NUM> proceeds to step <NUM> where it is determined whether a record gesture has been detected. For example, a user may be holding the device <NUM> in PDA pose. When the user brings the device <NUM> to his mouth, the gesture classifier <NUM> may classify this motion as a "to-mouth" gesture and cause the device <NUM> to execute a recording application. In some implementations, other gestures such as shaking or flipping the phone can be a record gesture. In response, the process <NUM> proceeds to step <NUM> where a recording process is started and a recording confirmation is displayed as described above. If not, the process <NUM> returns to step <NUM> where it determines if a record button has been pressed.

The process <NUM> loads settings into an endpointer at step <NUM>. The device <NUM> loads pose-specific speech detection parameters such as a speech energy threshold that can be used to detect speech. For example, in some cases, the speech energy threshold for a pose may be compared to detected auditory information. If the auditory information is greater than the speech energy threshold, this may indicate that a user is speaking to the device. In some implementations, poses may have an associated speech energy threshold that is based on the distance between the device <NUM> and a user when the device is in the specified pose. For instance, the device <NUM> may be closer to a user in telephone pose than it is in PDA pose. Accordingly, the speech energy threshold may be lower for the PDA pose than it is for the telephone pose because the user's mouth is farther from the device <NUM> in PDA pose.

At step <NUM>, an endpointer runs. For example, device <NUM> executes endpointer <NUM>. In response, the endpointer <NUM> uses parameters loaded at step <NUM> to determine whether the user is speaking to the device, and related events, such as the start and end of speech. For example, the endpointer <NUM> may use a speech energy threshold, along with inputs from the pose identifier <NUM>, speech detector <NUM>, and speaker identifier <NUM> to determine whether the user is speaking and, if so, whether the speech is beginning or ending.

At step <NUM>, an end-of-speech input is detected. As discussed above, the endpointer <NUM> may determine whether speech has ended using inputs from other device components and a speech energy threshold. If the end of speech input has been detected, recording may cease and an end of input (EOI) display indicating that recording has ended is provided at step <NUM>. For example, a message may appear on the screen of the device <NUM> or a sound may be played. In some cases, tactile feedback (e.g., a vibration) may be provided.

<FIG> illustrates an example alternative process <NUM> of multisensory speech detection. The process begins at step <NUM> where a pose is read from a pose detector. For example, the pose identifier <NUM> may provide the current pose of the device, or an indication of the current pose may be read from memory <NUM>.

At step <NUM>, it is determined whether the device <NUM> is in phone pose. For example, the pose identifier <NUM> can use inputs from the proximity sensor <NUM> and the gesture classifier <NUM> to determine if the device is in phone pose. In some cases, the pose of the device can be identified by determining how far the device is from the user and whether the device is within a set of predetermined angles. If the device <NUM> is in phone pose, a sound confirming that recording has begun may be played at step <NUM>. In some implementations, another type of feedback (e.g., a vibration or a display of a message) may be provided with, or instead of, the audio confirmation.

At step <NUM>, phone pose settings may be loaded into an endpointer. For example, a speech energy threshold associated with the phone pose may be read from memory <NUM> into the endpointer <NUM>.

Similarly, at step <NUM> it is determined whether the device is in walkie-talkie pose. As noted above, the pose identifier <NUM> can use inputs from the gesture classifier <NUM> and the proximity sensor <NUM> to determine the pose of the device. If the device is in walkie-talkie pose, confirmation that recording has begun may be displayed on the screen (in some cases, confirmation may also be tactile or auditory) at step <NUM> and walk-talkie pose settings may be loaded into an endpointer at step <NUM>.

At step <NUM>, it is determined whether the device is in PDA pose. In some cases, the pose of the device can be determined as described in regards to steps <NUM> and <NUM> above. If the device is not in PDA pose, the method can return to step <NUM>. If the device is in PDA pose, it can be determined whether a record button has been pressed at step <NUM>. If a record button has not been pressed, the method proceeds to step <NUM>, where it is determined if a record gesture has been detected. For example, as discussed in relation to step <NUM> of <FIG> above, the device <NUM> may detect a movement of the device <NUM> toward a user's mouth. In some cases, the device <NUM> may interpret this motion as a record gesture.

If a record button was pressed at step <NUM> or a record gesture was detected at step <NUM>, a message confirming that recording has begun can be displayed on the screen of the device <NUM> at step <NUM>. In some cases, the device <NUM> may vibrate or play a sound to indicate that recording has started. Subsequently, settings associated with the PDA pose may be loaded into an endpointer at step <NUM>. For example, a speech energy threshold may be loaded into the endpointer <NUM>.

For each of the poses described above, after the appropriate pose settings are read into an endpointer, the endpointer may be run at step <NUM>. For example, a processor <NUM> associated with the device <NUM> may execute instructions stored in memory that correspond to the endpointer <NUM>. Once the endpointer <NUM> has begun executing, the endpointer <NUM> may determine whether an end-of-speech input has been detected at step <NUM>. For example, the endpointer <NUM> may determine whether an end-of-speech input has been detected using outputs from the pose identifier <NUM>, speech detector <NUM>, speaker identifier <NUM>, and parameters associated with the pose that have been loaded into the endpointer <NUM>. For example, the endpointer <NUM> may determine when the device <NUM> is no longer in one of the specified poses using outputs from the previously mentioned sources. At step <NUM>, the process may play or display a confirmation that speech recording has ceased. For example, an end-of-recording message may be displayed on the device's <NUM> screen or a sound may be played. In some cases, the device <NUM> may vibrate.

<FIG> show example coordinate systems <NUM> and <NUM> for gesture recognition. <FIG> shows an illustrative Cartesian coordinate system <NUM> for a mobile device. The illustrative coordinate system <NUM> can be a three-dimensional coordinate system with X-, Y-, and Z-axes as shown in <FIG>. In some cases, an accelerometer (such as the accelerometer <NUM> shown in <FIG>) can be used to determine an angle of the mobile device in the coordinate system shown in <FIG>. The determined angle can, in turn, be used to determine a pose of the device.

For example, acceleration data provided by the accelerometer <NUM> may be smoothed by, for instance, using a digital filter (e.g., an infinite impulse response filter). In some cases, the accelerometer may have a sample frequency of <NUM>. In addition, the infinite impulse response filter may have a filtering factor of <NUM>. The magnitude of the instantaneous acceleration may be calculated from the residual of the filter. A resulting gravity vector may be projected onto XY and YZ planes of the coordinate system and the angle subtended by the projected components may be calculated using the inverse tangent of the components. The resulting two angles can be projected onto a new plane such as the one shown in <FIG> and critical angle bounding boxes <NUM> and <NUM> can be defined around the left and right hand positions of the phone to a user's ear. As described in further detail below, these bounding boxes can be used to detect gestures, among other things.

<FIG> is an example state machine <NUM> for gesture recognition. The state machine <NUM> can use the critical angle bounding boxes described above, along with proximity information, to classify gestures. The illustrative state machine can be clocked by several events: a specified proximity being detected, the device <NUM> being within a critical set of angles, or a time expiring. For example, the illustrative state machine can wait for a predetermined proximity to be detected at state <NUM>. In some cases, the state machine <NUM> may activate the proximity sensor <NUM> when either the instantaneous acceleration of the device is greater than a threshold or the device <NUM> is placed at a set of critical angles. In some cases, the critical angles may be angles that fall within the bounding boxes shown in <FIG>. For example, the left-most bounding box <NUM> may include angles between -<NUM> and -<NUM> degrees in the XY plane and -<NUM> and <NUM> degrees in the YZ plane. Similarly, bounding box <NUM> may include angles between <NUM> and <NUM> degrees in the XY plane and -<NUM> and <NUM> degrees in the YZ plane.

If the proximity sensor detects an object within a preset distance of the device <NUM>, the state machine <NUM> transitions to state <NUM> where it waits for an angle. In some cases, if the proximity sensor <NUM> detects a user within the predetermined distance and the device <NUM> was previously determined to be at the critical angles (e.g., the state machine was activated because the device <NUM> was placed at the critical angles) the state machine <NUM> transitions to the next state <NUM>. If the device <NUM> was not previously placed at the critical angles, the device <NUM> may wait for a preset period for the device to be placed at the critical angles; this preset period may allow any acceleration noise to settle. In some cases, the preset period may be one second. If the device is not placed at the critical angles within the predetermined period, the state machine <NUM> may transition back to state <NUM>. However, if the device <NUM> is detected at the critical angles within the predetermined threshold the state machine transitions to state <NUM> where a gesture is detected. In some cases, the gesture classifier <NUM> may classify the detected gesture. For example, the gesture may fall into the following categories: "to mouth," "from mouth," "facing user," "to ear," and "from ear. " In some implementations, other categories may be defined. If the device <NUM> is determined to no longer be at the critical angles, the state machine <NUM> may transition to state <NUM>, where the gesture has expired. In some implementations, a minimum debounce period may prevent this transition from happening because of angle bounce. For example, the minimum debounce period may be <NUM> seconds.

<FIG> illustrates another implementation of a state machine <NUM> for gesture recognition. <FIG> shows the illustrative state machine <NUM> responding to variations in gestures, where the gestures vary according to the detected acceleration (e.g., slow, medium, and fast gestures). The illustrative state machine <NUM> may be useful in implementations where the device <NUM> includes a proximity sensor <NUM> that does not detect a proximate condition if the proximity sensor <NUM> is activated when the device <NUM> is already proximate a surface or where activation of the proximity detector may trigger other actions such as switching off the screen. In some cases, to address this issue, the proximity sensor <NUM> may be activated when an instantaneous acceleration surpasses a threshold. In some cases, the proximity sensor <NUM> may be activated when the sensor <NUM> crosses the instantaneous acceleration across all axes.

The state machine <NUM> begins in an initial state <NUM>. If an acceleration above a threshold is detected, the machine <NUM> transitions to state <NUM> where it waits for proximity detection after the detected acceleration. In some implementations, the acceleration threshold may be <NUM>. In some cases, the wait may be <NUM> seconds. If the device <NUM> is proximate an object such as a user, the state machine <NUM> transitions to state <NUM> where it waits a predetermined time for the device to placed at the critical angles. In some cases, the wait may be one second. If the device is not placed at the critical angles within the specified time, the state machine returns to its initial state <NUM>. However, if the device is placed at the critical angles, the state machine <NUM> transitions to state <NUM> where a gesture is detected in the manner described above. When the device is no longer within the critical angles, the state machine <NUM> transitions to state <NUM> where the gesture has expired. These transitions may correspond to a fast gesture.

In some cases, after acceleration has been detected, the device <NUM> may be placed in critical angles and, as such, the state machine <NUM> can proceed to state <NUM>, where it waits for a proximity detection. If no proximity detection is made within a preset time, the state machine <NUM> can transition to state <NUM> where the waiting proximity time has expired and subsequently return to its initial state <NUM>. In some cases, the preset time may be one second. However, if a proximity detection is made before the preset time expires, the state machine <NUM> can transition to states <NUM> and <NUM> as described above. In some cases, this series of transitions may correspond to a medium-speed gesture.

If the state machine <NUM> is in its initial state <NUM> and the device <NUM> has been placed at the critical angles the state machine <NUM> can transition to state <NUM> where the state machine <NUM> waits for proximity detection. If proximity detection occurs before a timeout period, the state machine <NUM> proceeds to state <NUM> where a gesture is detected. If the device <NUM> is moved from the critical angles, the state machine <NUM> transitions to state <NUM> where the gesture has expired. This series of transitions may correspond to a gesture made at relatively slow pace.

<FIG> illustrate Bayes nets for pose and speech detection. In some cases, a Bayesian network <NUM> may be used to recognize gestures. Outputs from a proximity sensor <NUM>, accelerometer <NUM>, and speech detector <NUM> can be combined into a Bayesian network as shown in <FIG>. The Bayesian network <NUM> shown in <FIG> can represent the following distribution: <MAT> In equation (<NUM>), x_aud can represent an audio feature vector, x_accel can represent acceleration feature vector, and x_prox can represent a proximity feature vector. A hidden state variable, EPP, can represent a cross product of an endpointer speech EP and a pose state variable Pose. The EP and Pose variables can be discrete random variables.

<FIG> illustrates a factorization <NUM> of the hidden state into the EP vector and the Pose state variable. This factorization can facilitate better use of training data and faster inference. The distribution can be factored as follows: <MAT> In some cases, the distributions p(x_aud, x_accel | EP, Pose) and p(x_aud, x_accel | EP, Pose) and p (x_prox | Pose) can be Gaussian Mixture Models.

In some implementations, the posterior probability for EP can be used as input to an endpointer state machine. For example, <FIG> illustrates an endpointer state machine <NUM>. In the illustrative implementation shown in <FIG>, an EP posterior probability can be thresholded and a time frame may be determined to contain either noise or speech. In this example, noise may be represented by a zero value and speech can be represented by a one value. A circular buffer of thresholded values may be stored. A one value in a buffer can be used to drive the endpointer state machine shown in <FIG>. For example, if the initial state <NUM> is pre-speech and the number of one values in the circular buffer exceeds a threshold, the machine moves to state <NUM> "Possible Onset. " If the number of one values fall below the threshold the machine moves back to the "Pre-Speech" state <NUM>. The state machine <NUM> can transition backward and forward among the "Speech Present" <NUM>, "Possible Offset" <NUM> and "Post Speech" <NUM> states in a similar fashion.

<FIG> illustrates a dynamic Bayes net for pose and speech detection. <FIG> shows a collection of EPP states chained together in a Hidden Markov Model <NUM>. In the illustrative implementation, the State EPP can be a cross product of EP state and the Pose state and transitions between the states can be defined by a transition matrix. The illustrative gesture recognizer in <FIG> can be trained by employing an Expectation Maximization algorithm. Inference to determine a speech/noise state can be performed by the Viterbi algorithm or a Forward-Backward algorithm. In some cases, more complex states can be used. For instance the environment of the user (e.g., in the street, in a home, in a moving car, in a restaurant, etc.) or device could be inferred based upon signals from the sensors and used in the determination of the pose and endpointer state.

<FIG> show screenshots of an example graphical user interface for providing feedback about audio signal quality. In some implementations, the illustrative graphical user interface may provide feedback regarding audio signal quality before, during, and after a user speaks commands into a mobile computing device. For example, before a user speaks, the graphical user interface can provide visual or audio feedback that may indicate whether speech will be accurately captured by the device. In some cases, the feedback may indicate that the user should use the device in a particular manner (e.g., place the device in a particular pose) or warn the user that background noise may impair the detection and accurate recording of speech. In some implementations, the feedback may be used to limit the modes of operation available to the user or suggest an operating mode that may increase the chance of successful voice capture.

In some cases, as the user is speaking the graphical user interface can provide feedback on the quality of the audio captured by the device. For example, a visual indication of the amplitude of the recorded audio can be displayed on the screen while the user is speaking. This may provide the user an indication of whether background noise is interfering with sound recording or whether the user's commands are being properly recorded. After the user has finished speaking, the graphical user interface may display a representation of the captured voice commands to the user.

<FIG> shows an illustrative graphical user interface <NUM> for providing feedback about audio signal quality. The illustrative graphical user interface <NUM> can, in some cases, include a message area <NUM>. Visual indicators such as text and waveforms may be displayed in the message area <NUM> to indicate, for example, a mode of operation of the device or a representation of recorded audio. For example, as shown in <FIG>, when the device is in a recording mode, a "Speak Now" message may be displayed in the message area <NUM>. Messages indicating that current noise conditions may interfere with speech recording may be displayed in message area <NUM>. In some situations, the message area <NUM> may also show messages allowing a user to continue or cancel the recording operation. The preceding examples are illustrative; other types of data may be displayed in the message area <NUM>.

The illustrative graphical user interface <NUM> can also include a visual audio level indicator <NUM>. In an illustrative implementation, the visual audio level indicator <NUM> can indicate the amplitude of audio captured by a mobile device. For example, as a user is speaking the indicator <NUM> can go up an amount related to the amplitude of the detected speech. In some circumstances, the indicator <NUM> may allow a user to determine whether background noise is interfering with speech recording. For example, if the indicator <NUM> goes up before the user begins speaking, background noise may interfere with speech recording. If the indicator <NUM> does not go up while the user is speaking, this may indicate the user's voice commands are not being properly recorded.

In some cases, the audio level indicator <NUM> can display a representation of the log of the Root Means Square (RMS) level of a frame of audio samples. The log RMS level of the frame of audio samples may represent a background noise level. In some cases, the RMS value may be equal to <MAT>. In some cases, the log RMS level of a frame of audio samples may be determined by the following equation: <MAT> Here, xt can be an audio sample value at a time t.

In some cases, audio level indicator <NUM> may display a representation of a signal-to-noise ratio; i.e., strength of a speech signal relative to background noise. For example, the signal-to-noise ratio can be calculated using the following equation: <MAT> Like equation (<NUM>), xt can be an audio sample value at a time t, while NL can be an estimate of a noise level.

In an alternative implementation, the audio level indicator <NUM> can display a representation of a combination of the log RMS level of a frame of audio samples and a signal-to-noise ratio. For example, this combination can be determined as follows: <MAT> In this equation, α and β can be variables that can scale the background noise and signal-to-noise. For example, α can scale the RMS level of a frame of audio samples to represent decibel values (e.g., such that <NUM> db equals a full scale RMS level of a frame of audio). β can used to scale a signal-to-noise ratio in a similar fashion.

In some implementations, one or more of the background noise level, signal-to-noise ratio, or a combination of the two can be displayed on the graphical user interface <NUM>. For example, one or more of these measures may be displayed on the screen in different colors or in different areas of the screen. In some cases, one of these measures may be superimposed on one of the others. For example, data representing a signal-to-noise ratio may be superimposed on data representing a background noise level.

<FIG> also illustrates an example graphical user interface that includes visual waveform indicator <NUM>. The illustrative visual waveform indicator <NUM> can show a captured audio signal to a user. The waveform may, in some cases, be a stylized representation of the captured audio that represents an envelope of the speech waveform. In other cases, the waveform may represent a sampled version of the analog audio waveform.

The illustrative waveform may permit the user to recognize when a device has failed to record audio. For example, after a user has spoken an voice command, the application can show a waveform that represents the captured audio. If the waveform is a flat line, this may indicate that no audio was recorded.

<FIG> illustrates an example graphical user interface in different operating conditions. In some cases, it may be useful to adjust the options for interacting with a mobile device based on a level of background noise. For example, a user may want to enter voice commands into a mobile device. Depending on the background noise level, the user may need to hold the device close to his mouth for voice commands to be recognized by the device. However, in quieter situations the user may be able to hold the device at arm's length and enter voice commands. The illustrative graphical user interface may present a user with an interaction option based on the probability that the device can correctly recognize a voice command given a detected level of background noise. For example, as shown in <FIG>, in quiet conditions a graphical user interface may present a voice search option, represented by the graphical voice search button <NUM>. In circumstances where the background noise level is high, the voice search button <NUM> can be removed and a message indicating that the mobile device should be placed closer to the user's mouth may be displayed, as shown by the right-most image of the graphical user interface <NUM>. By holding the device closer to the user (e.g., holding the device in telephone pose), speech power may be increased by <NUM>-<NUM> decibels, making correct speech recognition more likely.

<FIG> and <FIG> are flow charts of an example processes <NUM> and <NUM> for background noise based mode selection. The processes <NUM> and <NUM> may be performed, for example, by a system such as the system shown in <FIG> and, for clarity of presentation, the description that follows uses that system as the basis of an example for describing the processes. However, another system, or combination of systems, may be used to perform the processes <NUM> and <NUM>.

<FIG> illustrates an example process <NUM> for background noise based mode selection. The example process <NUM> being at step <NUM> where environmental noise and/or a signal-to-noise ratio are estimated. For example, environmental noise and signal-to-noise ratio can be calculated using equations (<NUM>) and (<NUM>) above. At step <NUM> it is determined whether the environmental (i.e., background) noise and/or a signal-to-noise ratio are above a background noise level threshold value. For example, in one implementation, a device <NUM> may send an acoustic signal, as well as noise and speech level estimates and other environment-related parameters to a server. The server may determine whether the estimated noise and speech level estimates are above a background noise level threshold value. The background noise level threshold value may be based on prior noise and speech level estimates, environment-related parameters, and acoustic level signals sent to the server.

In some cases, the device <NUM> can correlate a particular noise level or type of environmental sound to recognition accuracy. For example, a noise level (NL) of 40dB fan noise may correspond to a word error rate (WER) of <NUM>%, while the WER might be <NUM>% when the noise is 70dB (assuming the user speaks at 80dB on average). These values may be transmitted to a server (e.g., remote device <NUM>) that can collect statistics to make a table from NL to WER.

Some noise types may be worse than others. For example, 50dB cafeteria noise might have the same WER as 70dB fan noise. The device <NUM> can perform environment characterization of this type by sending the audio to a server (such as remote device <NUM>) for mode determination.

If the background noise and/or signal-to-noise ratio is above the background level threshold, the process proceeds to step <NUM> where a voice search button is displayed as shown in <FIG>. If not, a dialog box or message may be displayed advising a user to use the device <NUM> in phone position at step <NUM>. Regardless, the method returns to <NUM> after step <NUM> or step <NUM>.

<FIG> shows an illustrative method <NUM> of background noise level estimation. The method <NUM> begins at step <NUM> where an RMS level of an audio sample is determined. For example, a microphone <NUM> can be used to capture a frame of audio signals (e.g., <NUM> milliseconds of audio) from the environment surrounding the mobile device <NUM>. The RMS level of the frame can be determined according to equation (<NUM>) above.

Optionally, at step <NUM> noise and speech levels may be initialized. For instance, if noise and speech levels have not already been set (as may be the case when the method <NUM> is executed for the first time) noise and speech levels may be initialized using an RMS level of an audio sample. In an illustrative example, the noise and speech levels may be set using the following equations: <MAT> <MAT> In equations (<NUM>) and (<NUM>), RMS can be an RMS level of an audio sample and α is a ratio of a previous estimate of noise or speech and a current estimate of noise or speech. This ratio may be initially set to zero and increase to <MAT>, where k is a number of time steps in an initial adaptation period.

At step <NUM>, a noise level may be updated. For example, a noise level can be compared with a RMS level of an audio sample, and the noise level can be adjusted according to the following equation: <MAT> Like equation (<NUM>), RMS can be an RMS level of an audio sample. In some cases, the sum of UpdateRateNL and UpdateRateRMS can equal one. If the noise level is less than an RMS level of an audio sample, UpdateRateNL may be. <NUM>, while UpdateRateRMS may be. If the noise level is greater than the RMS level of an audio sample, the noise level may be adjusted using equation (<NUM>), but UpdateRateNL may be. <NUM>, and UpdateRateRMS may be.

At step <NUM>, a speech level may be updated. For example, a speech level can be compared with an RMS level of an audio sample, and the speech sample can be adjusted according to the following equation: <MAT>.

If the speech level is greater than an RMS level of the audio sample, UpdateRateSL may equal. <NUM> and UpdateRateRMS can equal. If the speech level is less than an RMS level of the audio sample, UpdateRateSL may equal. <NUM> and UpdateRateRMS can equal. After the speech level is updated, the method <NUM> may return to step <NUM>.

In some implementations, other background noise level estimation methods may be used. For example, the methods disclosed in the following papers, which are herein incorporated by reference, may be used:.

Referring now to <FIG>, the exterior appearance of an exemplary device <NUM> that implements the multisensory speech detection methods described above is illustrated. In more detail, the hardware environment of the device <NUM> includes a display <NUM> for displaying text, images, and video to a user; a keyboard <NUM> for entering text data and user commands into the device <NUM>; a pointing device <NUM> for pointing, selecting, and adjusting objects displayed on the display <NUM>; an antenna <NUM>; a network connection <NUM>; a camera <NUM>; a microphone <NUM>; and a speaker <NUM>. Although the device <NUM> shows an external antenna <NUM>, the device <NUM> can include an internal antenna, which is not visible to the user.

The display <NUM> can display video, graphics, images, and text that make up the user interface for the software applications used by the device <NUM>, and the operating system programs used to operate the device <NUM>. Among the possible elements that may be displayed on the display <NUM> are a new mail indicator <NUM> that alerts a user to the presence of a new message; an active call indicator <NUM> that indicates that a telephone call is being received, placed, or is occurring; a data standard indicator <NUM> that indicates the data standard currently being used by the device <NUM> to transmit and receive data; a signal strength indicator <NUM> that indicates a measurement of the strength of a signal received by via the antenna <NUM>, such as by using signal strength bars; a battery life indicator <NUM> that indicates a measurement of the remaining battery life; or a clock <NUM> that outputs the current time.

The display <NUM> may also show application icons representing various applications available to the user, such as a web browser application icon <NUM>, a phone application icon <NUM>, a search application icon <NUM>, a contacts application icon <NUM>, a mapping application icon <NUM>, an email application icon <NUM>, or other application icons. In one example implementation, the display <NUM> is a quarter video graphics array (QVGA) thin film transistor (TFT) liquid crystal display (LCD), capable of <NUM>-bit or better color.

A user uses the keyboard (or "keypad") <NUM> to enter commands and data to operate and control the operating system and applications that provide for multisensory speech detection. The keyboard <NUM> includes standard keyboard buttons or keys associated with alphanumeric characters, such as keys <NUM> and <NUM> that are associated with the alphanumeric characters "Q" and "W" when selected alone, or are associated with the characters "*" and "<NUM>" when pressed in combination with key <NUM>. A single key may also be associated with special characters or functions, including unlabeled functions, based upon the state of the operating system or applications invoked by the operating system. For example, when an application calls for the input of a numeric character, a selection of the key <NUM> alone may cause a "<NUM>" to be input.

In addition to keys traditionally associated with an alphanumeric keypad, the keyboard <NUM> also includes other special function keys, such as an establish call key <NUM> that causes a received call to be answered or a new call to be originated; a terminate call key <NUM> that causes the termination of an active call; a drop down menu key <NUM> that causes a menu to appear within the display <NUM>; a backward navigation key <NUM> that causes a previously accessed network address to be accessed again; a favorites key <NUM> that causes an active web page to be placed in a bookmarks folder of favorite sites, or causes a bookmarks folder to appear; a home page key <NUM> that causes an application invoked on the device <NUM> to navigate to a predetermined network address; or other keys that provide for multiple-way navigation, application selection, and power and volume control.

The user uses the pointing device <NUM> to select and adjust graphics and text objects displayed on the display <NUM> as part of the interaction with and control of the device <NUM> and the applications invoked on the device <NUM>. The pointing device <NUM> is any appropriate type of pointing device, and may be a joystick, a trackball, a touch-pad, a camera, a voice input device, a touch screen device implemented in combination with the display <NUM>, or any other input device.

The antenna <NUM>, which can be an external antenna or an internal antenna, is a directional or omni-directional antenna used for the transmission and reception of radiofrequency (RF) signals that implement point-to-point radio communication, wireless local area network (LAN) communication, or location determination. The antenna <NUM> may facilitate point-to-point radio communication using the Specialized Mobile Radio (SMR), cellular, or Personal Communication Service (PCS) frequency bands, and may implement the transmission of data using any number or data standards. For example, the antenna <NUM> may allow data to be transmitted between the device <NUM> and a base station using technologies such as Wireless Broadband (WiBro), Worldwide Interoperability for Microwave ACCess (WiMAX), 10GPP Long Term Evolution (LTE), Ultra Mobile Broadband (UMB), High Performance Radio Metropolitan Network (HIPERMAN), iBurst or High Capacity Spatial Division Multiple Access (HC-SDMA), High Speed OFDM Packet Access (HSOPA), High-Speed Packet Access (HSPA), HSPA Evolution, HSPA+, High Speed Upload Packet Access (HSUPA), High Speed Downlink Packet Access (HSDPA), Generic Access Network (GAN), Time Division-Synchronous Code Division Multiple Access (TD-SCDMA), Evolution-Data Optimized (or Evolution-Data Only)(EVDO), Time Division-Code Division Multiple Access (TD-CDMA), Freedom Of Mobile Multimedia Access (FOMA), Universal Mobile Telecommunications System (UMTS), Wideband Code Division Multiple Access (W-CDMA), Enhanced Data rates for GSM Evolution (EDGE), Enhanced GPRS (EGPRS), Code Division Multiple Access-<NUM> (CDMA2000), Wideband Integrated Dispatch Enhanced Network (WiDEN), High-Speed Circuit-Switched Data (HSCSD), General Packet Radio Service (GPRS), Personal Handy-Phone System (PHS), Circuit Switched Data (CSD), Personal Digital Cellular (PDC), CDMAone, Digital Advanced Mobile Phone System (DAMPS), Integrated Digital Enhanced Network (IDEN), Global System for Mobile communications (GSM), DataTAC, Mobitex, Cellular Digital Packet Data (CDPD), Hicap, Advanced Mobile Phone System (AMPS), Nordic Mobile Phone (NMP), Autoradiopuhelin (ARP), Autotel or Public Automated Land Mobile (PALM), Mobiltelefonisystem D (MTD), Offentlig Landmobil Telefoni (OLT), Advanced Mobile Telephone System (AMTS), Improved Mobile Telephone Service (IMTS), Mobile Telephone System (MTS), Push-To-Talk (PTT), or other technologies. Communication via W-CDMA, HSUPA, GSM, GPRS, and EDGE networks may occur, for example, using a QUALCOMM MSM7200A chipset with an QUALCOMM RTR6285™ transceiver and PM7540™ power management circuit.

The wireless or wired computer network connection <NUM> may be a modem connection, a local-area network (LAN) connection including the Ethernet, or a broadband wide-area network (WAN) connection such as a digital subscriber line (DSL), cable high-speed internet connection, dial-up connection, T-<NUM> line, T-<NUM> line, fiber optic connection, or satellite connection. The network connection <NUM> may connect to a LAN network, a corporate or government WAN network, the Internet, a telephone network, or other network. The network connection <NUM> uses a wired or wireless connector. Example wireless connectors include, for example, an INFRARED DATA ASSOCIATION (IrDA) wireless connector, a Wi-Fi wireless connector, an optical wireless connector, an INSTITUTE OF ELECTRICALAND ELECTRONICS ENGINEERS (IEEE) Standard <NUM> wireless connector, a BLUETOOTH wireless connector (such as a BLUETOOTH version <NUM> or <NUM> connector), a near field communications (NFC) connector, an orthogonal frequency division multiplexing (OFDM) ultra wide band (UWB) wireless connector, a time-modulated ultra wide band (TM-UWB) wireless connector, or other wireless connector. Example wired connectors include, for example, a IEEE-<NUM> FIREWIRE connector, a Universal Serial Bus (USB) connector (including a mini-B USB interface connector), a serial port connector, a parallel port connector, or other wired connector. In another implementation, the functions of the network connection <NUM> and the antenna <NUM> are integrated into a single component.

The camera <NUM> allows the device <NUM> to capture digital images, and may be a scanner, a digital still camera, a digital video camera, other digital input device. In one example implementation, the camera <NUM> is a <NUM> mega-pixel (MP) camera that utilizes a complementary metal-oxide semiconductor (CMOS).

The microphone <NUM> allows the device <NUM> to capture sound, and may be an omni-directional microphone, a unidirectional microphone, a bi-directional microphone, a shotgun microphone, or other type of apparatus that converts sound to an electrical signal. The microphone <NUM> may be used to capture sound generated by a user, for example when the user is speaking to another user during a telephone call via the device <NUM>. Conversely, the speaker <NUM> allows the device to convert an electrical signal into sound, such as a voice from another user generated by a telephone application program, or a ring tone generated from a ring tone application program. Furthermore, although the device <NUM> is illustrated in <FIG> as a handheld device, in further implementations the device <NUM> may be a laptop, a workstation, a midrange computer, a mainframe, an embedded system, telephone, desktop PC, a tablet computer, a PDA, or other type of computing device.

<FIG> is a block diagram illustrating an internal architecture <NUM> of the device <NUM>. The architecture includes a central processing unit (CPU) <NUM> where the computer instructions that comprise an operating system or an application are processed; a display interface <NUM> that provides a communication interface and processing functions for rendering video, graphics, images, and texts on the display <NUM>, provides a set of built-in controls (such as buttons, text and lists), and supports diverse screen sizes; a keyboard interface <NUM> that provides a communication interface to the keyboard <NUM>; a pointing device interface <NUM> that provides a communication interface to the pointing device <NUM>; an antenna interface <NUM> that provides a communication interface to the antenna <NUM>; a network connection interface <NUM> that provides a communication interface to a network over the computer network connection <NUM>; a camera interface <NUM> that provides a communication interface and processing functions for capturing digital images from the camera <NUM>; a sound interface <NUM> that provides a communication interface for converting sound into electrical signals using the microphone <NUM> and for converting electrical signals into sound using the speaker <NUM>; a random access memory (RAM) <NUM> where computer instructions and data are stored in a volatile memory device for processing by the CPU <NUM>; a read-only memory (ROM) <NUM> where invariant low-level systems code or data for basic system functions such as basic input and output (I/O), startup, or reception of keystrokes from the keyboard <NUM> are stored in a non-volatile memory device; a storage medium <NUM> or other suitable type of memory (e.g. such as RAM, ROM, programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic disks, optical disks, floppy disks, hard disks, removable cartridges, flash drives), where the files that comprise an operating system <NUM>, application programs <NUM> (including, for example, a web browser application, a widget or gadget engine, and or other applications, as necessary) and data files <NUM> are stored; a navigation module <NUM> that provides a real-world or relative position or geographic location of the device <NUM>; a power source <NUM> that provides an appropriate alternating current (AC) or direct current (DC) to power components; and a telephony subsystem <NUM> that allows the device <NUM> to transmit and receive sound over a telephone network. The constituent devices and the CPU <NUM> communicate with each other over a bus <NUM>.

The CPU <NUM> can be one of a number of computer processors. In one arrangement, the computer CPU <NUM> is more than one processing unit. The RAM <NUM> interfaces with the computer bus <NUM> so as to provide quick RAM storage to the CPU <NUM> during the execution of software programs such as the operating system application programs, and device drivers. More specifically, the CPU <NUM> loads computer-executable process steps from the storage medium <NUM> or other media into a field of the RAM <NUM> in order to execute software programs. Data is stored in the RAM <NUM>, where the data is accessed by the computer CPU <NUM> during execution. In one example configuration, the device <NUM> includes at least 128MB of RAM, and 256MB of flash memory.

The storage medium <NUM> itself may include a number of physical drive units, such as a redundant array of independent disks (RAID), a floppy disk drive, a flash memory, a USB flash drive, an external hard disk drive, thumb drive, pen drive, key drive, a High-Density Digital Versatile Disc (HD-DVD) optical disc drive, an internal hard disk drive, a Blu-Ray optical disc drive, or a Holographic Digital Data Storage (HDDS) optical disc drive, an external mini-dual in-line memory module (DIMM) synchronous dynamic random access memory (SDRAM), or an external micro-DIMM SDRAM. Such computer readable storage media allow the device <NUM> to access computer-executable process steps, application programs and the like, stored on removable and non-removable memory media, to off-load data from the device <NUM>, or to upload data onto the device <NUM>.

A computer program product is tangibly embodied in storage medium <NUM>, a machine-readable storage medium. The computer program product includes instructions that, when read by a machine, operate to cause a data processing apparatus to store image data in the mobile device. In some embodiments, the computer program product includes instructions that perform multisensory speech detection.

The operating system <NUM> may be a LINUX-based operating system such as the GOOGLE mobile device platform; APPLE MAC OS X; MICROSOFT WINDOWS NT/WINDOWS <NUM>/WINDOWS XP/WINDOWS MOBILE; a variety of UNIX-flavored operating systems; or a proprietary operating system for computers or embedded systems. The application development platform or framework for the operating system <NUM> may be: BINARY RUNTIME ENVIRONMENT FOR WIRELESS (BREW); JAVA Platform, Micro Edition (JAVA ME) or JAVA <NUM> Platform, Micro Edition (J2ME) using the SUN MICROSYSTEMS JAVASCRIPT programming language; PYTHON™, FLASH LITE, or MICROSOFT. NET Compact, or another appropriate environment.

The device stores computer-executable code for the operating system <NUM>, and the application programs <NUM> such as an email, instant messaging, a video service application, a mapping application, word processing, spreadsheet, presentation, gaming, mapping, web browsing, JAVASCRIPT engine, or other applications. For example, one implementation may allow a user to access the GOOGLE GMAIL email application, the GOOGLE TALK instant messaging application, a YOUTUBE video service application, a GOOGLE MAPS or GOOGLE EARTH mapping application, or a GOOGLE PICASA imaging editing and presentation application. The application programs <NUM> may also include a widget or gadget engine, such as a TAFRI™ widget engine, a MICROSOFT gadget engine such as the WINDOWS SIDEBAR gadget engine or the KAPSULES™ gadget engine, a YAHOO! widget engine such as the KONFABULTOR™ widget engine, the APPLE DASHBOARD widget engine, the GOOGLE gadget engine, the KLIPFOLIO widget engine, an OPERA™ widget engine, the WIDSETS™ widget engine, a proprietary widget or gadget engine, or other widget or gadget engine that provides host system software for a physically-inspired applet on a desktop.

Although it is possible to provide for multisensory speech detection using the above-described implementation, it is also possible to implement the functions according to the present disclosure as a dynamic link library (DLL), or as a plug-in to other application programs such as an Internet web-browser such as the FOXFIRE web browser, the APPLE SAFARI web browser or the MICROSOFT INTERNET EXPLORER web browser.

The navigation module <NUM> may determine an absolute or relative position of the device, such as by using the Global Positioning System (GPS) signals, the GLObal NAvigation Satellite System (GLONASS), the Galileo positioning system, the Beidou Satellite Navigation and Positioning System, an inertial navigation system, a dead reckoning system, or by accessing address, internet protocol (IP) address, or location information in a database. The navigation module <NUM> may also be used to measure angular displacement, orientation, or velocity of the device <NUM>, such as by using one or more accelerometers.

<FIG> is a block diagram illustrating exemplary components of the operating system <NUM> used by the device <NUM>, in the case where the operating system <NUM> is the GOOGLE mobile device platform. The operating system <NUM> invokes multiple processes, while ensuring that the associated phone application is responsive, and that wayward applications do not cause a fault (or "crash") of the operating system. Using task switching, the operating system <NUM> allows for the switching of applications while on a telephone call, without losing the state of each associated application. The operating system <NUM> may use an application framework to encourage reuse of components, and provide a scalable user experience by combining pointing device and keyboard inputs and by allowing for pivoting. Thus, the operating system <NUM> can provide a rich graphics system and media experience, while using an advanced, standards-based web browser.

The operating system <NUM> can generally be organized into six components: a kernel <NUM>, libraries <NUM>, an operating system runtime <NUM>, application libraries <NUM>, system services <NUM>, and applications <NUM>. The kernel <NUM> includes a display driver <NUM> that allows software such as the operating system <NUM> and the application programs <NUM> to interact with the display <NUM> via the display interface <NUM>, a camera driver <NUM> that allows the software to interact with the camera <NUM>; a BLUETOOTH driver <NUM>; a M-Systems driver <NUM>; a binder (IPC) driver <NUM>, a USB driver <NUM> a keypad driver <NUM> that allows the software to interact with the keyboard <NUM> via the keyboard interface <NUM>; a WiFi driver <NUM>; audio drivers <NUM> that allow the software to interact with the microphone <NUM> and the speaker <NUM> via the sound interface <NUM>; and a power management component <NUM> that allows the software to interact with and manage the power source <NUM>.

The BLUETOOTH driver, which in one implementation is based on the BlueZ BLUETOOTH stack for LINUX-based operating systems, provides profile support for headsets and hands-free devices, dial-up networking, personal area networking (PAN), or audio streaming (such as by Advance Audio Distribution Profile (A2DP) or Audio/Video Remote Control Profile (AVRCP). The BLUETOOTH driver provides JAVA bindings for scanning, pairing and unpairing, and service queries.

The libraries <NUM> include a media framework <NUM> that supports standard video, audio and still-frame formats (such as Moving Picture Experts Group (MPEG)-<NUM>, H. <NUM>, MPEG-<NUM> Audio Layer-<NUM> (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR), Joint Photographic Experts Group (JPEG), and others) using an efficient JAVA Application Programming Interface (API) layer; a surface manager <NUM>; a simple graphics library (SGL) <NUM> for two-dimensional application drawing; an Open Graphics Library for Embedded Systems (OpenGL ES) <NUM> for gaming and three-dimensional rendering; a C standard library (LIBC) <NUM>; a LIBWEBCORE library <NUM>; a FreeType library <NUM>; an SSL <NUM>; and an SQLite library <NUM>.

The operating system runtime <NUM> includes core JAVA libraries <NUM>, and a Dalvik virtual machine <NUM>. The Dalvik virtual machine <NUM> is a custom, virtual machine that runs a customized file format (.

The operating system <NUM> can also include Mobile Information Device Profile (MIDP) components such as the MIDP JAVA Specification Requests (JSRs) components, MIDP runtime, and MIDP applications as shown in <FIG>. The MIDP components can support MIDP applications running on the device <NUM>.

With regard to graphics rendering, a system-wide composer manages surfaces and a frame buffer and handles window transitions, using the OpenGL ES <NUM> and two-dimensional hardware accelerators for its compositions.

The Dalvik virtual machine <NUM> may be used with an embedded environment, since it uses runtime memory very efficiently, implements a CPU-optimized bytecode interpreter, and supports multiple virtual machine processes per device. The custom file format (. DEX) is designed for runtime efficiency, using a shared constant pool to reduce memory, read-only structures to improve cross-process sharing, concise, and fixed-width instructions to reduce parse time, thereby allowing installed applications to be translated into the custom file formal at build-time. The associated bytecodes are designed for quick interpretation, since register-based instead of stack-based instructions reduce memory and dispatch overhead, since using fixed width instructions simplifies parsing, and since the <NUM>-bit code units minimize reads.

The application libraries <NUM> include a view system <NUM>, a resource manager <NUM>, and content providers <NUM>. The system services <NUM> includes a status bar <NUM>; an application launcher <NUM>; a package manager <NUM> that maintains information for all installed applications; a telephony manager <NUM> that provides an application level JAVA interface to the telephony subsystem <NUM>; a notification manager <NUM> that allows all applications access to the status bar and on-screen notifications; a window manager <NUM> that allows multiple applications with multiple windows to share the display <NUM>; and an activity manager <NUM> that runs each application in a separate process, manages an application life cycle, and maintains a cross-application history.

The applications <NUM> include a home application <NUM>, a dialer application <NUM>, a contacts application <NUM>, a browser application <NUM>, and a multispeech detection application <NUM>.

The telephony manager <NUM> provides event notifications (such as phone state, network state, Subscriber Identity Module (SIM) status, or voicemail status), allows access to state information (such as network information, SIM information, or voicemail presence), initiates calls, and queries and controls the call state. The browser application <NUM> renders web pages in a full, desktop-like manager, including navigation functions. Furthermore, the browser application <NUM> allows single column, small screen rendering, and provides for the embedding of HTML views into other applications.

<FIG> is a block diagram illustrating exemplary processes implemented by the operating system kernel <NUM>. Generally, applications and system services run in separate processes, where the activity manager <NUM> runs each application in a separate process and manages the application life cycle. The applications run in their own processes, although many activities or services can also run in the same process. Processes are started and stopped as needed to run an application's components, and processes may be terminated to reclaim resources. Each application is assigned its own process, whose name is the application's package name, and individual parts of an application can be assigned another process name.

Some processes can be persistent. For example, processes associated with core system components such as the surface manager <NUM>, the window manager <NUM>, or the activity manager <NUM> can be continuously executed while the device <NUM> is powered. Additionally, some application-specific process can also be persistent. For example, processes associated with the dialer application <NUM>, may also be persistent.

The processes implemented by the operating system kernel <NUM> may generally be categorized as system services processes <NUM>, dialer processes <NUM>, browser processes <NUM>, and maps processes <NUM>. The system services processes <NUM> include status bar processes <NUM> associated with the status bar <NUM>; application launcher processes <NUM> associated with the application launcher <NUM>; package manager processes <NUM> associated with the package manager <NUM>; activity manager processes <NUM> associated with the activity manager <NUM>; resource manager processes <NUM> associated with a resource manager <NUM> that provides access to graphics, localized strings, and XML layout descriptions; notification manger processes <NUM> associated with the notification manager <NUM>; window manager processes <NUM> associated with the window manager <NUM>; core JAVA libraries processes <NUM> associated with the core JAVA libraries <NUM>; surface manager processes <NUM> associated with the surface manager <NUM>; Dalvik virtual machine processes <NUM> associated with the Dalvik virtual machine <NUM>, LIBC processes <NUM> associated with the LIBC library <NUM>; and multispeech detection processes <NUM> associated with the multispeech detection application <NUM>.

The dialer processes <NUM> include dialer application processes <NUM> associated with the dialer application <NUM>; telephony manager processes <NUM> associated with the telephony manager <NUM>; core JAVA libraries processes <NUM> associated with the core JAVA libraries <NUM>; Dalvik virtual machine processes <NUM> associated with the Dalvik Virtual machine <NUM>; and LIBC processes <NUM> associated with the LIBC library <NUM>. The browser processes <NUM> include browser application processes <NUM> associated with the browser application <NUM>; core JAVA libraries processes <NUM> associated with the core JAVA libraries <NUM>; Dalvik virtual machine processes <NUM> associated with the Dalvik virtual machine <NUM>; LIBWEBCORE processes <NUM> associated with the LIBWEBCORE library <NUM>; and LIBC processes <NUM> associated with the LIBC library <NUM>.

The maps processes <NUM> include maps application processes <NUM>, core JAVA libraries processes <NUM>, Dalvik virtual machine processes <NUM>, and LIBC processes <NUM>. Notably, some processes, such as the Dalvik virtual machine processes, may exist within one or more of the systems services processes <NUM>, the dialer processes <NUM>, the browser processes <NUM>, and the maps processes <NUM>.

<FIG> shows an example of a generic computer device <NUM> and a generic mobile computer device <NUM>, which may be used with the techniques described here. Computing device <NUM> is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. Computing device <NUM> is intended to represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit the implementations described and/or claimed in this document.

Also, multiple computing devices <NUM> may be connected, with each device providing portions of the necessary operations (e.g., as a server bank, a group of blade servers, or a multiprocessor system).

The information carrier is a computer- or machine-readable medium, such as the memory <NUM>, the storage device <NUM>, memory on processor <NUM>, or a propagated signal.

The information carrier is a computer- or machine-readable medium, such as the memory <NUM>, expansion memory <NUM>, memory on processor <NUM>, or a propagated signal that may be received, for example, over transceiver <NUM> or external interface <NUM>.

Claim 1:
A method comprising:
determining, at data processing hardware (<NUM>) of a mobile device (<NUM>), whether a record button (<NUM>) has been pressed, the record button (<NUM>) being a button of the mobile device (<NUM>, <NUM>) that allows a user to initiate or end speech recording by pressing the button (<NUM>);
determining a gesture using a gesture classifier, wherein the gesture classifier classifies movement of the device;
detecting a pose of the device using a pose identifier;
if a record button press is not detected then determining whether a record gesture has been detected;
in response to determining that the record button has been pressed or that a record gesture has been detected:
initiating, by the data processing hardware, execution of an audio recording process using a microphone (<NUM>) of the mobile device (<NUM>);
notifying, by the data processing hardware (<NUM>), a user of the mobile device when execution of the audio recording process starts; and
loading device pose settings into an endpointer;
receiving, at the data processing hardware (<NUM>), a speech utterance from the user captured by the microphone (<NUM>) during execution of the audio recording process;
detecting an end-of-speech input by determining, using the endpointer (<NUM>), whether speech has ended using the output of the pose identifier and a speech energy threshold, and if it is determined that the end of speech input has been detected, ceasing recording and providing an end of input (EOI) display indicating that recording has ended;
and
using the recorded speech as input to an application running on the device (<NUM>) or running on an external device; wherein
notifying the user of the mobile device (<NUM>) when execution of the audio recording process starts comprises:
generating a visual notification (<NUM>) that indicates to the user when execution of the audio recording process starts; and
displaying, by the data processing hardware, the visual notification (<NUM>) on a user interface (<NUM>) of the mobile device (<NUM>); wherein
the visual notification (<NUM>) comprises a waveform graphic that represents the captured audio.