Geotagged environmental audio for enhanced speech recognition accuracy

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for enhancing speech recognition accuracy. In one aspect, a method includes receiving geotagged audio signals that correspond to environmental audio recorded by multiple mobile devices in multiple geographic locations, receiving an audio signal that corresponds to an utterance recorded by a particular mobile device, determining a particular geographic location associated with the particular mobile device, generating a noise model for the particular geographic location using a subset of the geotagged audio signals, where noise compensation is performed on the audio signal that corresponds to the utterance using the noise model that has been generated for the particular geographic location.

BACKGROUND

This specification relates to speech recognition.

As used by this specification, a “search query” includes one or more query terms that a user submits to a search engine when the user requests the search engine to execute a search query, where a “term” or a “query term” includes one or more whole or partial words, characters, or strings of characters. Among other things, a “result” (or a “search result”) of the search query includes a Uniform Resource Identifier (URI) that references a resource that the search engine determines to be responsive to the search query. The search result may include other things, such as a title, preview image, user rating, map or directions, description of the corresponding resource, or a snippet of text that has been automatically or manually extracted from, or otherwise associated with, the corresponding resource.

Among other approaches, a user may enter query terms of a search query by typing on a keyboard or, in the context of a voice query, by speaking the query terms into a microphone of a mobile device. When submitting a voice query, the microphone of the mobile device may record ambient noises or sounds, or “environmental audio,” in addition to spoken utterances of the user. For example, environmental audio may include background chatter or babble of other people situated around the user, or noises generated by nature (e.g., dogs barking) or man-made objects (e.g., office, airport, or road noise, or construction activity). The environmental audio may partially obscure the voice of the user, making it difficult for an automated speech recognition (“ASR”) engine to accurately recognize spoken utterances.

SUMMARY

In general, one innovative aspect of the subject matter described in this specification may be embodied in methods for adapting, training, selecting or otherwise generating, by an ASR engine, a noise model for a geographic area, and for applying this noise model to “geotagged” audio signals (or “samples,” or “waveforms”) that are received from a mobile device that is located in or near this geographic area. As used by this specification, “geotagged” audio signals refer to signals that have been associated, or “tagged,” with geographical location metadata or geospatial metadata. Among other things, the location metadata may include navigational coordinates, such as latitude and longitude, altitude information, bearing or heading information, or a name or an address associated with the location.

In further detail, the methods include receiving geotagged audio signals that correspond to environmental audio recorded by multiple mobile devices in multiple geographic locations, storing the geotagged audio signals, and generating a noise model for a particular geographic region using a selected subset of the geotagged audio signals. Upon receiving an utterance recorded by a mobile device within or near the same particular geographic area, the ASR engine may perform noise compensation on the audio signal using the noise model that is generated for the particular geographic region, and may perform speech recognition on the noise-compensated audio signal. Notably, the noise model for the particular geographic region may be generated before, during, or after receipt of the utterance.

In general, another innovative aspect of the subject matter described in this specification may be embodied in methods that include the actions of receiving geotagged audio signals that correspond to environmental audio recorded by multiple mobile devices in multiple geographic locations, receiving an audio signal that corresponds to an utterance recorded by a particular mobile device, determining a particular geographic location associated with the particular mobile device, generating a noise model for the particular geographic location using a subset of the geotagged audio signals, where noise compensation is performed on the audio signal that corresponds to the utterance using the noise model that has been generated for the particular geographic location.

These and other embodiments may each optionally include one or more of the following features. In various examples, speech recognition is performed on the utterance using the noise-compensated audio signal; generating the noise model further includes generating the noise model before receiving the audio signal that corresponds to the utterance; generating the noise model further includes generating the noise model after receiving the audio signal that corresponds to the utterance; for each of the geotagged audio signals, a distance between the particular geographic location and a geographic location associated the geotagged audio signal is determined, and the geotagged audio signals that are associated with geographic locations which are within a predetermined distance of the particular geographic location, or that are associated with geographic locations which are among the N closest geographic locations to the particular geographic location, are selected as the subset of the geotagged audio signals; the geotagged audio signals that are associated with the particular geographic location are selected as the subset of the geotagged audio signals; the subset of the geotagged audio signals are selected based on the particular geographic location, and based on context data associated with the utterance; the context data includes data that references a time or a date when the utterance was recorded by the mobile device, data that references a speed or an amount of motion measured by the particular mobile device when the utterance was recorded, data that references settings of the mobile device, or data that references a type of the mobile device; the utterance represents a voice search query, or an input to a digital dictation application or a dialog system; determining the particular geographic location further includes receiving data referencing the particular geographic location from the mobile device; determining the particular geographic location further includes determining a past geographic location or a default geographic location associated with the device; generating the noise model includes training a Gaussian Mixture Model (GMM) using the subset of the geotagged audio signals as a training set; one or more candidate transcriptions of the utterance are generated, a search query is executed using the one or more candidate transcriptions; the received geotagged audio signals are processed to exclude portions of the environmental audio that include voices of users of the multiple mobile devices; the noise model generated for the particular geographic location is selected from among multiple noise models generated for the multiple geographic locations; an area surrounding the particular geographic location is defined, a plurality of noise models associated with geographic locations within the area are selected from among the multiple noise models, a weighted combination of the selected noise models is generated, where the noise compensation is performed using the weighted combination of selected noise models; generating the noise model further includes generating the noise model for the particular geographic location using the subset of the geotagged audio signals and using an environmental audio portion of the audio signal that corresponds to the utterance; and/or an area is defined surrounding the particular geographic location, and the geotagged audio signals recorded within the area are selected as the subset of the geotagged audio signals.

Particular embodiments of the subject matter described in this specification may be implemented to realize one or more of the following advantages. The ASR engine may provide for better noise suppression of the audio signal. Speech recognition accuracy may be improved. Noise models may be generated using environmental audio signals that accurately reflect the actual ambient noise in a geographic area. Speech recognition and noise model generation may be performed at the server side, instead of on the client device, to allow for better process optimization and to increase computational efficiency.

DETAILED DESCRIPTION

FIG. 1is a diagram of an example system100that uses geotagged environmental audio to enhance speech recognition accuracy.FIG. 1also illustrates a flow of data within the system100during states (a) to (i), as well as a user interface158that is displayed on a mobile device104during state (i).

In more detail, the system100includes a server106and an ASR engine108, which are in communication with mobile client communication devices, including mobile devices102and the mobile device104, over one or more networks110. The server106may be a search engine, a dictation engine, a dialogue system, or any other engine or system that uses transcribed speech. The networks110may include a wireless cellular network, a wireless local area network (WLAN) or Wi-Fi network, a Third Generation (3G) or Fourth Generation (4G) mobile telecommunications network, a private network such as an intranet, a public network such as the Internet, or any appropriate combination thereof.

The states (a) through (i) depict a flow of data that occurs when an example process is performed by the system100. The states (a) to (i) may be time-sequenced states, or they may occur in a sequence that is different than the illustrated sequence.

Briefly, according the example process illustrated inFIG. 1, the ASR engine108receives geotagged, environmental audio signals130from the mobile devices102and generates geo-specific noise models112for multiple geographic locations. When an audio signal138that corresponds to an utterance recorded by the mobile device104is received, a particular geographic location associated with the mobile device104(or the user of the mobile device104) is determined. The ASR engine108transcribes the utterance using the geo-specific noise model that matches, or that is otherwise suitable for, the particular geographic location, and one or more candidate transcriptions146are communicated from the ASR engine108to the server106. Where the server106is a search engine, the server106executes one or more search queries using the candidate transcriptions146, generates search results152, and communicates the search results152to the mobile device104for display.

In more detail, during state (a), the mobile devices102communicate geotagged audio signals130that include environmental audio (referred to by this specification as “environmental audio signals”) to the ASR engine108over the networks110. In general, environmental audio may include any ambient sounds that occur (naturally or otherwise) at a particular location. Environmental audio typically excludes the sounds, utterances, or voice of the user of the mobile device.

The device102acommunicates an audio signal130athat has been tagged with metadata132athat references “Location A,” the device102bcommunicates an audio signal130bthat has been tagged with metadata132bthat references “Location B,” and the device102ccommunicates an audio signal130cthat has been tagged with metadata132cthat also references “Location B.” The metadata132may be associated with the audio signals130by mobile devices102, as illustrated, or the metadata may be associated with the audio signals130by the ASR engine108or by another server after inferring a location of a mobile device102(or of the user of the mobile device102).

The environmental audio signals130may each include a two-second (or more) snippet of relatively high quality audio, such as sixteen kilohertz lossless audio signals. The environmental audio signals130may be associated with metadata that references the geographic location of the respective mobile device102when the environmental audio was recorded, captured or otherwise obtained.

The environmental audio signals130may be manually uploaded from the mobile devices102to the ASR engine108. For instance, environmental audio signals130may be generated and communicated in conjunction with the generation and communication of images to a public image database or repository. Alternatively, for users who opt to participate, environmental audio signals130may be automatically obtained and communicated from the mobile devices102to the ASR engine108without requiring an explicit, user actuation before each environmental audio signal is communicated to the ASR engine108.

The metadata132may describe locations in any number of different formats or levels of detail or granularity. For example, the metadata132amay include a latitude and longitude associated with the then-present location of the mobile device102a, and the metadata132cmay include an address or geographic region associated with the then-present location of the mobile device102c. Furthermore, since the mobile device102bis illustrated as being in a moving vehicle, the metadata132bmay describe a path of the vehicle (e.g., including a start point and an end point, and motion data). Additionally, the metadata132may describe locations in terms of location type (e.g., “moving vehicle,” “on a beach,” “in a restaurant,” “in tall building,” “South Asia,” “rural area,” “someplace with construction noise,” “amusement park,” “on a boat,” “indoors,” “underground,” “on a street,” “forest”). A single audio signal may be associated with metadata that describes one or more locations.

The geographic location associated with the audio signal138may instead be described in terms of a bounded area, expressed as a set of coordinates that define the bounded area. Alternatively, the geographic location may be defined using a region identifier, such as a state name or identifier, city name, idiomatic name (e.g., “Central Park”), a country name, or the identifier of arbitrarily defined region (e.g., “cell/region ABC123”).

Before associating a location with the environmental audio signal, the mobile devices102or the ASR engine108may process the metadata to adjust the level of detail of the location information (e.g., to determine a state associated with a particular set of coordinates), or the location information may be discretized (e.g., by selecting a specific point along the path, or a region associated with the path). The level of detail of the metadata may also be adjusted by specifying or adding location type metadata, for example by adding an “on the beach” tag to an environmental audio signal whose associated geographic coordinates are associated with a beach location, or by adding a “someplace with lots of people” tag to an environmental audio signal that includes the sounds of multiple people talking in the background.

During state (b), the ASR engine108receives the geotagged environmental audio signals130from the mobile devices102, and stores the geotagged audio signals (or portions thereof) in the collection114of environmental audio signals, in the data store111. As described below, the collection is used for training, adapting, or otherwise generating one or more geographic location-specific (or “geo-specific”) noise models112.

Because environmental audio signals in the collection114should not include users' voices, the ASR engine108may use a voice activity detector to verify that the collection114of environmental audio signals only includes audio signals130that correspond to ambient noise, or to filter out or otherwise identify or exclude audio signals130(or portions of the audio signals130) that include voices of the various users of the mobile devices102.

The collection114of the ambient audio signals stored by the ASR engine108may include hundreds, thousands, millions, or hundreds of millions of environmental audio signals. In the illustrated example, a portion or all of the geo-tagged environmental audio signal130amay be stored in the collection114as the environmental audio signal124, a portion or all of the geo-tagged environmental audio signal130bmay be stored in the collection114as the environmental audio signal126a, and a portion or all of the geotagged environmental audio signal130cmay be stored in the collection114as the environmental audio signal120b.

Storing an environmental audio signal130in the collection may include determining whether a user's voice is encoded in the audio signal130, and determining to store or determining not to store the environmental audio signal130in the collection based on determining that the user's voice is or is not encoded in the audio signal130, respectively. Alternatively, storing an environmental audio signal in the collection may include identifying a portion of the environmental audio signal130that includes the user's voice, altering the environmental audio signal130by removing the portion that includes the user's voice or by associating metadata which references the portion that includes the user's voice, and storing the altered environmental audio signal130in the collection.

Other context data or metadata associated with the environmental audio signals130may be stored in the collection114as well. For example, the environmental audio signals included in the collection114can, in some implementations, include other metadata tags, such as tags that indicate whether background voices (e.g., cafeteria chatter) are present within the environmental audio, tags that identify the date on which a particular environmental audio signal was obtained (e.g., used to determine a sample age), or tags that identify whether a particular environmental audio signal deviates in some way from other environmental audio signals of the collection that were obtained in the same or similar location. In this manner, the collection114of environmental audio signals may optionally be filtered to exclude particular environmental audio signals that satisfy or that do not satisfy particular criteria, such as to exclude particular environmental audio signals that are older than a certain age, or that include background chatter that may identify an individual or otherwise be proprietary or private in nature.

In an additional example, data referencing whether the environmental audio signals of the collection114were manually or automatically uploaded may be tagged in metadata associated with the environmental audio signals. For example, some of the noise models112may be generated using only those environmental audio signals that were automatically uploaded, or that were manually uploaded, or different weightings may be assigned to each category of upload during the generating of the noise models.

Although the environmental audio signals of the collection114have been described as including an explicit tag that identifies a respective geographic location, in other implementations, such as where the association between an audio signal and a geographic location may be derived, the explicit use of a tag is not required. For example, a geographic location may be implicitly associated with an environmental audio signal by processing search logs (e.g., stored with the server106) to determine geographic location information for a particular environmental audio signal. Accordingly, receipt of a geo-tagged environmental audio signals by the ASR engine108may include obtaining an environmental audio signal that does not expressly include a geo-tag, and deriving and associating one or more geo-tags for the environmental audio signal.

During state (c), an audio signal138is communicated from the mobile device104to the ASR engine108over the networks110. Although the mobile device102is illustrated as being different a different device than the mobile devices104, in other implementations the audio signal138is communicated from one of the mobile devices104that provided an geo-tagged environmental audio signal130.

The audio signal138includes an utterance140(“Gym New York”) recorded by the mobile device104(e.g., when the user implicitly or explicitly initiates a voice search query). The audio signal138includes metadata139that references the geographic location “Location B.” In addition to including the utterance140, the audio signal138may also include a snippet of environmental audio, such as a two second snippet of environmental audio that was recorded before or after the utterance140was spoken. While the utterance140is described an illustrated inFIG. 1as a voice query, in other example implementations the utterance may be an voice input to dictation system or to a dialog system.

The geographic location (“Location B”) associated with the audio signal138may be defined using a same or different level of detail as the geographic locations associated with the environmental audio signals included in the collection114. For example, the geographic locations associated with the environmental audio signals included in the collection114may correspond to geographic regions, while the geographic location associated with the audio signal138may correspond to a particular geographic coordinate. Where the level of detail is different, the ASR engine108may process the geographic metadata139or the metadata associated with the environmental audio signals of the collection114to align the level of detail, so that a subset selection process can be performed.

The metadata139may be associated with the audio signal138by the mobile device104(or the user of the mobile device104) based on location information that is current when the utterance140is recorded, and may be communicated with the audio signal138from the mobile device104to the ASR engine108. Alternatively, the metadata may be associated with the audio signal138by the ASR engine108, based on a geographic location that the ASR engine108infers for the mobile device104(or the user of the mobile device104).

The ASR engine108may infer the geographic location using the user's calendar schedule, user preferences (e.g., as stored in a user account of the ASR engine108or the server106, or as communicated from the mobile device104), a default location, a past location (e.g., the most recent location calculated by a GPS module of the mobile device104), information explicitly provided by the user when submitting the voice search query, from the utterances104themselves, triangulation (e.g., WiFi or cell tower triangulation), a GPS module in the mobile device104, or dead reckoning. The metadata139may include accuracy information that specifies an accuracy of the geographic location determination, signifying a likelihood that the mobile device104was actually in the particular geographic location specified by the metadata139at the time when the utterance140was recorded.

Other metadata may also be included with the audio signal138. For example, metadata included with the audio signals may include a location or locale associated with the respective mobile device102. For example, the locale information may describe, among other selectable parameters, a region in which the mobile device102is registered, or the language or dialect of the user of the mobile device102. The speech recognition module118may use this information to select, train, adapt, or otherwise generate noise, speech, acoustic, popularity, or other models that match the context of the mobile device104.

In state (d), the ASR engine108selects a subset of the environmental audio signals in the collection114, and uses a noise model generating module116to train, adapt, or otherwise generate one or more noise models112(e.g., Gaussian Mixture Models (GMMs)) using the subset of the environmental audio signals, for example by using the subset of the environmental audio signals as a training set for the noise model. The subset may include all, or fewer than all of the environmental audio signals in the collection114.

In general, the noise models112, along with speech models, acoustic models, popularity models, and/or other models, are applied to the audio signal138to translate or transcribe the spoken utterance140into one or more textual, candidate transcriptions146, and to generate speech recognition confidence scores to the candidate transcriptions. The noise models, in particular, are used for noise suppression or noise compensation, to enhance the intelligibility of the spoken utterance140to the ASR engine108.

In more detail, the noise model generating module116may generate a noise model120bfor the geographic location (“Location B”) associated with the audio signal138using the collection114of audio signals, specifically the environmental audio signals126aand126bthat were geotagged as having been recorded at or near that geographic location, or at a same or similar type of location. Since the audio signal138is associated with this geographic location (“Location B”), the environmental audio included in the audio signal138itself may be used to generate a noise model for that geographic location, in addition to or instead of the environmental audio signals126aand126b. Similarly, the noise model generating module116may generate a noise model120afor another geographic location (“Location A”), using the environmental audio signal124that was geotagged as having been recorded at or near that other geographic location, or at a same or similar type of location. If the noise model generating module116is configured to select environmental audio signal that were geotagged as having been recorded near the geographic location associated with the audio signal138, and if “Location A” is near “Location B,” the noise model generating module116may generate a noise model120bfor “Location B” also using the environmental audio signal124.

In addition to the geotagged location, other context data associated with the environmental audio signals of the collection114may be used to select the subset of the environmental audio signals to use to generate the noise models112, or to adjust a weight or effect that a particular audio signal is to have upon the generation. For example, the ASR engine108may select a subset of the environmental audio signals in the collection114whose contextual information indicates that they are longer than or shorter than a predetermined period of time, or that they satisfy certain quality or recency criteria. Furthermore, the ASR engine108may select, as the subset, environmental audio signals in the collection114whose contextual information indicates that they were recorded using a mobile device that has a similar audio subsystem as the mobile device104.

Other context data which may be used to select the subset of the environmental audio signals from the collection114may include, in some examples, the time information, date information, data referencing a speed or an amount of motion measured by the particular mobile device during recording, other device sensor data, device state data (e.g., Bluetooth headset, speaker phone, or traditional input method), a user identifier if the user opts to provide one, or information identifying the type or model of mobile device. The context data, for example, may provide an indication of conditions surrounding the recording of the audio signal138.

In one example, context data supplied with the audio signal138by the mobile device104may indicate that the mobile device104is traveling at highway speeds along a path associated with a highway. The ASR108may infer that the audio signal138was recorded within a vehicle, and may select a subset of the environmental audio signals in the collection114that are associated with an “inside moving vehicle” location type. In another example, context data supplied with the audio signal138by the mobile device104may indicate that the mobile device104is in a rural area, and that the utterance140was recorded on a Sunday at 6:00 am. Based on this context data, the ASR108may infer that it accuracy of the speech recognition would not be improved if the subset included environmental audio signals that were recorded in urban areas during rush hour. Accordingly, the context data may be used by the noise model generating module116to filter the collection114of environmental audio signals when generating noise models112, or by the speech recognition module118to select an appropriate noise model112for a particular utterance.

In some implementations, the noise model generating module116may select a weighted combination of the environmental audio signals of the collection114based upon the proximity of the geographic locations associated with the audio signals to the geographic location associated with the audio signal138. The noise model generating module116may also generate the noise models112using environmental audio included in the audio signal138itself, for example environmental audio recorded before or after the utterances were spoken, or during pauses between utterances.

For instance, the noise model generating module116can first determine the quality of the environmental audio signals stored in the collection114relative to the quality of the environmental audio included in the audio signal138, and can choose to generate a noise model using the audio signals stored in the collection114only, using the environmental audio included in the audio signal138only, or any appropriate weighted or unweighted combination thereof. For instance, the noise model generating module116may determine that the audio signal138includes an insignificant amount of environmental audio, or that high quality environmental audio is stored for that particular geographic location in the collection114, and may choose to generate the noise model without using (or giving little weight to) the environmental audio included in the audio signal138.

In some implementations, the noise model generating module116selects, as the subset, the environmental audio signals from the collection114that are associated with the N (e.g., five, twenty, or fifty) closest geographic locations to the geographic location associated with the audio signal138. When the geographic location associated with the audio signal138describes a point or a place (e.g., coordinates), a geometric shape (e.g., a circle or square) may be defined relative to that that geographic location, and the noise model generating module116may select, as the subset, audio signals from the collection114that are associated with geographic regions that are wholly or partially located within the defined geometric shape.

If the geographic location associated with the audio signal138has been defined in terms of a location type (i.e., “on the beach,” “city”), and ASR engine108may select environmental audio signals that are associated with a same or a similar location type, even if the physical geographic locations associated with the selected audio signals are not physically near the geographic location associated with the audio signal138. For instance, a noise model for an audio signal that was recorded on the beach in Florida may be tagged with “on the beach” metadata, and the noise model generating module116may select, as the subset, environmental audio signals from the collection114whose associated metadata indicate that they were also recorded on beaches, despite the fact that they were recorded on beaches in Australia, Hawaii, or in Iceland.

The noise model generating module116may revert to selecting the subset based on matching location types, instead of matching actual, physical geographic locations, if the geographic location associated with the audio signal138does not match (or does not have a high quality match) with any physical geographic location associated with an environmental audio signal of the collection114. Other matching processes, such as clustering algorithms, may be used to match audio signals with environmental audio signals.

In addition to generating general, geo-specific noise models112, the noise model generating module116may generate geo-specific noise models that are targeted or specific to other criteria as well, such as geo-specific noise models that are specific to different device types or times of day. A targeted sub-model may be generated based upon detecting that a threshold criterion has been satisfied, such as determining that a threshold number of environmental audio signals of the collection114refer to the same geographic location, and share another same or similar context (e.g., time of day, day of the week, motion characteristics, device type, etc.).

The noise models112may be generated before, during, or after the utterance140has been received. For example, multiple environmental audio signals, incoming from a same or similar location as the utterance140, may be processed in parallel with the processing of the utterance, and may be used to generate noise models112in real time or near real time, to better approximate the live noise conditions surrounding the mobile device104.

In state (e), the speech recognition module118of the ASR engine108performs noise compensation on the audio signal138using the geo-specific noise model120bfor the geographic location associated with the audio signal138, to enhance the accuracy of the speech recognition, and subsequently performs the speech recognition on the noise-compensated audio signal. When the audio signal138includes metadata that describes a device type of the mobile device104, the ASR engine108may apply a noise model122that is specific to both the geographic location associated with the audio signal, and to the device type of the mobile device104. The speech recognition module118may generate one or more candidate transcriptions146that match the utterance encoded in the audio signal138, and speech recognition confidence values for the candidate transcriptions.

During state (f), one or more of the candidate transcriptions146generated by the speech recognition module118are communicated from the ASR engine108to the server106. Where the server106is a search engine, the candidate transcriptions may be used as candidate query terms, to execute one or more search queries. The ASR engine108may rank the candidate transcriptions146by their respective speech recognition confidence scores before transmitting them to the server106. By transcribing spoken utterances and providing candidate transcriptions to the server106, the ASR engine108may provide a voice search query capability, a dictation capability, or a dialogue system capability to the mobile device104.

The server106may execute one or more search queries using the candidate query terms, generates a file152that references search results160. The server106, in some examples, may include a web search engine used to find references within the Internet, a phone book type search engine used to find businesses or individuals, or another specialized search engine (e.g., a search engine that provides references to entertainment listings such as restaurants and movie theater information, medical and pharmaceutical information, etc.).

During state (h), the server106provides the file152that references the search results160to the mobile device104. The file152may be a markup language file, such as an eXtensible Markup Language (XML) or HyperText Markup Language (HTML) file.

During state (i), the mobile device104displays the search results160on a user interface158. Specifically, the user interface includes a search box157that displays the candidate query term with the highest speech recognition confidence score (“Gym New York”), an alternate query term suggestion region159that displays another of the candidate query term that may have been intended by the utterance140(“Jim Newark”), a search result160athat includes a link to a resource for “New York Fitness”160a, and a search result160bthat includes a link to a resource for “Manhattan Body Building”160b. The search result160amay further include a phone number link that, when selected, may be dialed by the mobile device104.

FIG. 2is a flowchart of an example of a process200. Briefly, the process200includes receiving one or more geotagged environmental audio signals, receiving an utterance associated with a geographic location, and generating a noise model based in part upon the geographic location. Noise compensation may be performed on the audio signal, with the noise model contributing to improving an the accuracy of speech recognition.

In more detail, when process200begins, a geotagged audio signal corresponding to environmental audio is received (202). The geotagged audio signal may be recorded by a mobile device in a particular geographic location. The geotagged audio signal may include associated context data such as a time, date, speed, or amount of motion measured during the recording of the geotagged audio signal or a type of device which recorded the geotagged audio signal. The received geotagged audio signal may be processed to exclude portions of the environmental audio that include a voice of a user of the mobile device. Multiple geotagged audio signals recorded in one or more geographic locations may be received and stored.

An utterance recorded by a particular mobile device is received (204). The utterance may include a voice search query, or may be an input to a dictation or dialog application or system. The utterance may include associated context data such as a time, date, speed, or amount of motion measured during the recording of the geotagged audio signal or a type of device which recorded the geotagged audio signal.

A particular geographic location associated with the mobile device is determined (206). For example, data referencing the particular geographic location may be received from the mobile device, or a past geographic location or a default geographic location associated with the mobile device may be determined.

A noise model is generated for the particular geographic location using a subset of geotagged audio signals (208). The subset of geotagged audio signals may be selected by determining, for each of the geotagged audio signals, a distance between the particular geographic location and a geographic location associated the geotagged audio signal; and selecting those geotagged audio signals which are within a predetermined distance of the particular geographic location, or that are associated with geographic locations which are among the N closest geographic locations to the particular geographic location.

The subset of geotagged audio signals may be selected by identifying the geotagged audio signals associated with the particular geographic location, and/or by identifying the geotagged audio signals that are acoustically similar to the utterance. The subset of geotagged audio signals may be selected based both on the particular geographic location and on context data associated with the utterance.

Generating the noise model may include training a GMM using the subset of geotagged audio signals as a training set. Some noise reduction or separation algorithms, such as non-negative matrix factorization (NMF), can use the feature vectors themselves, not averages that are represented by the Gaussian components. Other algorithms, such as Alqonquin, can use either GMMs or the feature vectors themselves, with artificial variances.

Noise compensation is performed on the audio signal that corresponds to the utterance, using the noise model that has been generated for the particular geographic location, to enhance the audio signal or otherwise take decrease the uncertainty of the utterance due to noise (210).

Speech recognition is performed on the noise-compensated audio signal (212). Performing the speech recognition may include generating one or more candidate transcriptions of the utterance. A search query may be executed using the one or more candidate transcriptions, or one or more of the candidate transcriptions can be provided as an output of a digital dictation application. Alternatively, one or more of the candidate transcriptions may be provided as an input to a dialog system, to allow a computer system to converse with the user of the particular mobile device.

FIG. 3is a flowchart of an example of a process300. Briefly, the process300includes collecting geotagged audio signals and generating multiple noise models based, in part, upon particular geographic locations associated with each of the geotagged audio signals. One or more of these noise models may be selected when performing speech recognition upon an utterance based, in part, upon a geographic location associated with the utterance.

In more detail, when process300begins, a geotagged audio signal corresponding to environmental audio is received (302). The geotagged audio signal may be recorded by a mobile device in a particular geographic location. The received geotagged audio signal may be processed to exclude portions of the environmental audio that include the voice of the user of the mobile device. Multiple geotagged audio signals recorded in one or more geographic locations may be received and stored.

Optionally, context data associated with the geotagged audio signal is received (304). The geotagged audio signal may include associated context data such as a time, date, speed, or amount of motion measured during the recording of the geotagged audio signal or a type of device which recorded the geotagged audio signal.

One or more noise models are generated (306). Each noise model may be generated for a particular geographic location or, optionally, a location type, using a subset of geotagged audio signals. The subset of geotagged audio signals may be selected by determining, for each of the geotagged audio signals, a distance between the particular geographic location and a geographic location associated the geotagged audio signal and selecting those geotagged audio signals which are within a predetermined distance of the particular geographic location, or that are associated with geographic locations which are among the N closest geographic locations to the particular geographic location. The subset of geotagged audio signals may be selected by identifying the geotagged audio signals associated with the particular geographic location. The subset of geotagged audio signals may be selected based both on the particular geographic location and on context data associated with the geotagged audio signals. Generating the noise model may include training a Gaussian Mixture Model (GMM) using the subset of geotagged audio signals.

An utterance recorded by a particular mobile device is received (308). The utterance may include a voice search query. The utterance may include associated context data such as a time, date, speed, or amount of motion measured during the recording of the geotagged audio signal or a type of device which recorded the geotagged audio signal.

A geographic location is detected (310). For example, data referencing the particular geographic location may be received from a GPS module of the mobile device.

A noise model is selected (312). The noise model may be selected from among multiple noise models generated for multiple geographic locations. Context data may optionally contribute to selection of a particular noise model among multiple noise models for the particular geographic location.

Speech recognition is performed on the utterance using the selected noise model (314). Performing the speech recognition may include generating one or more candidate transcriptions of the utterance. A search query may be executed using the one or more candidate transcriptions.

FIG. 4shows a swim lane diagram of an example of a process400for enhancing speech recognition accuracy using geotagged environmental audio. The process400may be implemented by a mobile device402, an ASR engine404, and a search engine406. The mobile device402may provide audio signals, such as environmental audio signals or audio signals that correspond to an utterance, to the ASR engine404. Although only one mobile device402is illustrated, the mobile device402may represent a large quantity of mobile devices402contributing environmental audio signals and voice queries to the process400. The ASR engine404may generate noise models based upon the environmental audio signals, and may apply one or more noise models to an incoming voice search query when performing speech recognition. The ASR engine404may provide transcriptions of utterances within a voice search query to the search engine406to complete the voice search query request.

The process400begins with the mobile device402providing408a geotagged audio signal to the ASR engine404. The audio signal may include environmental audio along with an indication regarding the location at which the environmental audio was recorded. Optionally, the geotagged audio signal may include context data, for example in the form of metadata. The ASR engine404may store the geotagged audio signal in an environmental audio data store.

The mobile device402provides410an utterance to the ASR engine404. The utterance, for example, may include a voice search query. The recording of the utterance may optionally include a sample of environmental audio, for example recorded briefly before or after the recording of the utterance.

The mobile device402provides412a geographic location to the ASR engine404. The mobile device, in some examples, may provide navigational coordinates detected using a GPS module, a most recent (but not necessarily concurrent with recording) GPS reading, a default location, a location derived from the utterance previously provided, or a location estimated through dead reckoning or triangulation of transmission towers. The mobile device402may optionally provide context data, such as sensor data, device model identification, or device settings, to the ASR engine404.

The ASR engine404generates414a noise model. The noise model may be generated, in part, by training a GMM. The noise model may be generated based upon the geographic location provided by the mobile device402. For example, geotagged audio signals submitted from a location at or near the location of the mobile device402may contribute to a noise model. Optionally, context data provided by the mobile device402may be used to filter geotagged audio signals to select those most appropriate to the conditions in which the utterances were recorded. For example, the geotagged audio signals near the geographic location provided by the mobile device402may be filtered by a day of the week or a time of day. If a sample of environmental audio was included with the utterance provided by the mobile device402, the environmental audio sample may optionally be included in the noise model.

The ASR engine404performs speech recognition416upon the provided utterance. Using the noise model generated by the ASR engine404, the utterance provided by the mobile device402may be transcribed into one or more sets of query terms.

The ASR engine404forwards418the generated transcription(s) to the search engine406. If the ASR engine404generated more than one transcription, the transcriptions may optionally be ranked in order of confidence. The ASR engine404may optionally provide context data to the search engine406, such as the geographic location, which the search engine406may use to filter or rank search results.

The search engine406performs420a search operation using the transcription(s). The search engine406may locate one or more URIs related to the transcription term(s).

The search engine406provides422search query results to the mobile device402. For example, the search engine406may forward HTML code which generates a visual listing of the URI(s) located.

Thus, particular embodiments have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims may be performed in a different order and still achieve desirable results.