Patent ID: 12243548

Like reference numbers and designations in the various drawings indicate like elements.

DETAILED DESCRIPTION OF EMBODIMENTS

Noise compensation systems are configured to compensate for environmental noise, e.g., ambient noise, within an audio environment. As used herein, the terms “ambient noise” and “environmental noise” refer to noise produced by one or more noise sources that are external to an audio playback system and/or a noise compensation system. The audio environment may, in some examples, be a home audio environment, e.g., one or more rooms of a home. In other examples, the audio environment may be another type of environment, such as an office environment, an automobile environment, a train environment, a street or sidewalk environment, a park environment, etc.

FIG.1Ashows an example of a noise compensation system. In this example, the noise compensation system100is configured to adjust the level of the input audio signal101based upon a noise estimate108. According to this example, the noise compensation system100includes a loudspeaker104, a microphone105, a noise estimator107and a noise compensator102. In some examples, the noise estimator107and the noise compensator102may be implemented via a control system (such as the control system210that is described below with reference toFIG.2), e.g., according to instructions stored on one or more non-transitory storage media. As noted above, the terms “speaker,” “loudspeaker” and “audio reproduction transducer” are used synonymously herein. As with other figures provided herein, the types, numbers and arrangements of elements shown inFIG.1Aare merely provided by way of example. Other implementations may include more, fewer and/or different types, numbers or arrangements of elements, e.g., more loudspeakers.

In this example, the noise compensator102is configured to receive an audio signal101from a file, a streaming service, etc. The noise compensator102may, for example, be configured to apply a gain adjustment algorithm, such as a frequency-dependent gain adjustment algorithm or a broadband gain adjustment algorithm.

In this example, the noise compensator102is configured to send a noise-compensated output signal103to the loudspeaker104. According to this example, the noise-compensated output signal103is also provided to, and is a reference signal for, the noise estimator107. In this example, microphone signals106are also sent to the noise estimator107from the microphone105.

According to this example, the noise estimator107is a component that is configured to estimate the level of noise in an environment that includes the system100. The noise estimator107may be configured to receive the microphone signals106and to calculate how much of the microphone signals106consist of noise and how much is due to the playback of the loudspeaker104. The noise estimator107may, in some examples, include an echo canceller. However, in some implementations the noise estimator107may simply measure the noise when a signal corresponding with silence (a “quiet playback interval”) is sent to the loudspeaker104. In some such examples, the quiet playback intervals may be instances of audio signals at or below a threshold level in one or more frequency bands. Alternatively, or additionally, in some examples the quiet playback intervals may be instances of audio signals at or below a threshold level during a time interval.

In this example, the noise estimator107is providing a noise estimate108to the noise compensator102. The noise estimate108may be a broadband estimate or a spectral estimate of the noise, depending on the particular implementation. In this example, the noise compensator102is configured to adjust the level of the output of the loudspeaker104based upon the noise estimate108.

The loudspeakers of some devices, such as mobile devices, often have rather limited capabilities. Accordingly, the type of volume adjustment provided by the system100will be generally limited by the dynamic range and/or the speaker protection components (e.g., limiters and/or compressors) of such loudspeakers. Noise compensation systems such as the noise compensation system100may apply gains that are either frequency-dependent gains or broadband gains.

While not yet commonplace in the consumer electronics market, the utility of onboard microphones in home entertainment devices to measure and compensate for background noise has been demonstrated. The primary reason for the lack of adoption of this functionality relates to a problem that this document will refer to as “noise source proximity ambiguity,” as the “proximity ambiguity problem” or simply as the “proximity problem.” In the simplest sense, this problem arises due to the fact that sound pressure level (SPL) is a measured property that quantifies “how much sound there is” at a specific point in space.

Because acoustic waves lose energy as they propagate through a medium, a measurement made at one point in space is meaningless for all other points without prior knowledge of the distance between those points, as well as some properties of the transmission medium, in this case, air at room temperature. In anechoic spaces, these propagation losses are simple to model by an inverse square law. This inverse square law does not hold true for reverberant (real) rooms, so ideally, to model propagation, the reverberation characteristics of the physical space are also known.

The proximity of the noise source to the listener is an important factor in determining the detrimental impact of the noise from that noise source on content audibility and intelligibility for the listener. A measurement of sound pressure level via a microphone at an arbitrary location, such as on the housing of a television, is not sufficient for determining the detrimental impact of the noise for the listener, because this microphone may see an identical sound pressure level for a very loud but distant noise source as for a quiet, nearby source.

The present disclosure provides various methods that can overcome at least some of these potential drawbacks, as well as devices and systems for implementing the presently-disclosed methods. Some disclosed implementations involve measuring the SPL of the ambient noise at a listener position. Some disclosed implementations involve inferring the noise SPL at a listener position from the level detected at an arbitrary microphone location, by knowing (or inferring) the proximity of listener and noise source to the microphone location. Various examples of the foregoing implementations are described below with reference toFIG.4et seq.

Some alternative implementations involve predicting (e.g., on a per-frequency basis), how much error is likely to occur in an ambient noise estimate that does not involve a solution the noise source proximity ambiguity problem. Some examples are described below with reference toFIGS.1B-3B.

If the system does not implement one of the solutions described in the foregoing paragraphs, some disclosed noise compensation methods may apply level adjustments to the device's output that result in the content reproduction being either too loud or too quiet to the listener.

FIG.1Bshows another example of a noise compensation system. According to this example, the noise compensation system110includes a television111, a microphone112that is configured for sampling the acoustic environment (also referred to herein as the “audio environment”) in which the noise compensation system110resides, and stereo loudspeakers113and114. Although not shown inFIG.1B, in this example the noise compensation system110includes a noise estimator and a noise compensator, which may be instances of the noise estimator107and the noise compensator102that are described above with reference toFIG.1A. In some examples, the noise estimator and the noise compensator may be implemented via a control system, such as a control system of the television111(which may be an instance of the control system210that is described below with reference toFIG.2), e.g., according to instructions stored on one or more non-transitory storage media.

As with other figures provided herein, the types, numbers and arrangements of elements shown inFIG.1Bare merely provided by way of example. Other implementations may include more, fewer and/or different types and numbers of elements, e.g., more loudspeakers. In some implementations, the noise compensation methods that are described with reference toFIGS.1B-1Dmay be implemented via a control system of a device other than a television, such as a control system of another device having a display (e.g., a laptop computer), a control system of a smart speaker, a control system of a smart hub, a control system of another device of an audio system, etc.

According to this example, the noise compensation system110is shown attempting to compensate for multiple noise sources, in order to illustrate the aforementioned ambiguity of noise source to listener proximity. In this example, the audio environment118in which the noise compensation system110resides also includes a listener116(who is assumed to be stationary in this example), a noise source115that is closer to the television111than the listener116and a noise source117that is farther from the television111than the listener116. In highly-damped rooms, a noise compensation system may overcompensate for the noise source115in the absence of one of the disclosed methods for solving, or compensating for, the proximity problem. In minimally-damped rooms, a noise compensation system may undercompensate for the noise source117in the absence of one of the disclosed methods for solving, or compensating for, the proximity problem, because the noise source117is closer to the listener116than to the microphone112.

In this example, the noise compensation system110is configured for implementing a method that is based, at least in part, on a “critical distance” analysis. As used herein, a “critical distance” is a distance from an acoustic source at which the direct propagated sound pressure is equal to the diffuse field sound pressure. This property is frequency dependent and is commonly given at ISO standard octave or ⅓rdoctave band center frequencies. Critical distance is primarily a property of the volume (meaning the three-dimensional size, not the loudness) and the reverberance of an audio environment (e.g., of a room), but is also influenced by the directivity of the noise source. For a typical domestic living room, with an omnidirectional source, the critical distance Dcis approximately 0.75 meters at 1 kHz.

In highly reverberant rooms, the noise compensation system110can provide adequate noise compensation, despite failing to solve the proximity problem. This is owing to the fact that in highly reverberant environments, the distribution of acoustic energy throughout the room approaches homogeneity outside the critical distance.

In other words, in highly reverberant rooms, which have small critical distances, it is likely that both the listener116and the television111will be outside the critical distance from the noise source. In that case, the reverberant sound dominates the direct sound, and the sound is relatively homogenous regardless of the source distance and the source location. Given those conditions, it is less likely that there will be a discrepancy between the noise SPL measured at the television microphone112and the noise SPL experienced by the listener116. This means that error in the noise estimate due to the proximity problem becomes less likely. Because both critical distance and reverberation times are frequency-dependent properties, this probability of error is dependent on frequency as well.

Unfortunately, most residential living rooms are not highly reverberant across all frequencies. In other words, at some frequencies most residential living rooms may have larger, and sometimes much larger, critical distances than 0.75 meters. It is therefore likely that the listener116and the television111may be situated inside of the critical distance at some frequencies. At those frequencies, a noise compensation system that has not solved (or compensated for) the proximity problem will produce noise estimates that are not accurate for the noise level at a listener position, and will therefore apply an incorrect noise compensation.

Therefore, some disclosed implementations involve predicting the probability of error due to the proximity problem. To solve this problem, existing functionality within some previously-deployed devices may be leveraged to identify features of the acoustic environment. At least some previously-deployed devices that implement noise compensation will also feature a room acoustics compensation system. Using information already available from an existing room acoustics compensation system, frequency-dependent reverberation times (also known as spectral decay times) can be computed. This is accomplished by taking the impulse response of the system (already calculated for the room acoustics compensation system), and splitting it into discrete frequency bands. The time from the peak of the impulse to the point at which it has reduced in magnitude by 60 dB is the reverberation time for that frequency band.

After the spectral decay times are determined, the spectral decay times can be used, along with some knowledge of the room volume and source directivity, to infer the critical distance, from which the control system may predict the probability of a noise estimate error due to the proximity problem. If a small critical distance is predicted for a particular frequency bin (which may also be referred to herein as a frequency range or frequency band), in some implementations this will result in a high confidence score (e.g., 1.0) for the ambient noise estimate in that frequency bin. According to some examples, the noise compensation system may then perform unconstrained noise compensation in that frequency bin. The unconstrained noise compensation may, in some examples, correspond with a “default” noise compensation that would have been performed in response to the ambient noise estimate in that frequency bin according to a noise compensation method, e.g., ensuring the level of played-back audio exceeds the level of ambient noise detected by the microphone112by at least a threshold amount. The unconstrained noise compensation may, in some examples, correspond with a noise compensation method in which the output signal level of at least some frequency bands is not constrained according to the output signal level of and/or imposed thresholds for, other frequency bands.

In frequency bins for which the predicted critical distance is larger, in some implementations this will result in a lower confidence score for these frequency bins. In some examples, a lower confidence score results in implementing a modified noise compensation method. According to some such examples, the modified noise compensation method corresponding with a low confidence score may be a more conservative noise compensation method in which the level of played-back audio is boosted less than the level would be boosted according to the default method, to reduce the likelihood of erroneously large corrections.

According to some examples, a minimum (e.g., zero) confidence score may correspond with a minimum applied gain (e.g., a minimum difference between a reproduced audio level and an estimated ambient noise level) and a maximum (e.g., 1.0) confidence score may correspond with an unconstrained or “default” level adjustment for noise compensation. In some examples, confidence values between the minimum and the maximum may correspond to linear interpolations between a level adjustment corresponding to the minimum confidence score (e.g., the minimum applied gain) and the “default” level adjustment for noise compensation.

In some implementations, a minimum (e.g., zero) confidence score may correspond with a timbre-preserving noise compensation method and a maximum (e.g., 1.0) confidence score may correspond with an unconstrained or “default” level adjustment for noise compensation. The term “timbre-preserving” may have a variety of meanings as used herein. Broadly speaking, a “timbre-preserving” method is one that at least partially preserves the frequency content, or timbre of an input audio signal. Some timbre-preserving methods may completely, or almost completely, preserve the frequency content of an input audio signal. A timbre-preserving method may involve constraining the output signal level of at least some frequency bands according to the output signal level and/or imposed thresholds of at least some other frequency bands. In some examples, a “timbre-preserving” method may involve constraining, at least to some degree, the output signal level of all non-isolated frequency bands. (In some examples, if a frequency band is “isolated,” then only the audio in that frequency band has an effect on the limiting gain that is applied.)

In some examples, confidence values may be inversely proportional to a timbre preservation setting. For example, if the minimum confidence value is 0.0 and the maximum confidence value is 1.0, a minimum (e.g., zero) confidence score may correspond with timbre preservation setting of 100% or 1.0. In some examples, a timbre preservation setting of 0.50 may correspond with a confidence value of 0.5. In some such examples, a confidence value of 0.25 may correspond to a timbre preservation setting of 0.75.

For the proximity problem to be considered unimportant in any given frequency bin, the listener must be outside the critical distance for that frequency bin. The critical distance for a particular frequency may inferred from the reverberation time for that frequency using a statistical reverberation time model, e.g., as follows:

Dc=0.0⁢5⁢7⁢QVTEquation⁢1

In Equation 1, Dcrepresents the critical distance, Q represents the directivity factor of the noise source (assumed to omni-directional in some implementations), V represents the volume of the room (e.g., in m3) and T represents the measured reverberation time, RT60, in seconds. RT60is defined as the time required for the amplitude of a theoretically perfect impulse to decay in amplitude by 60 dB.

In some examples, the volume of the room may be assumed to be a particular size, e.g., 60 m3, based on typical living room sizes. In some examples, the volume of the room may be determined according to input from a user, e.g., via a graphical user interface (GUI) at the time of unboxing/setup. The input may be numerical, e.g., based on a user's actual measurements or estimates. In some such implementations, a user may be presented with a set of “multiple choice” options (e.g., is your room a large room, a medium-sized room or a small room”) via the GUI. Each option may correspond with a different value of V.

In some implementations, the Equation 1 is solved for each of a plurality of frequency bins, e.g., for every frequency bin used by the noise compensation system110. According to some examples, a confidence score may be produced by the following method:It is assumed that the listener116will not be seated less than 2 meters from the television111.If the predicted critical distance is equal to, or smaller than 2 meters, the confidence score is set to 1.With increasing critical distance, the confidence score decreases, up to a lower bound, where Dc=5 m and confidence=0.

Alternative examples may involve alternative methods of determining a confidence score. The alternative methods may, for example, involve a different assumption about the proximity of the listener116to the television111and/or a different critical distance for the lower bound, e.g., of 4 meters, 4.5 meters, 5.5 meters, 6 meters, etc. Some implementations may involve measuring or estimating the actual position of the listener116and/or the distance between the listener116and the television111. Some implementations may involve obtaining user input regarding the actual position of the listener116and/or the distance between the listener116and the television111. Some examples may involve determining a location of a device, such as a cellular telephone or a remote control device, and assuming that the location of the device corresponds with the location of the listener.

According to various disclosed implementations, the above-described confidence scores represent the probability of errors in the noise estimation of the noise compensation system110. Given that there may, in some implementations, be no way to differentiate between an overestimate and an underestimate, in some such implementations the noise compensation system110may always assume the noise estimation error to be an overestimate. This assumption reduces the likelihood that the noise compensation system110will erroneously apply excessive gains to the audio reproduced by the loudspeakers113and114. Such implementations are potentially advantageous, because applying excessive gains would generally be a more perceptually obvious failure mode than the opposite case of applying insufficient gains to adequately overcome the ambient noise.

In some implementations, if the confidence score is 1, the frequency-dependent gains calculated by the noise compensation system110are applied unconstrained. According to some such implementations, for all confidence values less than 1, these frequency-dependent gains are scaled down.

FIG.1Cis a flow diagram that illustrates a method of using a spectral decay time measurement to score the confidence of noise estimates according to some disclosed examples. This figure shows the use of an impulse response, which in some implementations may have already been derived for the purpose of room acoustics compensation. According to this example, this impulse response is decomposed into discrete frequency bands corresponding to the bands that the noise compensation system operates in. The time taken for each of these band-limited impulse responses to decay by 60 dB is the reverberation time RT60 for that band.

FIG.1Dis a flow diagram that illustrates a method of using the noise estimate confidence scores in a noise compensation process according to some disclosed examples. The operations shown inFIGS.1C and1Dmay, for example, be performed via a control system such as the control system210that is described below with reference toFIG.2. The blocks of methods120and180, like other methods described herein, are not necessarily performed in the order indicated. Moreover, such methods may include more or fewer blocks than shown and/or described. In the examples shown inFIGS.1C and1D, blocks including multiple arrows indicate that the corresponding audio signal is separated into frequency bins by a filter bank.

The method120ofFIG.1Cmay correspond to a “set-up” mode that may, for example, occur when a television, an audio device, etc., is first installed in an audio environment. In this example, block125involves causing one or more audio reproduction transducers to play a room calibration signal. Here, block130involves recording the impulse response of the room to the room calibration signal via one or more microphones.

Here, block135involves transforming the impulse response from the time domain into the frequency domain: here, the corresponding audio signal is separated into frequency bins by a filter bank. In this example, block140involves performing a decay time analysis and determining the reverberation time, RT60, in seconds. This analysis involves finding the peak of each band-limited impulse response, counting the number of samples until the impulse response decays in magnitude by 60 dB, then dividing that number of samples by the sampling frequency in Hz. The result is the reverberation time RT60, in seconds, for that band.

According to this example, block145involves determining noise estimation confidence scores for each of a plurality of frequency bins, e.g., for every frequency bin used by the noise compensation system110. In some implementations, block145involves solving Equation 1 for each of the frequency bins. Although not shown inFIG.1C, a value of V, corresponding to the volume of the room, is also determined in method120, e.g., according to user input, based on a room measurement or estimation process according to sensor input or by using a default value. According to some examples, a confidence score may be produced by assuming that the listener116will not be seated less than 2 meters from the television111, setting the confidence score is set to 1 if the predicted critical distance is equal to, or smaller than 2 meters. With increasing critical distance, the confidence score may decrease, e.g., up to a lower bound where the critical distance is 5 m and the confidence score is zero. Alternative examples may involve alternative methods of determining a confidence score. In some instances, the confidence scores determined in block145may be stored in a memory.

In this example, the method180ofFIG.1Dcorresponds to a “run time” mode that may, for example, occur when a television, an audio device, etc., is in use on a day to day basis, after the method ofFIG.1Chas been performed. In this example, echo cancellation block155involves receiving microphone signals from the microphone111and also receiving an echo reference signal150, which may be a speaker feed signal provided to an audio reproduction transducer of the audio environment. Here, block160involves producing a noise estimate for each of a plurality of frequency bins (also referred to herein as frequency bands), based on output from the echo cancellation block155.

In this example, noise compensation scaling block165involves applying the confidence scores that were determined in block145in order to provide appropriate scaling, if any, for noise compensation gains that will be applied based on the frequency-dependent noise estimate received from block160. In some instances, the confidence scores that were determined in block145may have been stored for later use, e.g., in the run time operations of method180. The scaling determined by the noise compensation scaling block165may, for example, be performed according to one of the examples described above with reference toFIG.1B.

According to this example, block170involves determining frequency-dependent gains based on the scaling values received from the noise compensation scaling block165. Here, block175involves providing noise-compensated output audio data to one or more audio transducers of the audio environment.

FIG.2is a block diagram that shows examples of components of an apparatus capable of implementing various aspects of this disclosure. As with other figures provided herein, the types, numbers and arrangements of elements shown inFIG.2are merely provided by way of example. Other implementations may include more, fewer and/or different types and numbers of elements. According to some examples, the apparatus200may be configured for performing at least some of the methods disclosed herein. In some implementations, the apparatus200may be, or may include, a television, one or more components of an audio system, a mobile device (such as a cellular telephone), a laptop computer, a tablet device, a smart speaker, or another type of device. In some implementations, the apparatus200may be, or may include, a television control module. The television control module may or may not be integrated into a television, depending on the particular implementation. In some implementations, the television control module may be a separate device from a television and may, in some instances, either be sold separately from a television or as an add-on or optional device that may be included with a purchased television. In some implementations, the television control module may be obtainable from a content provider, such as a provider of television programs, movies, etc.

According to some alternative implementations the apparatus200may be, or may include, a server. In some such examples, the apparatus200may be, or may include, an encoder. Accordingly, in some instances the apparatus200may be a device that is configured for use within an audio environment, such as a home audio environment, whereas in other instances the apparatus200may be a device that is configured for use in “the cloud,” e.g., a server.

In this example, the apparatus200includes an interface system205and a control system210. The interface system205may, in some implementations, be configured for communication with one or more other devices of an audio environment. The audio environment may, in some examples, be a home audio environment. In other examples, the audio environment may be another type of environment, such as an office environment, an automobile environment, a train environment, a street or sidewalk environment, a park environment, etc. According to some implementations, the size and/or reverberation of the audio environment may be assumed, based on the audio environment type. For example, a default office size may be used for an office audio environment. The audio environment type may, for example, be determined according to user input or based on audio characteristics of the environment. The interface system205may, in some implementations, be configured for exchanging control information and associated data with audio devices of the audio environment. The control information and associated data may, in some examples, pertain to one or more software applications that the apparatus200is executing.

The interface system205may, in some implementations, be configured for receiving, or for providing, a content stream. The content stream may include audio data. The audio data may include, but may not be limited to, audio signals. In some instances, the audio data may include spatial data, such as channel data and/or spatial metadata. According to some implementations, the content stream may include metadata regarding a dynamic range of the audio data and/or metadata regarding one or more noise compensation methods. Metadata regarding a dynamic range of the audio data and/or metadata regarding one or more noise compensation methods may, for example, have been provided by one or more devices configured to implement a cloud-based service, such as one or more servers. Metadata regarding a dynamic range of the audio data and/or metadata regarding one or more noise compensation methods may, for example, have been provided by what may be referred to herein as an “encoder.” In some such examples, the content stream may include video data and audio data corresponding to the video data. Some examples of encoder and decoder operations are described below.

The interface system205may include one or more network interfaces and/or one or more external device interfaces (such as one or more universal serial bus (USB) interfaces). According to some implementations, the interface system205may include one or more wireless interfaces. The interface system205may include one or more devices for implementing a user interface, such as one or more microphones, one or more speakers, a display system, a touch sensor system and/or a gesture sensor system. In some examples, the interface system205may include one or more interfaces between the control system210and a memory system, such as the optional memory system215shown inFIG.2. However, the control system210may include a memory system in some instances. The interface system205may, in some implementations, be configured for receiving input from one or more microphones in an environment.

The control system210may, for example, include a general purpose single- or multi-chip processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, and/or discrete hardware components.

In some implementations, the control system210may reside in more than one device. For example, in some implementations a portion of the control system210may reside in a device within one of the environments depicted herein and another portion of the control system210may reside in a device that is outside the environment, such as a server, a mobile device (e.g., a smartphone or a tablet computer), etc. In other examples, a portion of the control system210may reside in a device within one of the environments depicted herein and another portion of the control system210may reside in one or more other devices of the environment. For example, control system functionality may be distributed across multiple smart audio devices of an environment, or may be shared by an orchestrating device (such as what may be referred to herein as a smart home hub) and one or more other devices of the environment. In other examples, a portion of the control system210may reside in a device that is implementing a cloud-based service, such as a server, and another portion of the control system210may reside in another device that is implementing the cloud-based service, such as another server, a memory device, etc. The interface system205also may, in some examples, reside in more than one device.

In some implementations, the control system210may be configured for performing, at least in part, the methods disclosed herein. According to some examples, the control system210may be configured for implementing methods of content stream processing.

Some or all of the methods described herein may be performed by one or more devices according to instructions (e.g., software) stored on one or more non-transitory media. Such non-transitory media may include memory devices such as those described herein, including but not limited to random access memory (RAM) devices, read-only memory (ROM) devices, etc. The one or more non-transitory media may, for example, reside in the optional memory system215shown inFIG.2and/or in the control system210. Accordingly, various innovative aspects of the subject matter described in this disclosure can be implemented in one or more non-transitory media having software stored thereon. The software may, for example, include instructions for controlling at least one device to process a content stream, to encode a content stream, to decode a content stream, etc. The software may, for example, be executable by one or more components of a control system such as the control system210ofFIG.2.

In some examples, the apparatus200may include the optional microphone system220shown inFIG.2. The optional microphone system220may include one or more microphones. In some implementations, one or more of the microphones may be part of, or associated with, another device, such as a speaker of the speaker system, a smart audio device, etc. In some examples, the apparatus200may not include a microphone system220. However, in some such implementations the apparatus200may nonetheless be configured to receive microphone data for one or more microphones in an audio environment via the interface system210. In some such implementations, a cloud-based implementation of the apparatus200may be configured to receive microphone data, or a noise metric corresponding at least in part to the microphone data, from one or more microphones in an audio environment via the interface system210.

According to some implementations, the apparatus200may include the optional loudspeaker system225shown inFIG.2. The optional loudspeaker system225may include one or more loudspeakers, which also may be referred to herein as “speakers” or, more generally, as “audio reproduction transducers.” In some examples (e.g., cloud-based implementations), the apparatus200may not include a loudspeaker system225.

In some implementations, the apparatus200may include the optional sensor system230shown inFIG.2. The optional sensor system230may include one or more touch sensors, gesture sensors, motion detectors, etc. According to some implementations, the optional sensor system230may include one or more cameras. In some implementations, the cameras may be free-standing cameras. In some examples, one or more cameras of the optional sensor system230may reside in a smart audio device, which may be a single purpose audio device or a virtual assistant. In some such examples, one or more cameras of the optional sensor system230may reside in a television, a mobile phone or a smart speaker. In some examples, the apparatus200may not include a sensor system230. However, in some such implementations the apparatus200may nonetheless be configured to receive sensor data for one or more sensors in an audio environment via the interface system210.

In some implementations, the apparatus200may include the optional display system235shown inFIG.2. The optional display system235may include one or more displays, such as one or more light-emitting diode (LED) displays. In some instances, the optional display system235may include one or more organic light-emitting diode (OLED) displays. In some examples, the optional display system235may include one or more displays of a television. In other examples, the optional display system235may include a laptop display, a mobile device display, or another type of display. In some examples wherein the apparatus200includes the display system235, the sensor system230may include a touch sensor system and/or a gesture sensor system proximate one or more displays of the display system235. According to some such implementations, the control system210may be configured for controlling the display system235to present one or more graphical user interfaces (GUIs).

According to some such examples the apparatus200may be, or may include, a smart audio device. In some such implementations the apparatus200may be, or may include, a wakeword detector. For example, the apparatus200may be, or may include, a virtual assistant.

FIG.3is a flow diagram that outlines one example of a disclosed method. The blocks of method300, like other methods described herein, are not necessarily performed in the order indicated. Moreover, such methods may include more or fewer blocks than shown and/or described. The method300may be performed by an apparatus or system, such as the apparatus200that is shown inFIG.2and described above. In some examples, the blocks of method300may be performed by one or more devices within an audio environment, e.g., an audio system controller or another component of an audio system, such as a smart speaker, a television, a television control module, a smart speaker, a mobile device, etc. In some implementations, the audio environment may include one or more rooms of a home environment. In other examples, the audio environment may be another type of environment, such as an office environment, an automobile environment, a train environment, a street or sidewalk environment, a park environment, etc. However, in alternative implementations at least some blocks of the method300may be performed by a device that implements a cloud-based service, such as a server.

In this implementation, block305involves receiving, by a control system and via an interface system, microphone signals corresponding to ambient noise from a noise source location in or near an audio environment. In some implementations, the control system and the interface system may be the control system210and the interface system205shown inFIG.2and described above.

In this example, block310involves determining or estimating, by the control system, a listener position in the audio environment. According to some examples, block310may involve determining a listener position according to a default value of an assumed listener position, e.g., that the listener is 2 meters in front of a television or other device or at least 2 meters in front of the television or other device, that the listener is seated on a piece of furniture having a known location with reference to a television or other device, etc. However, in some implementations block310may involve determining a listener position according to user input, determining a listener position according to sensor input (e.g., from a camera of the sensor system230shown inFIG.2), etc. Some examples may involve determining a location of a device, such as a cellular telephone or a remote control device, and assuming that the location of the device corresponds with the listener position.

According to this example, block315involves estimating, by the control system, at least one critical distance. As noted elsewhere herein, a critical distance is a distance from the noise source location at which directly propagated sound pressure is equal to diffuse field sound pressure. In some examples, block315may involve retrieving at least one estimated critical distance from a memory in which the results of the method ofFIG.1C, or a similar method, have been stored. Some such methods may involve controlling, via the control system, an audio reproduction transducer system in an audio environment to reproduce one or more room calibration sounds. The audio reproduction transducer system including one or more audio reproduction transducers. Some such methods may involve receiving, by the control system and via the interface system, microphone signals corresponding to the audio environment's response to the one or more room calibration sounds. Some such methods may involve determining, by the control system and based on the microphone signals, a reverberation time for each of a plurality of frequencies. Some such methods may involve determining or estimating an audio environment volume of the audio environment (in other words, determining the size of the audio environment in cubic feet, cubic meters, etc.), e.g., as disclosed elsewhere herein. According to some such examples, estimating the at least one critical distance may involve calculating, based at least in part on the plurality of frequency-dependent reverberation times and the audio environment volume, a plurality of estimated frequency-based critical distances. Each estimated frequency-based critical distance of the plurality of estimated frequency-based critical distances may correspond to a frequency of the plurality of frequencies.

In this example, block320involves estimating whether the listener position is within the at least one critical distance. According to some examples, block320may involve estimating whether the listener position is within each frequency-based critical distance of the plurality of frequency-based critical distances. In some examples, method300may involve transforming the microphone signals corresponding to the ambient noise from a time domain into a frequency domain and determining a frequency band ambient noise level estimate for each of a plurality of ambient noise frequency bands. According to some such examples, method300may involve determining a frequency-based confidence level for each of the frequency band ambient noise level estimates. Each frequency-based confidence level may, for example, correspond to an estimate or a probability of whether the listener position is within each frequency-based critical distance. In some examples, each frequency-based confidence level may be inversely proportional to each frequency-based critical distance.

According to this implementation, block325involves implementing a noise compensation method for the ambient noise based, at least in part, on at least one estimate of whether the listener position is within the at least one critical distance. In some examples, block325may involve implementing a frequency-based noise compensation method based on the frequency-based confidence level for each ambient noise frequency band. According to some such examples, the frequency-based noise compensation method may involve applying a default noise compensation method for each ambient noise frequency band for which the confidence level is at or above a threshold confidence level. In some instances, the threshold confidence level may be the maximum confidence level, e.g., 1.0. However, in other examples in which the maximum confidence level is 1.0, the threshold confidence level may be another confidence level, e.g., 0.80, 0.85, 0.90, 0.95, etc.

In some examples, the frequency-based noise compensation method may involve modifying a default noise compensation method for each ambient noise frequency band for which the confidence level is below a threshold confidence level. According to some such examples, modifying the default noise compensation method may involve reducing a default noise compensation level adjustment for one or more frequency bands.

In some examples, confidence values between the minimum and the threshold confidence level (e.g., the maximum confidence level) may correspond to a linear interpolations between the minimum applied gain and the “default” level adjustment for noise compensation. In some implementations, a minimum (e.g., zero) confidence score may correspond with a timbre-preserving noise compensation method and a maximum (e.g., 1.0) confidence score may correspond with an unconstrained or “default” level adjustment for noise compensation. In some examples, confidence values may be inversely proportional to a timbre preservation setting. For example, if the minimum confidence value is 0.0 and the maximum confidence value is 1.0, a minimum (e.g., zero) confidence score may correspond with timbre preservation setting of 100% or 1.0. In some examples, a timbre preservation setting of 0.50 may correspond with a confidence value of 0.5. In some such examples, a confidence value of 0.25 may correspond to a timbre preservation setting of 0.75.

According to some examples, method300may involve receiving, by the control system and via the interface system, a content stream that includes audio data. In some such examples, implementing the noise compensation method in block325may involve applying the noise compensation method to the audio data to produce noise-compensated audio data. Some such implementations may involve providing, by the control system and via the interface system, the noise-compensated audio data to one or more audio reproduction transducers of the audio environment. Some such implementations may involve rendering, by the control system, the noise-compensated audio data to produce rendered audio signals. Some such implementations may involve providing, by the control system and via the interface system, the rendered audio signals to at least some audio reproduction transducers of a set of audio reproduction transducers of the audio environment.

FIGS.4A and4Bshows additional examples of noise compensation system components.FIG.4Cis a timing diagram that shows examples of operations that may be performed via the noise compensation system shown inFIGS.4A and4B. According to these examples, the noise compensation system410includes a television411, a television microphone412that is configured for sampling the audio environment in which the noise compensation system410resides, stereo loudspeakers413and414, and a remote control417for the television411. Although not shown inFIGS.4A and4B, in this example, the noise compensation system410includes a noise estimator and a noise compensator, which may be instances of the noise estimator107and the noise compensator102that are described above with reference toFIG.1A.

In some examples, the noise estimator107and the noise compensator102may be implemented via a control system, such as a control system of the television411(which may be an instance of the control system210that is described below with reference toFIG.2), e.g., according to instructions stored on one or more non-transitory storage media. Similarly, the operations described with reference toFIGS.4A-5may be via a control system of the television411, via a control system of the remote control417, or via both control systems, depending on the particular implementation. In some implementations, the noise compensation methods that are described with reference toFIGS.4A-5may be implemented via a control system of a device other than a television and/or a remote control device, such as a control system of another device having a display (e.g., a laptop computer), a control system of a smart speaker, a control system of a smart hub, a control system of another device of an audio system, etc. In some implementations, a smart phone (a cellular telephone) or a smart speaker (e.g., a smart speaker configured to provide virtual assistant functionality) may be configured to perform the operations that are described with reference toFIGS.4A-4Cas being performed by the remote control417. As with other figures provided herein, the types, numbers and arrangements of elements shown inFIGS.4A-4Care merely provided by way of example. Other implementations may include more, fewer and/or different types, numbers or arrangements of elements, e.g., more loudspeakers and/or more microphones, more or fewer operations, etc. For example, in other implementations the arrangement of elements on the remote control417(e.g., of the remote control microphone253, the radio transceiver252B and/or the infrared (IR) transmitter251) may be different. In some such examples, the radio transceiver252B and/or the infrared (IR) transmitter251may reside on the front side of the remote control417, e.g., the side that is shown pointing towards the television411inFIG.4B.

In the example shown inFIG.4A, the audio environment400in which the noise compensation system410resides also includes a listener416(who is assumed to be stationary in this example) and a noise source415that is closer to the television411than the listener416. The type and location of the noise source415are merely shown by way of example. In alternative examples the listener416may not be stationary. In some such examples, the listener416will be assumed to be in the same location as the remote control417or another device capable of providing similar functionality, such as a cellular telephone.

In the examples shown inFIGS.4A and4B, the remote control417is battery powered and incorporates a remote control microphone253. In some implementations, the remote control417includes the remote control microphone253because the remote control417is configured to provide voice assistant functionality. In order to conserve battery life, in some implementations the remote control microphone253is not sampling the ambient noise at all times, nor is the remote control microphone253transmitting a continuous stream to the television411. Rather, in some such examples, the remote control microphone253is not always on, but instead “listens” when the remote control417receives corresponding input, such as a button being pushed.

In some examples, the remote control microphone253can be used to provide noise level measurements when polled by the television411, in order to resolve the proximity problem. In some such implementations the remote control microphone253may be awakened in response to a signal from the television411to the remote control417, e.g., from the radio transceiver252A to the radio transceiver252B shown inFIG.4B. The signal from the television411may be in response to ambient noise detected by the television microphone412. Alternatively, or additionally, in some examples the remote control417may be polled by the television411at regular intervals for short, time-windowed ambient noise recordings. According to some examples, when the ambient noise abates, the television411may cause the polling to be discontinued. In some alternative examples, the television411may be configured to cause the polling to be discontinued when the television411has an adequate conversion function between the noise level at the location of the remote control417and the television microphone412. According to some such implementations, the television411may be configured to resume polling to upon receiving an indication that the remote control417has moved, e.g., upon receiving inertial sensor signals from the remote control417corresponding to movement. In some implementations, the levels of recordings made via the remote control microphone253may be used to determine whether the noise estimates made at the television411are valid for the listener position (which in some examples is presumed to correspond with the position of the remote control417), thereby ensuring that background noise is not over compensated or undercompensated due to proximity error.

According to some examples, upon a polling request from the television411, the remote control417may send a short recording of audio detected by the remote control microphone253to the television411across a wireless connection, e.g., from the radio transceiver252B to the radio transceiver252A shown inFIG.4B. The control system of the television411may be configured to remove the output of the loudspeakers413and414from the recording, e.g., by passing the recording through an echo canceller. In some examples, the control system of the television411may be configured to compare the residual noise recording with the ambient noise detected by the television microphone412in order to determine whether the ambient noise from the noise source is louder at the television microphone412or the listener position. In some implementations, a noise estimate made according to input from the television microphone412may be scaled accordingly, e.g., according to the ratio of the ambient noise level detected by the remote control microphone253to the ambient noise level detected by the television microphone412.

According to some implementations, signals sent by the infrared (IR) transmitter251of the remote control417and received by IR receiver250of the television411may be used as a synchronization reference, e.g., to time-align the echo reference with the remote control's recordings for the purpose of echo cancellation. Such implementations can solve the problem of clock synchronization between the remote control417and the television411without the need for clock signals to be transmitted continuously, which would have an unacceptable impact on battery life.

FIG.4Cshows a detailed example of one such implementation. In this example, time is depicted as the horizontal axis and a variety of different operations are illustrated perpendicular to various portions of the vertical axis. In this example, the audio that is played back by television loudspeakers413and414ofFIGS.4A and4Bis represented as the waveform261.

According to this example, the television411sends a radio signal271to the remote control417, via the radio transceiver252A. The radio signal271may, for example, be sent in response to ambient noise detected by the television microphone412. In this example, the radio signal271includes instructions for the remote control417to record an audio segment via the remote control microphone253. In some examples, the radio signal271may include a start time (e.g., the time Trefshown inFIG.4C), information for determining the start time, a time interval, etc., for the remote control417to record the audio segment.

In this example, the remote control417records signals received by the remote control microphone253as the audio segment272during a recorded audio segment time interval Trec. According to this example, the remote control417sends a signal265to the television411indicating the recorded audio segment time interval Trec. Here, the signal265indicates that the recorded audio segment time interval Tree begins at time Trefand ends at a time263, when the signal265ceases to be transmitted. In this example, the remote control417sends the signal265via the IR transmitter251. Accordingly, the television411can identify the time interval for a content stream audio segment269that is being reproduced by the television loudspeakers413and414during the recorded audio segment time interval Trec.

In this example, the remote control417subsequently sends a signal266to the television411that includes the recorded audio segment. According to this implementation, a control system of the television411performs an echo cancellation process based on the recorded audio segment and the content stream audio segment269, in order to obtain an ambient noise signal270at the location of the remote control417, which in this example is presumed to correspond with a location of the listener416. In some such implementations, a control system of the television411is configured for implementing a noise compensation method for the audio data that is to be reproduced by the television loudspeakers413and414based, at least in part, on the ambient noise signal270, to produce noise-compensated audio data.

FIG.5is a flow diagram that outlines one example of a disclosed method. The blocks of method500, like other methods described herein, are not necessarily performed in the order indicated. Moreover, such methods may include more or fewer blocks than shown and/or described.

The method500may be performed by an apparatus or system, such as the apparatus200that is shown inFIG.2and described above. In some examples, the blocks of method500may be performed by one or more devices within an audio environment, e.g., an audio system controller or another component of an audio system, such as a smart speaker, a television, a television control module, a smart speaker, a mobile device, etc. In some implementations, the audio environment may include one or more rooms of a home environment. In other examples, the audio environment may be another type of environment, such as an office environment, an automobile environment, a train environment, a street or sidewalk environment, a park environment, etc. However, in alternative implementations at least some blocks of the method500may be performed by a device that implements a cloud-based service, such as a server.

In this implementation, block505involves receiving, by a first device control system and via a first interface system of a first device in an audio environment, a content stream that includes content audio data. According to some examples, the first device may be a television or a television control module. In some such examples, the content stream also may include content video data corresponding to the content audio data. However, in other examples the first device may be another type of device, such as a laptop computer, a smart speaker, a sound bar, etc.

In this example, block510involves receiving, by the first device control system and via the first interface system, first microphone signals from a first device microphone system of the first device. The first device microphone system may include one or more microphones. According to some examples in which the first device is a television or a television control module, the first microphone signals may be received from one or more microphone that are in, on or near a television, such as the television microphone412that is described above with reference toFIGS.4A and4B. According to this implementation, block515involves detecting, by the first device control system and based at least in part on the first microphone signals, ambient noise from a noise source location in or near the audio environment.

According to this example, block520involves causing, by the first device control system, a first wireless signal to be transmitted from the first device to a second device in the audio environment via the first interface system. In this example, the first wireless signal includes instructions for the second device to record an audio segment via a second device microphone system. In some implementations, the second device may be a remote control device, a smart phone or a smart speaker. According to some examples, the first wireless signal may be sent via radio waves or microwaves. In some examples block520may involve sending the signal271, as described above with reference toFIG.4C. According to this example, the first wireless signal is responsive to detecting the ambient noise in block515. According to some examples, the first wireless signal may be responsive to determining that the detected ambient noise is greater than or equal to an ambient noise threshold level.

In some instances, the first wireless signal may include a second device audio recording start time or information for determining the second device audio recording start time. In some examples, information for determining the second device audio recording start time may include, or may be, instructions for waiting until a frequency hop occurs in cases in which the first wireless signal is transmitted via a frequency hopping system (e.g. Bluetooth). In some examples, information for determining the second device audio recording start time may include, or may be, instructions for waiting until a time slot is available in cases in which the first wireless signal is transmitted via a time division multiplexed wireless system. In some examples, the first wireless signal may indicate a second device audio recording time interval.

According to this example, block525involves receiving, by the first device control system and via the first interface system, a second wireless signal from the second device. According to some examples, the second wireless signal may be sent via infrared waves. In some examples block525may involve receiving the signal265, as described above with reference toFIG.4C. In some examples, the second wireless signal may indicate a second device audio recording start time. In some examples, the second wireless signal may indicate a second device audio recording time interval. According to some examples, the second wireless signal (or a subsequent signal from the second device) may indicate a second device audio recording end time. In some such examples, the method500may involve receiving, by the first device control system and via the first interface system, a fourth wireless signal from the second device, the fourth wireless signal indicating a second device audio recording end time.

In this example, block530involves determining, by the first device control system, a content stream audio segment time interval for a content stream audio segment. In some examples block530may involve determining the time interval for the content stream audio segment269, as described above with reference toFIG.4C. In some instances, the first device control system content stream may be configured to determine the audio segment time interval according to a second device audio recording start time and a second device audio recording end time, or according to a second device audio recording start time and a second device audio recording time interval. In some examples that involve receiving a fourth wireless signal from the second device indicating a second device audio recording end time, method500may involve determining a content stream audio segment end time based on the second device audio recording end time.

According to this example, block535involves receiving, by the first device control system and via the first interface system, a third wireless signal from the second device, the third wireless signal including a recorded audio segment captured via the second device microphone. In some examples block535may involve receiving the signal266, as described above with reference toFIG.4C.

In this example, block540involves determining, by the first device control system, a second device ambient noise signal at the second device location based, at least in part, on the recorded audio segment and the content stream audio segment. In some examples block540may involve performing an echo cancellation process based on the recorded audio segment and the content stream audio segment269, in order to obtain an ambient noise signal270at the location of the remote control417, as described above with reference toFIG.4C.

According to this example, block545involves implementing, by the first device control system, a noise compensation method for the content audio data based, at least in part, on the second device ambient noise signal, to produce noise-compensated audio data. In some examples, method500may involve receiving, by the first device control system and via the first interface system, second microphone signals from the first device microphone system during a second device audio recording time interval. Some such examples may involve detecting, by the first device control system and based at least in part on the first microphone signals, a first device ambient noise signal corresponding to the ambient noise from the noise source location. In such examples, the noise compensation method may be based, at least in part, on the first device ambient noise signal.

According to some such examples, the noise compensation method may be based, at least in part, on a comparison of the first device ambient noise signal and the second device ambient noise signal. In some examples, the noise compensation method may be based, at least in part, on a ratio of the first device ambient noise signal and the second device ambient noise signal.

Some examples may involve providing (e.g., by the first device control system and via the first interface system) the noise-compensated audio data to one or more audio reproduction transducers of the audio environment. Some examples may involve rendering (e.g., by the first device control system) the noise-compensated audio data to produce rendered audio signals. Some such examples may involve providing (e.g., by the first device control system and via the first interface system) the rendered audio signals to at least some audio reproduction transducers of a set of audio reproduction transducers of the audio environment. In some such examples, at least one of the reproduction transducers of the audio environment may reside in the first device.

FIG.6shows an additional example of a noise compensation system. In this example,FIG.6shows an example of a noise compensation system having three microphones, which allow a control system to determine the location of a noise source. In the example shown inFIG.6, the noise compensation system710includes a television711and television microphones702a,702band702c. In some alternative examples, the noise compensation system710may include a remote control for the television711, which may in some instances be configured to function like the remote control417. Although not shown inFIG.6, the noise compensation system includes a noise estimator and a noise compensator, which may be instances of the noise estimator107and a the noise compensator102that are described above with reference toFIG.1A.

In some examples, the noise estimator107and the noise compensator102may be implemented via a control system, such as a control system of the television611(which may be an instance of the control system210that is described below with reference toFIG.2), e.g., according to instructions stored on one or more non-transitory storage media. Similarly, the operations described with reference toFIGS.6-7Bmay be via a control system of the television611, via a control system of remote control, or via both control systems, depending on the particular implementation. In some implementations, the noise compensation methods that are described with reference toFIGS.6-7Bmay be implemented via a control system of a device other than a television and/or a remote control device, such as a control system of another device having a display (e.g., a laptop computer), a control system of a smart speaker, a control system of a smart hub, a control system of another device of an audio system, etc.

In the examples shown inFIG.6, the audio environment in which the noise compensation system710resides also includes a listener616(who is assumed to be stationary in this example) and a noise source615. The location of the listener616may, in some examples, be presumed to be the same as, or in close proximity to, the location of a remote control. In this instance, the noise source615is closer to the listener616than the television611. The type and location of the noise source615are merely shown by way of example.

As with other figures provided herein, the types, numbers and arrangements of elements shown inFIGS.6-7Aare merely provided by way of example. Other implementations may include more, fewer and/or different types, numbers or arrangements of elements, e.g., more loudspeakers and/or more microphones, more or fewer operations, etc.

FIG.6shows examples of acoustic propagation paths707a,707band707cfrom the noise source615to the microphones702. In this example, the acoustic propagation paths707a,707band707chave different lengths and therefore different arrival times at each microphone. In the example shown inFIG.67, no synchronization is required between the multiple devices, because the microphones702a,702band702care part of the television711and are controlled by the same control system.

According to some examples, the cross-correlation function of the recorded ambient noise from the microphones702a,702band702cmay be computed to determine the time difference of arrival between microphones. The path length difference is the time difference (seconds) multiplied by the speed of sound (meters per second). Based on the path length difference, the distance from the listener616to the television711and the known distance between the microphones702a,702band702c, the location of the noise source615can be solved. In some examples, the location of the noise source615may be calculated using a two-dimensional (2D) hyperbolic position location algorithm, such as one of the methods described in Chapter 1.21, 1.22, 2.1 or 2.2 of Dalskov, D.,Locating Acoustic Sources with Multilateration-Applied to Stationary and Moving Sources, (Aalborg University, Jun. 4, 2014), which are hereby incorporated by reference. A specific example of one alternative solution is described below with reference toFIGS.7A and7B.

FIG.7Ais an example of a graph that indicates signals received by the microphones shown inFIG.6. In this example,FIG.7Ashows an example correlation analysis of the three microphones in order to determine the time difference of arrival (TDOA) of the noise source615at microphones702aand702crelative to central microphone702b. According to this example, the elements ofFIG.7Aare as follows:712arepresents the cross correlation of microphone702aand reference microphone702b;712brepresents the autocorrelation of reference microphone702b;712crepresents the cross correlation of microphone702cand reference microphone702b;713ais a peak in the cross correlation from which the TDOA of microphone702awith respect to reference microphone702bis determined. It can be seen in this example that sound arrives at microphone702abefore reference microphone702b, yielding a negative TDOA for microphone702a;713bis a peak in the auto correlation of reference microphone702b. In this example, we define time 0 to be the location of this peak. In some alternative embodiments, the autocorrelation function712bcan be deconvolved with the cross-correlation functions712aand712cprior to estimating TDOA in order to create sharper peaks;713cis a peak in the cross correlation from which the TDOA of microphone702cwith respect to reference microphone702bmay be determined. It can be seen in this example that sound arrived at microphone702cafter reference microphone702b, yielding a positive TDOA for microphone702c;714ais visual representation of the TDOA for microphone702awith respect to reference microphone702b. Mathematically we will treat TDOA714aas a negative quantity in this example because sound arrives at microphone702abefore it arrives at microphone702b; and714bis a visual representation of the TDOA for microphone702cwith respect to reference microphone702b. This TDOA will be a positive quantity in this example.FIG.7Bshows the noise source ofFIG.6in a different location of the audio environment. In this example, the arrangement shown inFIG.6has been redrawn to emphasize the geometric nature of the problem and to label the lengths of the sides of each triangle. In this example, the noise source615is shown coming from the right of the diagram rather than the left as depicted inFIG.6andFIG.7A. This is so that x coordinate of the noise source720ais a positive quantity, to aid in clearly defining the coordinate system.

In the example shown inFIG.7B, the elements are as follows:615represents the noise source to be located;702a-crepresent the locations of the three microphones shown inFIG.6. Here, reference microphone702bis shown to be the origin of a two-dimensional cartesian coordinate system;720arepresents the x coordinate (in metres) of the noise source615relative to the origin centred on reference microphone702b;720brepresents the y coordinate (in metres) of the noise source615relative to the origin centred on reference microphone702b;721arepresents the distance between microphone702aand microphone702b. In this example, microphone702ais positioned on the television d metres to the left of reference microphone702b. In one example, d=0.4 m;721brepresents the distance between microphone702band microphone702c. In this example, microphone702ais positioned on the television d metres to the right of reference microphone702b;722represents the noise source615projected on the x axis of the cartesian coordinate system;707a-crepresent the acoustic path lengths (in metres) from the noise source615to each of the microphones702a-c;708bcorresponds to the symbol r, which we define to mean the distance, in metres, from the noise source615to reference microphone702bin this example;708acorresponds to the sum of the symbols r+a. In this example, we define the symbol a to mean the path length difference between707aand707b, so that the length of acoustic path707ais r+a. The acoustic path length r+a can be computed from the TDOA of microphone702awith respect to microphone702b(see714ainFIG.7A, which was negative in that example, but positive in this example) by multiplying the TDOA by the speed of sound in the medium. For example, if the TDOA714ais +0.0007 s and the speed of sound is 343 metres/second, then a=0.2401 m;708ccorresponds to the sum of the symbols r+b. In this example, we define the symbol b to mean the path length difference between707cand707b, so that the length of acoustic path707cis r+b. The acoustic path length r+b can be computed from the TDOA of microphone702cwith respect to microphone702b(see714cinFIG.7A, which was positive in that example, but negative in this example) by multiplying the TDOA by the speed of sound in the medium. For example, if the TDOA714cis −0.0006 s and the speed of sound is 343 metres/second, then b=−0.2058 m. In some implementations, a control system may be configured to determine a more precise speed of sound for the audio environment according to input from a temperature sensor.

We now write Pythagoras' Theorem for triangle (702b,615,722):
r2=x2+y2Equation 2

Pythagoras' Theorem for triangle (702a,615,722) may be written as follows:
(r+a)2=(x+d)2+y2Equation 3

Pythagoras' Theorem for triangle (702c,615,722) may be written as follows:
(r+b)2=(x−d)2+y2Equation 4

Together, Equations 2, 3 and 4 form a system of three simultaneous equations in unknowns r, x, y. We are particularly interested to know r, the distance in metres from the noise source to reference microphone702b.

This system of equations may be solved for r as follows:
r=−(a2+b2−2d2)/(2(a+b))  Equation 5

For the example values given above:
a=0.2401 m,b=−0.2058 m,d=0.4 m

we can conclude that r=3.206 m. Therefore, the noise source615lies approximately 3.2 m from the reference microphone702bin this example.

In addition to estimating a noise source location, some implementations may involve determining or estimating a listener position. Referring again toFIG.6, the distance from the listener616to the television611may be estimated or determined in different ways, depending on the particular implementation. According to some examples, the distance from the listener616to the television611may be determined during an initial setup of the television611by the listener616or by another user. In other examples, the location of the listener161and/or the distance from the listener616to the television611may be determined according to input from one or more sensors, e.g., according to input from one or more cameras, according to input from additional microphones or according to input from other sensors of the sensor system230that is described above with reference toFIG.2. In other examples, in the absence of user input or sensor input, the distance from the listener616to the television611may be determined according to a default distance, which may be an average distance from a typical listener to a television. In some examples, it may be assumed that the listener is within a certain angle from a normal to the television screen, e.g., within 10 degrees, within 15 degrees, within 20 degrees, within 25 degrees, within 30 degrees, etc.

According to some implementations, a noise compensation may be based, at least in part, on a determined or estimated listener location and a determined or estimated noise source location. For example, by knowing where the listener616is (or assuming where the listener616is relative to the television711) and knowing the location of the noise source615and the corresponding noise level at the television711, a noise estimate for the location of the listener616can be calculated using a propagation loss model. This predicted noise compensation value for the listener's location may be used directly by a noise compensation system.

In some alternative implementations, the predicted noise level at the listener position may be further modified to include a confidence value. For example, if the noise source is relatively far away the listener position (or from a plurality of the most likely listener positions and doesn't have a large variation in the predicted noise estimates between the most likely listener positions), then the noise estimate will have a high confidence. Otherwise the noise estimate may have a lower confidence. The list of likely listener positions may change depending on the context of the system. Furthermore, according to some examples the noise estimate confidence can be further augmented if there are multiple microphones measuring the noise level, potentially at various locations of the audio environment. If the measured noise levels at various locations of the audio environment are all consistent with the propagation loss model, this can provide a higher confidence for the noise estimate than if the measured noise levels at the various locations are inconsistent with the propagation loss model.

If the noise compensation system has a high confidence in the noise estimate for the listener location, in some implementations the noise compensation system may be configured to implement an unconstrained noise compensation method. Alternatively the noise compensation system may implement a more constrained noise compensation method if the noise compensation system has a low confidence in the noise estimate for the listener location.

FIG.8shows an example of a floor plan of an audio environment, which is a living space in this example. As with other figures provided herein, the types, numbers and arrangements of elements shown inFIG.8are merely provided by way of example. Other implementations may include more, fewer and/or different types, numbers or arrangements of elements.

According to this example, the environment800includes a living room810at the upper left, a kitchen815at the lower center, and a bedroom822at the lower right. Boxes and circles distributed across the living space represent a set of loudspeakers805a-805h, at least some of which may be smart speakers in some implementations, placed in locations convenient to the space, but not adhering to any standard prescribed layout (arbitrarily placed). In some examples, the television830may be configured to implement one or more disclosed embodiments, at least in part. In this example, the environment800includes cameras811a-811e, which are distributed throughout the environment. In some implementations, one or more smart audio devices in the environment800also may include one or more cameras. The one or more smart audio devices may be single purpose audio devices or virtual assistants. In some such examples, one or more cameras of the optional sensor system130may reside in or on the television830, in a mobile phone or in a smart speaker, such as one or more of the loudspeakers805b,805d,805eor805h. Although cameras811a-811eare not shown in every depiction of the environment800presented in this disclosure, each of the environments800may nonetheless include one or more cameras in some implementations.

Some aspects of present disclosure include a system or device configured (e.g., programmed) to perform one or more examples of the disclosed methods, and a tangible computer readable medium (e.g., a disc) which stores code for implementing one or more examples of the disclosed methods or steps thereof. For example, some disclosed systems can be or include a programmable general purpose processor, digital signal processor, or microprocessor, programmed with software or firmware and/or otherwise configured to perform any of a variety of operations on data, including an embodiment of disclosed methods or steps thereof. Such a general purpose processor may be or include a computer system including an input device, a memory, and a processing subsystem that is programmed (and/or otherwise configured) to perform one or more examples of the disclosed methods (or steps thereof) in response to data asserted thereto.

Some embodiments may be implemented as a configurable (e.g., programmable) digital signal processor (DSP) that is configured (e.g., programmed and otherwise configured) to perform required processing on audio signal(s), including performance of one or more examples of the disclosed methods. Alternatively, embodiments of the disclosed systems (or elements thereof) may be implemented as a general purpose processor (e.g., a personal computer (PC) or other computer system or microprocessor, which may include an input device and a memory) which is programmed with software or firmware and/or otherwise configured to perform any of a variety of operations including one or more examples of the disclosed methods. Alternatively, elements of some embodiments of the inventive system are implemented as a general purpose processor or DSP configured (e.g., programmed) to perform one or more examples of the disclosed methods, and the system also includes other elements (e.g., one or more loudspeakers and/or one or more microphones). A general purpose processor configured to perform one or more examples of the disclosed methods may be coupled to an input device (e.g., a mouse and/or a keyboard), a memory, and a display device.

Another aspect of present disclosure is a computer readable medium (for example, a disc or other tangible storage medium) which stores code for performing (e.g., coder executable to perform) one or more examples of the disclosed methods or steps thereof.

While specific embodiments of the present disclosure and applications of the disclosure have been described herein, it will be apparent to those of ordinary skill in the art that many variations on the embodiments and applications described herein are possible without departing from the scope of the disclosure described and claimed herein. It should be understood that while certain forms of the disclosure have been shown and described, the disclosure is not to be limited to the specific embodiments described and shown or the specific methods described.