Acoustical listening area mapping and frequency correction

A smart speaker device for acoustical listening area mapping and frequency correction includes a non-transitory storage configured to maintain a listening area response map indicating filter settings corresponding to each of a plurality of locations within a listening area, a microphone array, a loudspeaker; and a controller. The controller is programmed to execute a frequency correcting application to identify a current location of a mobile device in the listening area based on ultrasonic audio received to the microphone array from the mobile device, access the listening area response map to retrieve filter settings corresponding to the current location, and apply the filter settings to an audio stream to be output to the loudspeaker to correct for frequency response of the loudspeaker at the current location of the mobile device.

TECHNICAL FIELD

Aspects of the disclosure generally relate to acoustical listening area mapping and frequency correction.

BACKGROUND

Due to room modes and other acoustic effects, frequency response of a speaker in a room or other listening area can vary greatly as the listener moves around. These deviations in frequency response can cause large differences in perceived balance of the speaker, as well as boomy resonances at various frequencies.

SUMMARY

In one or more illustrative examples, a smart speaker device for acoustical listening area mapping and frequency correction includes a non-transitory storage configured to maintain a listening area response map indicating filter settings corresponding to each of a plurality of locations within a listening area, a microphone array, a loudspeaker, and a controller. The controller is programmed to execute a frequency correcting application to identify a current location of a mobile device in the listening area based on ultrasonic audio received to the microphone array from the mobile device, access the listening area response map to retrieve filter settings corresponding to the current location, and apply the filter settings to an audio stream to be output to the loudspeaker to correct for frequency response of the loudspeaker at the current location of the mobile device.

In one or more illustrative embodiments, a smart speaker device for acoustical listening area mapping and frequency correction includes a non-transitory storage configured to maintain a listening area response map indicating filter settings corresponding to each of a plurality of locations within a listening area, a microphone array, a loudspeaker, and a controller. The controller is programmed to execute a frequency correcting application to identify a current location of a mobile device in the listening area based on ultrasonic audio received to the microphone array from the mobile device, output frequency test audio from the loudspeaker to be received by the mobile device, receive, from the mobile device, information indicative of room response at the current location, generate a room correction for the current location according to the information indicative of the room response, the room correction indicating filter settings for the current location, and update the listening area response map to indicate the filter settings as corresponding to the current location.

In one or more illustrative embodiments, a method for acoustical listening area mapping and frequency correction includes identifying a current location of a mobile device in a listening area based on ultrasonic audio received to a microphone array of a smart speaker device from the mobile device; accessing a listening area response map stored to a memory of the smart speaker device to retrieve filter settings corresponding to the current location, the listening area response map indicating filter settings corresponding to each of a plurality of locations within a listening area; and applying the filter settings to an audio stream to be output to a loudspeaker of the smart speaker device to correct for frequency response of the loudspeaker at the current location of the mobile device.

DETAILED DESCRIPTION

Cell phones are capable of producing audio frequencies in the ultrasonic region. This is evidenced by the fact that young kids have been known to use specialized ultrasonic ring tones that adults cannot hear. A smart speaker may utilize a microphone array to better locate users and adaptively beam-form to increase signal to noise for speech recognition. These arrays could be used to locate a person on a continuous basis (e.g., not only while they are speaking) via ultrasound.

An application installed to a user's phone or other mobile device may be programmed to cause the device to emit short ultrasound pulses at short intervals. An application installed to the smart speaker may then monitor these signals and determine the user's location via triangulation using the microphone array. The ultrasonic signal may be well-suited to precise determination of arrival times due to its short wavelength. In situations where the ultrasound is occluded by objects and/or the user's own body, the smart speaker may default to generic (unspecified) location equalization. To avoid audibility of the ultrasound signal, the emitted pulses may be very short, and only emitted when music or other audio is being played (to mask the sound). Further, the pulses may be emitted responsive to detected movement of the mobile device, so if there is no change in position the locating sounds may not be required.

During setup of the smart speaker in a listening area, the application installed to the smart speaker may be programmed to cause the smart speaker to emit a low-frequency test signal using one or more loudspeakers of the smart speaker device. A connected measurement application on the mobile device measures the low frequency response of the speaker in the listening area, as the user moves the mobile device to various locations that the user is likely to occupy in the listening area. Simultaneously, the above described triangulation method may be used to locate the user and create a “map” of the listening area, i.e., a low frequency response for each location in the listening area (or at least each location that the user is likely to be in). As one possible optimization, during the learning phase the user may spend more time in the locations that he/she is more likely to inhabit, thus weighting the generic solution to be a better compromise. Once the learning is complete, a corresponding correction map may be calculated by the smart speaker, which results in optimized low frequency response at all locations in the listening area. The smart speaker may also calculate a weighted average of the most likely positions and use that to make the best possible correction for instances where location of the user by the smart speaker is inconclusive.

At runtime, the ultrasonic triangulation component runs, allowing the smart speaker to know where the user currently is located in the listening area. Using the previously-generated listening area correction map from the learning phase, the smart speaker may determine the best correction to be applied. This filter may be applied in real-time to whatever the user is listening to on the smart speaker. If the person moves, the filter may be updated, and the update may be performed gradually to avoid detection. In instances where triangulation is not working or produces inconclusive results, perhaps due to occlusion of the source or other reason, the listening area correction defaults to a generic solution which is based on the measurements at all locations. Thus, optimization of smart speaker frequency response can be performed for a user to allow for optimized and constant sound as the user moves about the listening area, without requiring additional hardware be added to the smart speaker or mobile device.

FIG. 1illustrates a system100including a smart speaker102and a mobile device126, configured for acoustical listening area mapping and frequency correction. The smart speaker102receives audio through a microphone array104or other audio input, and passes the audio through an analog to digital (A/D) converter106to be identified or otherwise processed by an audio processor108. The audio processor108also generates audio output, which may be passed through a digital to analog (D/A) converter112and amplifier114for reproduction by one or more loudspeakers116of the smart speaker102. The smart speaker102also includes a controller118connected to the audio processor108configured to maintain a listening area response map152and execute a frequency correcting application158. Based on the input audio by the audio processor108, the controller118in a learning mode uses the loudspeakers116to play frequency test audio154to be received by the mobile device126and create the listening area response map152of equalizations for corresponding locations responsive to results of the frequency test included in a wireless signal received to a wireless transceiver124of the smart speaker102from the mobile device126. In a playback mode, the controller118determines the current location of the user responsive to receipt of a high-frequency audio output156received from the mobile device126and directs the audio processor108to filter the audio signal being played back in accordance with the predetermined equalization settings for the current location identified from the listening area response map152.

The mobile device126receives audio through a microphone128of the mobile device126, and passes the audio through an A/D converter130to be identified or otherwise processed by an audio processor134. The audio processor134also generates audio output, which may be passed through a D/A converter136and amplifier138for reproduction by one or more loudspeakers140of the mobile device126. The mobile device126also includes a controller142connected to the audio processor134configured to execute a frequency correcting application158to determine listening area response based on the frequency test audio154, provide the results of the frequency test in the wireless signal provided by the wireless transceiver148. The controller142may also indicate the location of the mobile device126according to high-frequency audio output156sent using the loudspeakers140of the mobile device126. It should be noted that the illustrated system100is merely an example, and more, fewer, and/or differently located elements may be used.

More specifically, the microphone array104may include a plurality of microphone elements arranged such that sounds in the listening area may reach the microphone elements at different times. These differences in timing may be used to determine a direction from which the sounds were received. The A/D converter106receives audio input signals from the microphone array104. The A/D converter106converts the received signals from an analog format into a digital signal in a digital format for further processing by the audio processor108.

While only one is shown, one or more audio processors108may be included in the smart speaker102. The audio processors108may be one or more computing devices capable of processing audio and/or video signals, such as a computer processor, microprocessor, a digital signal processor, or any other device, series of devices or other mechanisms capable of performing logical operations. The audio processors108may operate in association with a memory110to execute instructions stored in the memory110. The instructions may be in the form of software, firmware, computer code, or some combination thereof. The memory110may be any form of one or more data storage devices, such as volatile memory, non-volatile memory, electronic memory, magnetic memory, optical memory, or any other form of data storage device. In addition to instructions, operational parameters and data may also be stored in the memory110.

The audio processor108may also be configured to provide an audio output signal including media content or other audio to be provided from the smart speaker102. The audio processor108may also filter the audio output in accordance with filter settings received to the audio processor108. The D/A converter112receives the digital output signal from the audio processor108and converts it from a digital format to an output signal in an analog format. The output signal may then be made available for use by the amplifier114or other analog components for further processing.

The amplifier114may be any circuit or standalone device that receives audio input signals of relatively small magnitude, and outputs similar audio signals of relatively larger magnitude. Audio input signals may be received by the amplifier114and output on one or more connections to the loudspeakers116. In addition to amplification of the amplitude of the audio signals, the amplifier114may also include signal processing capability to shift phase, adjust frequency equalization, adjust delay or perform any other form of manipulation or adjustment of the audio signals in preparation for being provided to the loudspeakers116. As noted above, the signal processing functionality may additionally or alternately occur within the domain of the audio processor108. Also, the amplifier114may include capability to adjust volume, balance and/or fade of the audio signals provided to the loudspeakers116. In an alternative example, the loudspeakers116may include the amplifier114, such that the loudspeakers116are self-powered.

The loudspeakers116may be of various sizes and may operate over various ranges of frequencies. Each of the loudspeakers116may include a single transducer, or in other cases multiple transducers. The loudspeakers116may also be operated in different frequency ranges such as a subwoofer, a woofer, a midrange, and a tweeter. Multiple loudspeakers116may be included in the smart speaker102.

The controller118may include various types of computing apparatus in support of performance of the functions of the smart speaker102described herein. In an example, the controller118may include one or more processors120configured to execute computer instructions, and a storage medium122on which the computer-executable instructions and/or data may be maintained. A computer-readable storage medium (also referred to as a processor-readable medium or storage122) includes any non-transitory (e.g., tangible) medium that participates in providing data (e.g., instructions) that may be read by a computer (e.g., by the processor(s)120). In general, a processor120receives instructions and/or data, e.g., from the storage122, etc., to a memory and executes the instructions using the data, thereby performing one or more processes, including one or more of the processes described herein. Computer-executable instructions may be compiled or interpreted from computer programs created using a variety of programming languages and/or technologies including, without limitation, and either alone or in combination, JAVA, C, C++, C#, ASSEMBLY, FORTRAN, PASCAL, VISUAL BASIC, PYTHON, JAVA SCRIPT, PERL, PL/SQL, etc.

As shown, the controller118may include a wireless transceiver124or other network hardware configured to facilitate communication between the controller118and other networked devices. As one possibility, the wireless transceiver124may be a Wi-Fi transceiver configured to connect to a local-area wireless network to access a communications network. As another possibility, the wireless transceiver124may be a cellular network transceiver configured to communicate data over a cellular telephone network.

On the mobile device126, the microphone128may provide signals based on received audio to the A/D converter130for conversion from an analog format into a digital signal for further processing by the audio processor134. While only one is shown, one or more audio processor134may be included in the mobile device126. As with the audio processors108, the audio processor134may be one or more computing devices capable of processing audio and/or video signals, such as a computer processor, microprocessor, a digital signal processor, or any other device, series of devices or other mechanisms capable of performing logical operations. The audio processors108may operate in association with a memory132to execute instructions stored in the memory132. The instructions may be in the form of software, firmware, computer code, or some combination thereof. The memory132may be any form of one or more data storage devices, such as volatile memory, non-volatile memory, electronic memory, magnetic memory, optical memory, or any other form of data storage device. In addition to instructions, operational parameters and data may also be stored in the memory132.

The audio processor134may also be configured to provide an audio output signal including media content or other audio to be provided from the mobile device126. The D/A converter136receives the digital output signal from the audio processor134and converts it from a digital format to an output signal in an analog format. Similar to as discussed with elements114and116of the smart speaker102, the output signal may then be made available for use by the amplifier138or other analog components for further processing and output by the loudspeakers140.

The controller142may include various types of computing apparatus in support of performance of the functions of the mobile device126described herein. In an example, the controller142may include one or more processors144configured to execute computer instructions, and a storage medium146on which the computer-executable instructions and/or data may be maintained. As shown, the controller142also includes a wireless transceiver148or other network hardware configured to facilitate communication between the controller142and other networked devices such as the smart speaker102.

The mobile device126may also include a human machine interface (HMI)150. In some examples, the HMI150may include a touchscreen display that may be used to display information and also receive user input. The HMI150may also include other controls and/or displays that may be used to receive user input and provide input to a user.

The listening area response map152is a data structure configured to store equalization information corresponding to locations within a listening area. For instance, the listening area response map152may indicate a low frequency response for each of a plurality of locations in the listening area. Additionally, or alternately, the listening area response map152may include equalization or other filter settings that may be used to correct for the low frequency response indexed to each of a plurality of locations in the listening area. The listening area response map152may be stored to the storage122of the smart speaker102.

The frequency test audio154is an audio output provided by the loudspeakers116of the smart speaker102based on a frequency test signal. The frequency test signal may be a sweep, test tones, or other test signal that may be used to determine the in-room frequency response of the loudspeakers116at a measurement location.

The high-frequency audio output156is a high-frequency audio output provided by the loudspeakers140of the mobile device126. The high-frequency audio output156may be provided in the form of one or more pulses, bursts, chirps, frequency sweeps, or other forms of audio output that may be used to determine an origination location of the high-frequency audio output156. In many examples, the high-frequency audio output156may be at an ultrasonic frequency or frequencies above the hearing range of typical humans, so as to be playable without being perceived by listeners. In some cases, the high-frequency audio output156may be added to existing audio output of the loudspeakers140so as to disguise the sound of the high-frequency audio output156.

The frequency correcting application158is an example of an application installed to the storage122of the smart speaker102. When executed by the smart speaker102, the frequency correcting application158may be programmed to cause the smart speaker102to perform operations of a learning mode in which the listening area response map152is created for a listening area, and of a playback mode in which the listening area response map152is used to filter the output of the smart speaker102. Further aspects of the operation of the frequency correcting application158are described with respect toFIGS. 3 and 5.

The listener application160is an example of an application installed to the storage146of the mobile device126. When executed by the mobile device126, the listener application160may be programmed to cause the mobile device126to perform operations of a learning mode in which frequency measurements are made based on reception of the frequency test audio154at the microphone128of the mobile device126as well as the transmission of a signal from the mobile device126to the smart speaker102including the frequency measurements. The listener application160may also be programmed to cause the mobile device126to play the high-frequency audio output156via the loudspeaker140for reception by the microphone array104of the smart speaker102to allow the smart speaker102to locate the mobile device126in the listening area. Further aspects of the operation of the listener application160are described with respect toFIGS. 2 and 4.

FIG. 2illustrates an example process200for operation of the mobile device126in generation of the listening area response map152. In an example, the process200may be performed by execution by the mobile device126of the listener application160in a learning mode. For instance, the listener application160may transition to a learning mode responsive to user input to the HMI150of the mobile device126(e.g., via a menu selection), or responsive to a command to begin learning received by the mobile device126from the smart speaker102(e.g., received wirelessly via a wireless signal over WiFi or another protocol from the wireless transceiver148of the mobile device126to the wireless transceiver124of the smart speaker102, encoded in an audio format and provided by the loudspeaker116to be received by the microphone128and interpreted by the mobile device126).

At operation202, the mobile device126sends a request to the smart speaker102to play the frequency test audio154via the loudspeakers116of the smart speaker102. In an example, the request may be sent as a wireless signal over WiFi or another protocol from the wireless transceiver148of the mobile device126to the wireless transceiver124of the smart speaker102. In another example, the request may be encoded in an audio format, and may be sent from the loudspeaker140to be received by the microphone array104of the smart speaker102. In yet a further example, if the mobile device126is in the learning mode, the mobile device126may listen for the frequency test audio154and may analyze the signal once received without sending an additional request to the smart speaker102.

At204, the mobile device126measures the listening area response at the location of the mobile device126. In an example, the frequency test audio154is received by the microphone128of the mobile device126, which is used to record amplitude measurements for the frequencies of audio included in the frequency test audio154provided by the smart speaker102. For instance, these measurements may be used to identify the low-frequency response characteristics of the location of the listening area at which the mobile device126is currently located.

The mobile device126sends the listening area response to the smart speaker102at206. In an example, the listening area response may be sent as a wireless signal over WiFi or another protocol from the wireless transceiver148of the mobile device126to the wireless transceiver124of the smart speaker102. In another example, the listening area response may be encoded in an audio format, and may be sent from the loudspeaker140to be received by the microphone array104of the smart speaker102.

At operation208, the mobile device126sends the high-frequency audio output156to be received by the smart speaker102. In an example, the mobile device126may utilize the loudspeaker140to send the high-frequency audio output156to be picked up by the microphone array104of the smart speaker102, to allow the smart speaker102to attempt to locate the mobile device126within the listening area. In some cases, the high-frequency audio output156is explicitly provided by the mobile device126in a predefined manner prior to, concurrent with, and/or after the sending of the listening area response data to the smart speaker102. In another example, the high-frequency audio output156is provided by the mobile device126periodically, independent of the transmission of the listening area response to the smart speaker102.

The mobile device126determines whether to learn listening area response data for an additional location at210. In an example, the listener application160may provide a prompt to the HMI150of the mobile device126asking the user of the mobile device126whether the user has other locations within the listening area to measure. If the HMI150receives input indicating that additional locations are to be measured, control returns to operation202. If not, the mobile device126may inform the smart speaker102that learning is complete (e.g., via wireless signal or audio communication), and the process200ends.

FIG. 3illustrates an example process300for operation of the smart speaker102in generation of the listening area response map152. In an example, the process300may be performed by execution by the smart speaker102of the frequency correcting application158in a learning mode.

At operation302, the smart speaker102provides a frequency test signal as an audio output. In an example, the smart speaker102plays the frequency test audio154via the loudspeakers116of the smart speaker102responsive to receipt of the request at operation202. In another example, the smart speaker102plays the frequency test audio154responsive to an indication to check response at a different location (e.g., such as discussed at operation210), or automatically responsive to entering learning mode).

The smart speaker102receives room response information from the mobile device126at operation304. In an example, the smart speaker102receives the information sent at operation206of the process200.

At306, the smart speaker102identifies a location of the mobile device126. In an example, the smart speaker102receives the high-frequency audio output156sent at operation208of the process200, and uses the high-frequency audio output156to determine a location of the mobile device126. For instance, the smart speaker102may utilize time and phase differences in the signals received from each of the microphones of the microphone array104to calculate an angle of incidence of received audio to the microphone array104. It should be noted that identifying the actual location of the mobile device126within the listening area is not critical. Instead, it is more important for the location determination to be repeatable, so that the mapping of the location can be used to identify later instances where the mobile device126is at the same location.

At operation308, the smart speaker102generate a room correction for the identified location of the mobile device126. In an example, the room correction may be determined as filters to be applied to audio output to reduce nonlinearities in response in the room response information received at operation304. As one possibility, the room correction may be determined as an equalization in the form of an inverse of the differences in the room response information compared to a target response (e.g., a flat response, a target equalization, etc.). As another possibility, the room correction may be determined as a set of one or more parametric filters, which each include a frequency center point, a gain (positive or negative), and a Q which determines how wide or narrow the filter is. For instance, the frequency correcting application158may define one or more parametric EQ settings using an algorithm designed to minimize difference between the measured response and the target response.

At310, the smart speaker102updates the listening area response map152. In an example, the smart speaker102saves the room correction determined at operation308indexed according to the location determined at operation306. After operation310, the process300ends.

FIG. 4illustrates an example process400for the operation of the mobile device126to send location updates to the smart speaker102. In an example, the process400may be performed by execution by the mobile device126of the listener application160in a playback mode.

At402, the mobile device126determines whether the mobile device126is in playback mode. In an example, the playback mode may be entered responsive to a user of the mobile device126requesting (e.g., via the HMI150) for the mobile device126to play back audio content. The payback mode may be exited responsive to completion of the playback. In another example, the mobile device126may determine the smart speaker102to be in playback mode if the smart speaker102is not identified as being in the learning mode discussed in detail above. If the smart speaker102is in playback mode, control passes to operation404. Otherwise, control remains at operation402.

At operation404, the mobile device126determines whether an event occurred to cause the mobile device126to send a location update. For instance, the listener application160may send location updates periodically, and may accordingly determine to send an update responsive to expiration of a timer. In another example, the listener application160may additionally or alternately send location updates responsive to identifying movement of the mobile device126. For instance, the mobile device126may include one or more accelerometers that provide signals indicative of acceleration of the mobile device126in one or more directions. If one or more such events have occurred, control passes to operation406.

At406, similar to as discussed above with respect to operation208of the process200, the mobile device126sends the high-frequency audio output156to be received by the smart speaker102. This update may be used to allow the smart speaker102to track the location of the mobile device126and therefore the location of the user of the mobile device126. After operation406, the process400continues to operation402.

FIG. 5illustrates an example process500for the operation of the smart speaker102to filter audio output in accordance with the location of the mobile device126. In an example, the process300may be performed by execution by the smart speaker102of the frequency correcting application158in a playback mode.

At502, the smart speaker102determines whether the smart speaker102is in playback mode. In an example, the playback mode may be entered responsive to a user of the mobile device126requesting (e.g., via the HMI150) for the mobile device126to play back audio content. The payback mode may be exited responsive to completion of the playback. In another example, the smart speaker102be in playback mode if the smart speaker102is not identified as being in the learning mode discussed in detail above. If the smart speaker102is in playback mode, control passes to operation504. Otherwise, control remains at operation502.

The smart speaker102identifies a location of the mobile device126at504. In an example, the high-frequency audio output156as received by the microphone array104of the smart speaker102may be compared with mapped locations of the listening area response map152saved using a process such as the processes200and300.

At operation506, the smart speaker102retrieves filter parameters for the listening location of the mobile device126. For example, if a matching location is identified at operation502, then filter settings for that location are retrieved from the listening area response map152. If, however, a match is not identified, then other settings for the location may be used. For instance, the smart speaker102may utilize an average of the filter parameters across all locations of the listening area response map152.

At508, the smart speaker102applies the filter parameters for the listening location to an audio stream. At510, the smart speaker102provides the audio stream to loudspeakers116of the smart speaker102to generate audio output. Accordingly, the audio output of the smart speaker102may be filtered according to the current location of the mobile device126. After operation508, control returns to operation502.

FIG. 6illustrates an example diagram600of the smart speaker102filtering audio output for a mobile device126at a first location in a listening area. As shown, the mobile device126is located at the first location in the listening area and sends the high-frequency audio output156that allows the smart speaker102to identify the mobile device126as being located at the first location. Responsive to the identification, the smart speaker102uses the listening area response map152to filter audio output602provided by the smart speaker102in accordance with the filtering associated with the first location.

FIG. 7illustrates an example diagram700of the smart speaker102filtering audio output for a mobile device126at a second location in a listening area. As shown, the mobile device126is now located at the second location in the listening area. Using the high-frequency audio output156from the mobile device126, the smart speaker102now identifies that the mobile device126as located at the second location, and uses the listening area response map152to filter the audio output602in accordance with the filtering associated with the second location.

FIG. 8illustrates an example diagram800of the smart speaker102filtering audio output for multiple mobile devices126A and126B. Notably, the mobile device126A is located at the first location, while the mobile device126B is located at the second location. Using the high-frequency audio output156A from the mobile device126A and the high-frequency audio output156B from the mobile device126B, the smart speaker102identifies that the mobile device126A is located at the first location and the mobile device126B is located at the second location. Accordingly, since the smart speaker102cannot apply both the filter for the first location and the filter for the second location simultaneously, the smart speaker102may instead provide a combined filter, such as an average of the equalization for the first and second locations. Or, the smart speaker102may provide a default equalization, which, for example, may be an average of all equalizations for the listening are as recorded in the response map152.

Other variations on the system100are possible as well. For instance, in determining an average equalization, the system100may weigh the equalizations for the amount of time that a user spends in various locations within the listening area when determining an average equalization. For instance, if a user spends 60% of his time at one location and 40% at a second location, if the location of the user cannot be determined, then the smart speaker102may utilize an average equalization that is a weighted average that is ⅗ the equalization of the first location and ⅖ the equalization of the second area.