Patent Description:
The present disclosure relates to improvements for the audio playback of media content. In particular, this disclosure relates setting and applying preferred noise compensation for the audio of media content being played in a variety of environments, particularly on mobile devices.

Audio playback for media with dialog (movies, televisions shows, etc.) are normally created to be enjoyed in a relatively quiet environment, such as at home or in a theater. However, it is becoming increasingly common for people to consume such content on-the-go with their mobile devices. This becomes an issue because it can be difficult to make out what the actors are saying when there is too much ambient noise (vehicle noise, crowds, etc.) or due to audio quality limitations of mobile hardware or type of audio playback (headphones, etc.) equipment used.

A common solution is to use noise-cancelling headphones/earbuds. However, this can be an expensive solution and has the downside of cutting out ambient noise that the user might want to hear (car horns, sirens, yelled warnings, etc.). <CIT> relates to speech intelligibility, in which the perceived quality of a speech signal output from a user apparatus is improved by storing ambient noise profiles each indicating a model power distribution of a respective ambient noise type as a function of frequency; the ambient noise profile at the user apparatus is measured, the measured ambient noise profile is correlated with each of the stored ambient noise profiles, the stored ambient noise profile is selected with which the measured ambient noise profile is most highly correlated, and the speech signal is manipulated in dependence on which of the stored ambient noise profiles is selected, so as to form an improved speech signal. <CIT> relates to systems and methods disclosed therein provide for low cost hearing assistance to improve intelligible hearing for those with normal hearing and to greatly improve hearing intelligibility for those with hearing problems. One goal of the systems and methods disclosed therein is to make hearing assistance algorithms easily accessible and available by implementing such algorithms using non-dedicated hardware platforms such as non-dedicated mobile computing devices, e.g., smartphones, PDA's and the like. In exemplary embodiments, the systems and method of the disclosure integrate hearing assistance algorithms with multimedia algorithms in an API stack (similar to the implementation of audio effects such as stereo widening and psychoacoustic bass enhancement) thereby addressing processing delay concerns.

Various audio processing systems and methods are disclosed herein. Some such systems and methods involve creating and using audio adjustment profiles that are customized for the user and specific to different ambient conditions.

According to a first aspect, a method of configuring a mobile device for use with ambient noise for a user of the mobile device is described, the method comprising: receiving, from the user, a case identification of the ambient noise; receiving, from the user, a noise level for the ambient noise; receiving, from the user, a dialog boost level for the ambient noise at the noise level; receiving, from the user, a graphic equalizer setting for the ambient noise at the noise level; playing, from the mobile device, a sample audio for the user while the user sets the dialog boost level and the graphic equalizer setting; and storing on the mobile device the dialog boost level and graphic equalizer setting for the case identification at the noise level in a profile, wherein the device is configured to play audio media using the dialog boost level and graphic equalizer setting when the profile is selected by the user.

According to a second aspect, a method of adjusting audio for a mobile device for a user is described, the method comprising: receiving a profile selection from the user wherein the profile selection relates at least to an ambient noise condition; receiving a noise level of the ambient noise condition from the user; retrieving a dialog boost level and a graphic equalizer setting from memory on the mobile device; adjusting levels of the audio using the dialog boost level and the graphic equalizer setting.

Some or all of the methods described herein may be performed by one or more devices according to instructions (e.g. software) stored on one or more non-transitory media. Such non-transitory media may include memory devices such as those described herein, including but not limited to random access memory (RAM) devices, read-only memory (ROM) devices, etc. Accordingly, various innovative aspects of the subject matter described in this disclosure may be implemented in a non-transitory medium having software stored thereon. The software may, for example, be executable by one or more components of a control system such as those disclosed herein. The software may, for example, include instructions for performing one or more of the methods disclosed herein.

At least some aspects of the present disclosure may be implemented via an apparatus or apparatuses. For example, one or more devices may be configured for performing, at least in part, the methods disclosed herein. In some implementations, an apparatus may include an interface system and a control system. The interface system may include one or more network interfaces, one or more interfaces between the control system and memory system, one or more interfaces between the control system and another device and/or one or more external device interfaces.

Like reference numbers and designations in the various drawings generally indicate like elements, but different reference numbers do not necessarily designate different elements between different drawings.

A solution to the problem of providing intelligible speech in media playback (audio or audio/visual) in a noisy environment (ambient noise) is described herein by creating and using dialog boost and equalizer settings in a profile for a particular user in a particular noise level and type (environment type).

The term "mobile device" as used herein refers to a device capable of audio playback and capable to be used in multiple locations by being carried on the person of the user. Examples include cell phones, laptop computers, tablet computers, mobile game systems, wearable devices, small media players, and the like.

The term "ambient condition" or "case" or "case identification" as used herein refers to a category of noisy location/environment that may or may not interfere with the enjoyment of listening to audio media on a mobile device. Examples include the home (e.g. "default"), outside in a populated area (e.g. walking), on public transportation, in a noisy indoor environment (e.g. airport), and others.

The term "dialog boost" refers to the application of a general sound amplification of speech components of audio with negligible amplification of the non-speech components. For example, dialog boost can be performed as an algorithm that continuously monitors the audio being played, detects the presence of dialog, and dynamically applies processing to improve the intelligibility of the spoken portion of audio content. In some embodiments, dialog boost analyzes features from the audio signal and applies a pattern-recognition system to detect the presence of dialogue from moment to moment. When speech is detected, the speech spectrum is altered where necessary to accentuate the speech content in a way that allows the listener to hear it more succinctly.

The term "equalization" or "graphic equalizer" or "GED" refers to frequency-based amplitude adjustment of audio. In a true GED, the amplitude settings would be set by sliders whose positions correspond to the frequency ranges they control, but herein GED also refers to a particular setting a graphic equalizer might have, giving a specific frequency response curve.

The term "media" or "content" as used herein refers to anything with audio content. This can be music, movies, videos, video games, telephone conversations, alerts, etc. In particular, the systems and methods herein are most useful for media that has a combination of speech and non-speech components, but the systems and methods can be applied to any media.

<FIG> shows an example flowchart for creating profiles for different ambient conditions (cases). The user selects to start the setup <NUM> from the device user interface (UI) and a sample playback sample is played <NUM> for the user. This sample can be selected by system, or by the user. The user selects what volume <NUM> is played at. The user can then be taken through the different ambient noise cases <NUM> (default, walking, public transport, airport, etc.) either automatically or manually (user selected). The system can either go through all of the cases or just a select subset, including just one selected case. If the selected case is not a default case, then the user inputs an estimated ambient noise level <NUM> for their current situation, and a dialog boost level <NUM> and a graphic equalizer (GEQ) setting <NUM> that, in combination, gives the user an optimal listening experience in their subjective opinion. These settings <NUM>,<NUM>, which can be done multiple times in any order (not necessarily in the order shown in the Figures), are set based on the sample playback <NUM> of audio with speech components. Once the dialog boost and GEQ settings <NUM>,<NUM> are set according to the user's preference, they are stored in a database/memory <NUM> of profiles for future use. The system can then determine if all applicable cases have been setup <NUM>. If it is, the setup ends <NUM>. If not, then the system can go to the next case <NUM> and repeat the settings process for that case. In some embodiments the settings profiles saved are also indexed based on the injected noise level <NUM>.

In some embodiments, the dialog boost level and/or the GEQ setting are each one value from a short list of possible settings. For example, "<NUM>" from a range of <NUM>-<NUM>. In some embodiments, the settings are a real value related to the setting, such as +<NUM> dB (e.g. at a specific frequency range).

<FIG> shows an example flowchart for using created profiles according to methods described herein. The user starts their media <NUM> and selects the case profile <NUM> that best describes their current situation. If the profile is indexed based on ambient noise level, that can be selected as well. Then the system retrieves 210the profile from a database/memory <NUM> matching the selected case (and, if applicable, noise level). The system then determines if the playback is in a mobility situation <NUM> (that is, a situation requiring dialog boost and GEQ adjustment). This determination can be from user input, device identification, location data, or other means. If the system determines that this is not a mobility situation, then normal playback/mixing <NUM> occurs from the media being played <NUM> in the presence of whatever the ambient noise <NUM> is for the user. This continues until the case profile is changed <NUM> at which point the new profile is retrieved <NUM> and the process begins again. In some embodiments, a new mobility status check is performed with or before the case switching <NUM> and the process only repeats if there is a mobility situation. If there is a mobility situation found <NUM>, then dialog boost <NUM> and GEQ adjustments <NUM> are applied to the mixing <NUM> to adjust the media playback <NUM> to provide intelligible dialog in spite of the ambient noise <NUM>.

<FIG> shows an example flowchart for creating a profile, including the use of synthesized ambient noise for virtual mixing of ambient noise with the media (as opposed to real mixing of actual ambient noise with the media, such as provided in <FIG> and <FIG>). The system is similar to that from <FIG>, except that a check is done <NUM> to see if the user is creating the profile at the location of the noise or is pre-setting the case from a relatively noise-less environment (e.g. home). This check can be determined by querying the user or by location services determining the mobile device is at "home". In some embodiments, the system always assumes the user is in a relatively noise-less environment. If the user is not at the location, then a case (ambient noise condition) is selected <NUM>, either by the user or the system. Ambient noise for that case is synthesized <NUM>. In some embodiments, this can be or be based on a pre-recorded noise saved in a database/memory <NUM>. This noise is added to the play sample <NUM>, and a noise level <NUM> can be set by the user for the level they expect to experience, thereby adjusting the simulated noise <NUM>. Dialog boost level <NUM> and GEQ level <NUM> can be set in the same manner as the at-location settings are performed. The settings are then saved to a database/memory <NUM> for future use. In some embodiments, the recorded ambient noise is taken from an ambisonic source and rendered into a binaural format.

<FIG> shows an example of the perceptual differences the system can make and how comparisons made against a reference condition can be used to assess performance. As it can be seen, dialog boost is preferred over a reference condition of no boost and increases the intelligibility of the dialog, making the adjustments beneficial to the user for media where understanding the dialog is important. For example, <FIG> shows that a dialog enhancement (DE) of level "<NUM>" <NUM> shows a high level of user preference <NUM> and subjective intelligibility <NUM>, and so might be a preferred setting for most users.

<FIG> and <FIG> and <FIG> and <FIG> show example graphs for dialog boost. <FIG> and <FIG> show graphs of different dialog boost settings for the speech component of the media. As shown, the different settings show distinct curves. In contrast, <FIG> and <FIG> show graphs for the same different dialog boost settings but for the non-speech component of the media, where there is negligible difference between the curves for the different settings (i.e. the dialog boost does not boost non-speech components). <FIG> and <FIG> represent dialog boost levels with smaller intervals between levels than <FIG> and <FIG>. Different curves can be used depending on how noisy the environment is: the noisier the ambient noise is, the more aggressive the curve can be in terms of having more dialog boost on overall playback content. <FIG> shows response curves where the dialog boost levels show, for the speech components of the audio, a strong boost in low frequencies <NUM> compared to higher frequencies <NUM>. In contrast, <FIG> shows a strong boost in higher frequencies <NUM> compared to lower frequencies <NUM>. In both cases, <FIG> and <FIG> show that the non-speech components have negligible boost across all frequencies.

<FIG> shows an example UI (specifically in this case a graphical user interface, GUI) for setting the profiles. On a mobile device <NUM> the input for the setting can be presented in a simplified form for ease of use. A noise level control <NUM> can be presented as a finite number (e.g. <NUM>-<NUM>) of noise levels, for example from <NUM> as no noise with increasing values as increasing noise levels in even increments (either in actual dB or perceptual steps). The dialog boost setting <NUM> can be presented as a graphical slider from no boost to maximum boost. Likewise, the GEQ setting <NUM> can be simplified into a single range of one value (here shown as a slider) to select preset GEQ settings (e.g. corresponding to "bright", "flat", "deep", etc. tones). The cases <NUM> can be shown as icons (with or without text). For example, "default" can be shown as a house, "walking" can be shown with a person, "public transportation" can be shown as a train or bus, and "indoor venue" can be shown with an airplane (to indicate an airport). Other cases and icons can be used, such that the icon provides the user with a quick reference to the case it represents.

<FIG> shows an example mobile device architecture for implementing the features and processes described herein, according to an embodiment. Architecture <NUM> can be implemented in any electronic device, including but not limited to: a desktop computer, consumer audio/visual (AV) equipment, radio broadcast equipment, mobile devices (e.g., smartphone, tablet computer, laptop computer, wearable device). In the example embodiment shown, architecture <NUM> is for a smart phone and includes processor(s) <NUM>, peripherals interface <NUM>, audio subsystem <NUM>, loudspeakers <NUM>, microphone <NUM>, sensors <NUM> (e.g., accelerometers, gyros, barometer, magnetometer, camera), location processor <NUM> (e.g., GNSS receiver), wireless communications subsystems <NUM> (e.g., Wi-Fi, Bluetooth, cellular) and I/O subsystem(s) <NUM>, which includes touch controller <NUM> and other input controllers <NUM>, touch surface <NUM> and other input/control devices <NUM>. Memory interface <NUM> is coupled to processors <NUM>, peripherals interface <NUM> and memory <NUM> (e.g., flash, RAM, ROM). Memory <NUM> stores computer program instructions and data, including but not limited to: operating system instructions <NUM>, communication instructions <NUM>, GUI instructions <NUM>, sensor processing instructions <NUM>, phone instructions <NUM>, electronic messaging instructions <NUM>, web browsing instructions <NUM>, audio processing instructions <NUM>, GNSS/navigation instructions <NUM> and applications/data <NUM>. Audio processing instructions <NUM> include instructions for performing the audio processing described herein. Other architectures with more or fewer components can also be used to implement the disclosed embodiments.

The system can be provided as a service driven from a remote server, as a standalone program on the device, integrated into a media player application, or included as part of the operating system as part of its sound settings.

Claim 1:
A method of configuring a mobile device for use with ambient noise for a user of the mobile device, the method comprising:
receiving, from the user, a case identification of the ambient noise;
receiving, from the user, a noise level for the ambient noise;
receiving, from the user, a dialog boost level for the ambient noise at the noise level;
receiving, from the user, a graphic equalizer setting for the ambient noise at the noise level;
playing, from the mobile device, a sample audio for the user while the user sets the dialog boost level and the graphic equalizer setting;
and
storing on the mobile device the dialog boost level and graphic equalizer setting for the case identification at the noise level in a profile, wherein the device is configured to play audio media using the dialog boost level and graphic equalizer setting when the profile is selected by the user,
the method further comprising:
simulating the ambient noise at the noise level; and
mixing the simulated ambient noise with the sample audio prior to the playing the sample audio.