VIRTUAL AUDITORY DISPLAY FILTERS AND ASSOCIATED SYSTEMS, METHODS, AND NON-TRANSITORY COMPUTER-READABLE MEDIA

An example method includes receiving audio signals associated with virtual auditory space locations. Digital filters are selected based on the virtual auditory space locations. The digital filters include one or more notch filters including one or more center frequencies that are based on generally sigmoidal distributions of center frequencies as a function of virtual auditory space location. The notch filters are configured to produce one or more notches in frequency spectrums of audio signals when applied to the audio signals. The digital filters are applied to the audio signals to obtain processed audio signals. Output audio signals are generated based on the processed audio signals. The output audio signals are provided to devices for producing virtual auditory display sound.

TECHNICAL FIELD

The present technology generally relates to virtual auditory display filters, and more particularly to generating virtual auditory display filters, applying virtual auditory display filters to audio signals to generate virtual auditory display sound in virtual auditory space, and applications related to virtual auditory display filters.

BACKGROUND

Three-dimensional (3D) sound systems may be implemented by arranging multiple speakers in a space, thereby allowing sound to arrive from different directions. Headphones, headsets, and earbuds (collectively headphones) are often used to listen to music or other audio. A headphone may simulate 3D sound using a head-related transfer function (HRTF). An HRTF may be a compressed representation of how sound waves interact with the human head and ears. More generally, an HRTF may be used to simulate the effect of sound waves traveling through a 3D space.

SUMMARY

In some aspects, the techniques described herein relate to one or more non-transitory computer-readable media including executable instructions that when executed by one or more processors of a system cause the system to perform a method, the method including: generating, for each of multiple virtual auditory space locations, one or more first digital filters, the one or more first digital filters including one or more first notch filters, the one or more first notch filters including one or more first center frequencies, the one or more first center frequencies based on a first generally sigmoidal distribution of center frequencies as a function of virtual auditory space location, the one or more first notch filters configured to produce one or more first notches in a first frequency spectrum of a first audio signal based on the one or more first center frequencies when applied to the first audio signal; generating, for each of the multiple virtual auditory space locations, one or more second digital filters, the one or more second digital filters including one or more second notch filters, the one or more second notch filters including one or more second center frequencies, the one or more second center frequencies based on a second generally sigmoidal distribution of center frequencies as a function of virtual auditory space location, the one or more second notch filters configured to produce one or more second notches in a second frequency spectrum of a second audio signal based on the one or more second center frequencies when applied to the second audio signal; receiving an audio signal, the audio signal having one or more audio sub-signals, an audio sub-signal associated with a virtual auditory space location; for each audio sub-signal of the one or more audio sub-signals: selecting, based on the virtual auditory space location associated with the audio sub-signal, particular one or more first digital filters and particular one or more second digital filters; applying the particular one or more first digital filters to the audio sub-signal to obtain a first processed audio sub-signal; and applying the particular one or more second digital filters to the audio sub-signal to obtain a second processed audio sub-signal; generating, based on multiple first processed audio sub-signals, a first output audio signal for a first device; generating, based on multiple second processed audio sub-signals, a second output audio signal for a second device; and providing the first output audio signal to the first device and the second output audio signal to the second device.

In some aspects, the techniques described herein relate to one or more non-transitory computer-readable media wherein the virtual auditory space location is a first virtual auditory space location, and the method further includes: receiving a head orientation of a user; and for each audio sub-signal of the one or more audio sub-signals, determining, based on the first virtual auditory space location associated with the audio sub-signal and the head orientation, a second virtual auditory space location, wherein selecting, based on the virtual auditory space location associated with the audio sub-signal, the particular one or more first digital filters and the particular one or more second digital filters includes selecting, based on the second virtual auditory space location, the particular one or more first digital filters and the particular one or more second digital filters.

In some aspects, the techniques described herein relate to one or more non-transitory computer-readable media wherein the particular one or more first digital filters are first particular one or more first digital filters, the particular one or more second digital filters are first particular one or more second digital filters, the head orientation of the user is a first head orientation of the user, and the method further includes: receiving a personalization audio signal associated with a third virtual auditory space location; selecting, based on the third virtual auditory space location, second particular one or more first digital filters and second particular one or more second digital filters; applying the second particular one or more first digital filters to the personalization audio signal to obtain a first processed personalization audio signal; applying the second particular one or more second digital filters to the personalization audio signal to obtain a second processed personalization audio signal; generating, based on the first processed personalization audio signal, a third output audio signal for the first device; generating, based on the second processed personalization audio signal, a fourth output audio signal for the second device; providing the third output audio signal to the first device and the fourth output audio signal to the second device; receiving a second head orientation of the user; determining, based on the second head orientation, a fourth virtual auditory space location; determining a delta between the third virtual auditory space location and the fourth virtual auditory space location; and modifying, based on the delta, the one or more first digital filters and the one or more second digital filters.

In some aspects, the techniques described herein relate to one or more non-transitory computer-readable media wherein modifying, based on the delta, the one or more first digital filters and the one or more second digital filters includes modifying the one or more first center frequencies on which the one or more first notch filters are based and the one or more second center frequencies on which the one or more second notch filters are based.

In some aspects, the techniques described herein relate to one or more non-transitory computer-readable media, the method further including generating, using one or more image processing algorithms, a first notch mask and a second notch mask, the first notch mask specifying a first gain modifier as a function of virtual auditory space location, the second notch mask specifying a second gain modifier as a function of virtual auditory space location, wherein: the one or more first notch filters include the one or more first center frequencies and a first gain as modified by the first gain modifier, and the one or more first notch filters are configured to produce one or more first notches in the first frequency spectrum of the first audio signal based on the one or more first center frequencies and the first gain when applied to the first audio signal, and the one or more second notch filters include the one or more second center frequencies and a second gain as modified by the second gain modifier, and the one or more second notch filters are configured to produce one or more second notches in the second frequency spectrum of the second audio signal based on the one or more second center frequencies and the second gain when applied to the second audio signal.

In some aspects, the techniques described herein relate to one or more non-transitory computer-readable media wherein the one or more image processing algorithms include one or more of a gaussian function, a sharpening function, a contrast adjustment function, a color correction function, a thresholding function, an edge detection function, and a segmentation function.

In some aspects, the techniques described herein relate to one or more non-transitory computer-readable media, the method further including: receiving a selection of an acoustic environment; and determining, based on the acoustic environment, a first acoustic environment digital filter and a second acoustic environment digital filter, wherein for each audio sub-signal of the one or more audio sub-signals, applying the particular one or more first digital filters to the audio sub-signal to obtain the first processed audio sub-signal includes applying the particular one or more first digital filters and the first acoustic environment digital filter to the audio sub-signal to obtain the first processed audio sub-signal, and applying the particular one or more second digital filters to the audio sub-signal to obtain the second processed audio sub-signal includes applying the particular one or more second digital filters and the second acoustic environment digital filter to the audio sub-signal to obtain the second processed audio sub-signal.

In some aspects, the techniques described herein relate to one or more non-transitory computer-readable media wherein the acoustic environment is represented by one or more ambisonic arrays and determining, based on the acoustic environment, the first acoustic environment digital filter and the second acoustic environment digital filter includes, determining, based on the one or more ambisonic arrays, the first acoustic environment digital filter and the second acoustic environment digital filter.

In some aspects, the techniques described herein relate to one or more non-transitory computer-readable media wherein the one or more first digital filters and the one or more second digital filters are infinite impulse response filters.

In some aspects, the techniques described herein relate to one or more non-transitory computer-readable media wherein the first device includes a first ear-worn device, and the second device includes a second ear-worn device.

In some aspects, the techniques described herein relate to a system including at least one processor and at least one memory including executable instructions that when executed by the at least one processor cause the system to: generate, for each of multiple virtual auditory space locations, one or more first digital filters, the one or more first digital filters including one or more first notch filters, the one or more first notch filters including one or more first center frequencies, the one or more first center frequencies based on a first generally sigmoidal distribution of center frequencies as a function of virtual auditory space location, the one or more first notch filters configured to produce one or more first notches in a first frequency spectrum of a first audio signal based on the one or more first center frequencies when applied to the first audio signal; generate, for each of the multiple virtual auditory space locations, one or more second digital filters, the one or more second digital filters including one or more second notch filters, the one or more second notch filters including one or more second center frequencies, the one or more second center frequencies based on a second generally sigmoidal distribution of center frequencies as a function of virtual auditory space location, the one or more second notch filters configured to produce one or more second notches in a second frequency spectrum of a second audio signal based on the one or more second center frequencies when applied to the second audio signal; receive an audio signal, the audio signal having one or more audio sub-signals, an audio sub-signal associated with a virtual auditory space location; for each audio sub-signal of the one or more audio sub-signals: select, based on the virtual auditory space location associated with the audio sub-signal, particular one or more first digital filters and particular one or more second digital filters; apply the particular one or more first digital filters to the audio sub-signal to obtain a first processed audio sub-signal; and apply the particular one or more second digital filters to the audio sub-signal to obtain a second processed audio sub-signal; generate, based on multiple first processed audio sub-signals, a first output audio signal for a first device; generate, based on multiple second processed audio sub-signals, a second output audio signal for a second device; and provide the first output audio signal to the first device and the second output audio signal to the second device.

In some aspects, the techniques described herein relate to a system wherein the virtual auditory space location is a first virtual auditory space location, and the executable instructions that when executed by the at least one processor further cause the system to: receive a head orientation of a user; and for each audio sub-signal of the one or more audio sub-signals, determine, based on the first virtual auditory space location associated with the audio sub-signal and the head orientation, a second virtual auditory space location, wherein to select, based on the virtual auditory space location associated with the audio sub-signal, the particular one or more first digital filters includes to select, based on the second virtual auditory space location, the particular one or more first digital filters, and to select, based on the virtual auditory space location associated with the audio sub-signal, the particular one or more second digital filters includes to select, based on the second virtual auditory space location, the particular one or more second digital filters.

In some aspects, the techniques described herein relate to a system wherein the one or more first digital filters are first one or more first digital filters, the one or more second digital filters are first one or more second digital filters, the particular one or more first digital filters are first particular one or more first digital filters, the particular one or more second digital filters are first particular one or more second digital filters, the head orientation is a first head orientation, the audio signal having one or more audio sub-signals is a first audio signal having first one or more audio sub-signals, and the executable instructions that when executed by the at least one processor further cause the system to: receive a personalization audio signal that has a third virtual auditory space location; select, based on the third virtual auditory space location, second particular one or more first digital filters and second particular one or more second digital filters; apply the second particular one or more first digital filters to the personalization audio signal to obtain a first processed personalization audio signal; apply the second particular one or more second digital filters to the personalization audio signal to obtain a second processed personalization audio signal; generate, based on the first processed personalization audio signal, a third output audio signal for the first device; generate, based on the second processed personalization audio signal, a fourth output audio signal for the second device; provide the third output audio signal to the first device and the fourth output audio signal to the second device; receive a second head orientation of the user; determine, based on the second head orientation, a fourth virtual auditory space location; determine a delta between the third virtual auditory space location and the fourth virtual auditory space location; and selecting, based on the delta, second one or more first digital filters and second one or more second digital filters, the second one or more first digital filters and the second one or more second digital filters for use while receiving a second input audio signal having second one or more audio sub-signals.

In some aspects, the techniques described herein relate to a system wherein the executable instructions that when executed by the at least one processor further cause the system to generate, using one or more image processing algorithms, a first notch mask and a second notch mask, the first notch mask specifying a first gain modifier based on the virtual auditory space location, the second notch mask specifying a second gain modifier based on the virtual auditory space location, wherein: the one or more first notch filters are generated using the one or more first center frequencies based on the first generally sigmoidal distribution of center frequencies as a function of virtual auditory space location and a first gain as modified by the first gain modifier and the one or more first notch filters are configured to produce one or more first notches in the first frequency spectrum of the first audio signal based on the one or more first center frequencies and the first gain when applied to the first audio signal, and the one or more second notch filters are generated using the one or more second center frequencies based on the second generally sigmoidal distribution of center frequencies as a function of virtual auditory space location and a second gain as modified by the second gain modifier and the one or more second notch filters are configured to produce one or more second notches in the second frequency spectrum of the second audio signal based on the one or more second center frequencies and the second gain when applied to the second audio signal.

In some aspects, the techniques described herein relate to a system wherein the executable instructions that when executed by the at least one processor further cause the system to: receive a selection of an acoustic environment; and determine based on the acoustic environment, a first acoustic environment digital filter and a second acoustic environment digital filter, wherein for each audio sub-signal of the one or more audio sub-signals, to apply the particular one or more first digital filters to the audio sub-signal to obtain the first processed audio sub-signal includes to apply the particular one or more first digital filters and the first acoustic environment digital filter to the audio sub-signal to obtain the first processed audio sub-signal, and to apply the particular one or more second digital filters to the audio sub-signal to obtain the second processed audio sub-signal includes to apply the particular one or more second digital filters and the second acoustic environment digital filter to the audio sub-signal to obtain the second processed audio sub-signal.

In some aspects, the techniques described herein relate to a system wherein the one or more first digital filters and the one or more second digital filters are infinite impulse response filters.

In some aspects, the techniques described herein relate to a system wherein the first device includes a first ear-worn device, and the second device includes a second ear-worn device.

In some aspects, the techniques described herein relate to a method including: generating a first virtual auditory display filter, the first virtual auditory display filter including a first set of first functions, one or more first functions, when applied to a first audio signal having a first location in virtual auditory space, generating a first processed audio signal having a first frequency response with one or more first notches at one or more first center frequencies that are based on the first location, the one or more first notches having one or more first peak-to-trough depths of at most −10 dB; generating a second virtual auditory display filter, the second virtual auditory display filter including a second set of second functions, one or more second functions, when applied to the first audio signal, generating a second processed audio signal having a second frequency response with one or more second notches at one or more second center frequencies that are based on the first location, the one or more second notches having one or more second peak-to-trough depths of at most −10 dB; receiving a second audio signal having a second location in the virtual auditory space; applying the first virtual auditory display filter, including a first subset of first functions selected based on the second location, to the second audio signal to generate a third processed audio signal having a third frequency response; applying the second virtual auditory display filter, including a second subset of second functions selected based on the second location, to the second audio signal to generate a fourth processed audio signal having a fourth frequency response; providing the third processed audio signal to a first sound output device; and providing the fourth processed audio signal to a second sound output device.

In some aspects, the techniques described herein relate to a method where the one or more first center frequencies are based on a first generally sigmoidal distribution of center frequencies as a function of location in the virtual auditory space and the one or more second center frequencies are based on a second generally sigmoidal distribution of center frequencies as a function of location in the virtual auditory space.

In some aspects, the techniques described herein relate to a method, further including receiving a head orientation of a user, wherein: applying the first virtual auditory display filter, including the first subset of first functions selected based on the second location, to the second audio signal to generate a third processed audio signal having a third frequency response includes applying the first virtual auditory display filter, including a third subset of first functions selected based on the second location and the head orientation, to the second audio signal to generate the third processed audio signal having the third frequency response, and applying the second virtual auditory display filter, including the second subset of second functions selected based on the second location, to the second audio signal to generate a fourth processed audio signal having a fourth frequency response includes applying the second virtual auditory display filter, including a fourth subset of second functions selected based on the second location and the head orientation, to the second audio signal to generate the fourth processed audio signal having the fourth frequency response.

In some aspects, the techniques described herein relate to a method, further including: generating, using one or more image processing algorithms, a first notch mask and a second notch mask, the first notch mask specifying a first depth modifier as a function of a location in the virtual auditory space, the second notch mask specifying a second depth modifier as a function of the location in the virtual auditory space; modifying the one or more first peak-to-trough depths based on the first depth modifier; and modifying the one or more second peak-to-trough depths based on the second depth modifier.

In some aspects, the techniques described herein relate to a method wherein the one or more image processing algorithms include one or more of a gaussian function, a sharpening function, a contrast adjustment function, a color correction function, a thresholding function, an edge detection function, and a segmentation function.

In some aspects, the techniques described herein relate to a method wherein the first set of first functions include first infinite impulse response digital filters and the second set of second functions include second infinite impulse response digital filters.

In some aspects, the techniques described herein relate to a method including: receiving a set of multiple first digital filters, one or more first digital filters generated for each of multiple virtual auditory space locations, the one or more first digital filters including one or more first notch filters, the one or more first notch filters including one or more first center frequencies, the one or more first center frequencies based on a first generally sigmoidal distribution of center frequencies as a function of virtual auditory space location, the one or more first notch filters configured to produce one or more first notches in a first frequency spectrum of a first audio signal based on the one or more first center frequencies when applied to the first audio signal; receiving a set of multiple second digital filters, one or more second digital filters generated for each of multiple virtual auditory space locations, the one or more second digital filters including one or more second notch filters, the one or more second notch filters including one or more second center frequencies, the one or more second center frequencies based on a second generally sigmoidal distribution of center frequencies as a function of virtual auditory space location, the one or more second notch filters configured to produce one or more second notches in a second frequency spectrum of a second audio signal based on the one or more second center frequencies when applied to the second audio signal; receiving a personalization audio signal that has a virtual auditory space location; selecting, based on the virtual auditory space location, particular one or more first digital filters and particular one or more second digital filters; applying the particular one or more first digital filters to the personalization audio signal to obtain a first processed personalization audio signal; applying the particular one or more second digital filters to the personalization audio signal to obtain a second processed personalization audio signal; providing a first output audio signal based on the first processed personalization audio signal to a first device and a second output audio signal based on the second processed personalization audio signal to a second device; receiving a user perception of first sound output by the first device and second sound output by the second device; and modifying, based on the user perception, the set of multiple first digital filters and the set of multiple second digital filters.

In some aspects, the techniques described herein relate to a method wherein the virtual auditory space location is a first virtual auditory space location and wherein modifying, based on the user perception, the set of multiple first digital filters and the set of multiple second digital filters includes: determining, based on the user perception, a second virtual auditory space location; determining a delta between the first virtual auditory space location and the second virtual auditory space location; and modifying, based on the delta, the set of multiple first digital filters and the set of multiple second digital filters.

In some aspects, the techniques described herein relate to a method wherein receiving the user perception includes receiving a head orientation of a user and wherein determining, based on the user perception, the second virtual auditory space location, includes determining, based on the head orientation of the user, the second virtual auditory space location.

In some aspects, the techniques described herein relate to a method wherein receiving the user perception includes receiving one or more gestures of a user and wherein determining, based on the user perception, the second virtual auditory space location, includes determining, based on the one or more gestures of the user, the second virtual auditory space location.

In some aspects, the techniques described herein relate to a method wherein the set of multiple first digital filters is a first set of multiple first digital filters, the set of multiple second digital filters is a first set of multiple second digital filters, wherein modifying, based on the user perception, the set of multiple first digital filters includes selecting, based on the user perception, a second set of multiple first digital filters, and wherein modifying, based on the user perception, the set of multiple second digital filters includes selecting, based on the user perception, a second set of multiple second digital filters.

In some aspects, the techniques described herein relate to a method wherein modifying, based on the user perception, the set of multiple first digital filters includes modifying the one or more first center frequencies and wherein modifying, based on the user perception, the set of multiple second digital filters includes modifying the one or more second center frequencies.

In some aspects, the techniques described herein relate to a method, further including: determining, based on the user perception, a spatialization precision estimate; and providing the spatialization precision estimate.

DETAILED DESCRIPTION

An HRTF may be for one person. Generating an individual HRTF typically requires a highly specialized environment and acoustic testing equipment. A person must remain still for approximately 30 minutes in an anechoic chamber while audio signals are emitted from different known locations. A microphone is placed in each ear of the person to capture the audio signals. However, this method presents challenges as there may be spurious responses due to factors such as the chamber, the audio signal source(s) and the microphone, that need to be eliminated in order to obtain an accurate Head Related Impulse Response (HRIR) which can then be converted to an HRTF. Furthermore, any movement by the person may affect the measurements, which may result in an inaccurate HRTF for the person. Another practical limitation of measuring an HRIR is that the time to collect directly scales with the number of discrete coordinates and practically limits the resolution of the resulting HRTF.

So-called universal HRTFs have been utilized to overcome disadvantages of individual HRTFs. Such universal HRTFs may be produced by averaging or otherwise combining measurements from multiple persons. However, such combining typically results in losing the individual characteristics of each person that are necessary to produce accurate virtual 3D sound for the person. As a result, such universal HRTFs may not accurately locate sound in virtual 3D space for all users, especially sound that is located directly in front of a user at approximately zero degrees azimuth and zero degrees elevation.FIG.8Edepicts an example HRTF810.

Another prior approach has attempted to simulate a personalized HRTF using photogrammetry of the head, torso, and pinna, or using other methods with highly precise head, torso, and pinna scanning via time of flight or structured light. A physical acoustics model is then generated based on the resulting scanned form. However, this approach may not yield convincing rendering of virtual 3D space, because after the physical scan is measured, the physics-based simulation of sound interacting with the modeled surface may introduce complexity and inaccuracy in the resulting psychoacoustic cues.

The technology described herein provides technical solutions to the technical problems of the prior approaches described above. The technology may utilize virtual auditory display filters that may result in accurately rendered sounds in their locations in virtual auditory space. The virtual auditory display filters may utilize spectral shaping techniques, using equalizers, filters, and/or dynamic range compression, to manipulate the frequency spectrum of audio signals. Virtual auditory display filters may be generated without resort to direct physical measurements (for example, measurements in an anechoic chamber, photogrammetry, etc.).

Virtual auditory display filters may be or include functions that manipulate a frequency spectrum of an audio signal. Virtual auditory display filters may be or include digital filters, such as parametric equalization (EQ) filters that allow for adjustment of parameters such as the center frequency, gain, quality (Q or q), cutoff frequency, slope, bandwidth and/or filter type. The parameters may be set as a function of a location of a sound in virtual auditory space. The functions or the digital filters may affect the frequency spectrum of an audio signal by creating notches and peaks in the audio signal. The notches, peaks, and other spectral shaping of the audio signal accurately places the resulting sound in virtual auditory space. Furthermore, the notches, peaks, and other spectral shaping of the audio signal produces a processed audio signal that may be used to output high-quality clear sound that, in the example of music recordings, may accurately represent the original recorded performance and allow listeners to hear subtleties and nuances of the original recorded performance. As described herein, a digital filter may refer to a digital filter, a function, and/or some combination of one or more functions or one or more digital filters.

Virtual auditory space may be described as a virtual 3D sound environment of a person in which the person may perceive a sound as emanating from any location in the virtual 3D sound environment. In the described technology, each location in virtual auditory space may have an associated function or digital filter that is applied to audio signals that have that location. The application of the function or digital filter to an audio signal with a location results in sound, which may be referred to as virtual auditory display sound, that is perceived by the person as coming from that location.

Accordingly, the person, who may be wearing headphones, earbuds, or other ear-worn devices, may experience virtual auditory display sound. Other advantages of the described technology will be apparent.

FIG.1is a diagram of an environment150in which a virtual auditory display system and virtual auditory display devices that interface with the virtual auditory display system may operate in some embodiments. As depicted, the environment150includes a virtual auditory display system102and a virtual auditory display device100. The virtual auditory display system102and the virtual auditory display device100may together comprise a system. The virtual auditory display system102and the virtual auditory display device100may together render sounds in virtual auditory space for a wearer of the virtual auditory display device100.

The virtual auditory display system102may include a binauralizer138. The binauralizer may include a system memory118, which may include a left ear digital filter map120aand a right ear digital filter map120b. The binauralizer138may also include a left ear convolution engine116a, a right ear convolution engine116b, and a spatialization engine114. The virtual auditory display system102may also include other components, modules and/or engines, such as those described with reference to, for example,FIG.2A.

In some embodiments, the virtual auditory display system102may be or include a software application that may execute on a digital device. A digital device is any device with at least one processor and memory. Digital devices are discussed further herein, for example, with reference toFIG.16. For example, the virtual auditory display system102may be a software application that executes on a general-purpose computing device, such as a laptop or desktop computer. As another example, the virtual auditory display system102may be a software application that executes on a mobile device such as a phone or a tablet. In other embodiments, the virtual auditory display system102be or include a software application or a firmware application that executes on a special-purpose computing device, such as on the virtual auditory display device100.

The virtual auditory display device100may include a first ear-worn device102aand a second ear-worn device102b. The first ear-worn device102aand the second ear-worn device102bmay each be any ear-worn, ear-mounted or ear-proximate device such as an earphone of a pair of earphones, an earbud of a pair of earbuds, a headphone of a headset, a speaker of a virtual reality headset, and the like. In some embodiments, the virtual auditory display device100may be an embodiment of the virtual auditory display devices as described in the aforementioned co-pending U.S. Patent Application No. ______, filed on the same day herewith, and entitled “VIRTUAL AUDITORY DISPLAY DEVICES AND ASSOCIATED SYSTEMS, METHODS, AND DEVICES.” The first ear-worn device102aand/or the second ear-worn device102bmay include components, such as an inertial measurement unit (IMU), an accelerometer, a gyroscope, and/or a magnetometer, that detect a head orientation of a wearer wearing the first ear-worn device102aand the second ear-worn device102b.

In some embodiments, a digital device (for example, a laptop or desktop computer) may receive an encoded audio file106that has one or more channels of audio. Examples of an encoded audio file106include 2.0 (two channels of audio), 2.1 (three channels of audio), 5.1 (six channels of audio), 7.1.4 (12 channels of audio), and 9.1.6 (16 channels of audio). The digital device may decode the encoded audio file106to obtain decoded audio objects108and an input audio signal112that includes one or more audio sub-signals (alternately, audio channels). Each of the decoded audio objects108and/or the audio sub-signals may have associated coordinates which identify the location of the audio object in virtual auditory space. The coordinates may be cartesian coordinates, spherical coordinates, and/or polar coordinates. Although specific examples of encoded audio files are described herein, the technology is not limited to such examples, and may be used with audio files that have any number of channels.

The digital device may send the coordinates110to the spatialization engine114and the input audio signal112to the left ear convolution engine116aand the right ear convolution engine116b. In some embodiments, the virtual auditory display system102receives the encoded audio file106and decodes the encoded audio file106to obtain the decoded audio objects108and the input audio signal112.

As described with reference to, for example,FIGS.11A and11B, a user interface component of the virtual auditory display system102may provide a user interface that allows the user to select an acoustic environment. The spatialization engine114may receive a selection134of the acoustic environment132via the user interface component from the wearer and utilize the selection134to process audio signals that are sent to the first ear-worn device102aand the second ear-worn device102bof the virtual auditory display device100.

As described with reference to, for example,FIGS.15A through15J, the user interface component of the virtual auditory display system102may provide a user interface128that allows the user to perform a calibration and/or personalization procedure136to calibrate and/or personalize the virtual auditory display system102. The wearer may use the user interface128to personalize the virtual auditory display system102so that the user's perception of the location of a sound matches the location of the sound in virtual auditory space. The user-perceived location of the sound may be sent in a signal130to the spatialization engine114.

The spatialization engine114may determine, based on the acoustic environment132, a first acoustic environment digital filter and a second acoustic environment digital filter. An acoustic environment digital filter may be or include a digital filter that is applied to an audio signal to manipulate the audio signal so as to produce the effect of the audio being played, generated or produced in a particular acoustic environment. The spatialization engine114may provide the first acoustic environment digital filter to the left ear convolution engine116aand the second acoustic environment digital filter to the right ear convolution engine116b.

While the virtual auditory display system102is receiving the input audio signal112, one or both of the first ear-worn device102aand the second ear-worn device102bmay detect a head orientation of a wearer of the virtual auditory display device100and provide the head orientation and an audio source distance126(which may be specified by the wearer) to the virtual auditory display system102.

The binauralizer138may, for each audio sub-signal of the one or more audio sub-signals, obtain multiple first processed audio sub-signals and multiple second processed audio sub-signals. The binauralizer138may do so by determining, based on the virtual auditory space location associated with the audio sub-signal and the head orientation, a particular first location in the virtual auditory space for the audio sub-signal. The left ear digital filter map120amaps locations in virtual auditory space to digital filters and/or functions for the first ear-worn device102aand the right ear digital filter map120bmaps locations in virtual auditory space to digital filters and/or functions for the second ear-worn device102b.

Virtual auditory display filters may be or include functions and/or digital filters that the virtual auditory display system102applies to audio signals to create virtual auditory display sound. A generation system, discussed in more detail with reference to, for example,FIGS.3A and3B, may generate the virtual auditory display filters that the virtual auditory display system102applies to audio signals.

The binauralizer138may select a particular first digital filter and/or function from the left ear digital filter map120aand a particular second digital filter and/or function from the right ear digital filter map120bin the system memory118. The binauralizer138may provide the particular first digital filter and/or function to the left ear convolution engine116aand the particular second digital filter and/or function to the right ear convolution engine116b.

The left ear convolution engine116amay apply the particular first digital filter and/or function and the first acoustic environment digital filter to the audio sub-signal to obtain a first processed audio sub-signal. The left ear convolution engine116amay then generate, based on the multiple first processed audio sub-signals, an output audio signal122afor the first ear-worn device102a. The right ear convolution engine116bmay apply the particular second digital filter and/or function and the second acoustic environment digital filter to the audio sub-signal to obtain a second processed audio sub-signal. The right ear convolution engine116bmay then generate, based on the multiple second processed audio sub-signals, an output audio signal122bfor the second ear-worn device102b. The graph124adepicts an example impulse response for the output audio signal122aand the graph124bdepicts an example impulse response for the output audio signal122b.

FIG.2Ais a block diagram depicting components of the virtual auditory display system102in some embodiments. The virtual auditory display system102may include the binauralizer138, a communication module202, an audio input module204, an audio output module206, a calibration and personalization module208, a user interface module210, and a data storage220.

The communication module202may send requests and/or data between components of the virtual auditory display system102and any other components or devices, such as the virtual auditory display device100and a generation system380(described with reference to, for example,FIGS.3A and3B). The communication module202may also receive requests and/or data between components of the virtual auditory display system102and any other components or devices.

The audio input module204may receive the input audio signal112from, for example, the general purpose computing device on which the virtual auditory display system102executes. The audio output module206may provide the output audio signal122ato the first ear-worn device102aand the output audio signal122bto the second ear-worn device102b.

The calibration and personalization module208may calibrate IMUs and/or other sensors of the first ear-worn device102aand the second ear-worn device102b. The calibration and personalization module208may also generate personalization audio signals and receive personalization information for personalizing filters. The user interface module210may provide user interfaces that allow users to, among other things, select an acoustic environment, select an audio visualization, control audio volume, and request calibration and/or personalization procedures be performed by the virtual auditory display system102.

The data storage220may include data stored, accessed, and/or modified by any of the engines, components, modules or the like of the virtual auditory display system102. The data storage220may include any number of data storage structures such as tables, databases, lists, and/or the like. The data storage220may include data that is stored in memory (for example, random access memory (RAM)), on disk, or some combination of in-memory and on-disk.

FIG.2Bis a block diagram depicting components of the first ear-worn device102aand the second ear-worn device102bin some embodiments. The first ear-worn device102amay include a memory250, an IMU sensor system252(inertial measurement unit sensor system), a magnetometer254, a microcontroller256, a power management component258, an audio DSP260(audio digital signal processor), microphones262, and speakers264. The second ear-worn device102bmay include a memory250, an IMU sensor system252(inertial measurement unit sensor system), a magnetometer254, an audio DSP260, microphones262, and speakers264.

The memory250may store software and/or firmware. The IMU sensor system252and/or the magnetometer254may detect a head orientation of a wearer of the virtual auditory display device100and/or user interactions with the virtual auditory display device100. The microcontroller256may execute software and/or firmware stored in the memory250or in the storage of the microcontroller256.

The power management component258may provide power management. The audio DSP260may process audio signals to perform functions such as noise cancellation. The microphones262may capture audio, such as environmental audio and/or audio from a wearer of the first ear-worn device102a. The speakers264may output sound based on the output audio signal122aand the output audio signal122b.

The first ear-worn device102aand/or the second ear-worn device102bmay include components other than those depicted inFIG.2B, such as switches, interconnects, and oscillators. The first ear-worn device102amay be the primary device and the second ear-worn device102bmay be the secondary device. As such, the second ear-worn device102bmay not include a microcontroller256. In some embodiments, the second ear-worn device102bincludes a microcontroller256.

An engine, component, module, or the like of the virtual auditory display system102, the first ear-worn device102a, the second ear-worn device102b, or a generation system380(described with reference to, for exampleFIG.3B) may be hardware, software, firmware, or any combination. For example, each engine, component, module or the like may include functions performed by dedicated hardware (for example, an Application-Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), or the like), software, instructions maintained in memory, and/or any combination. Software and/or firmware may be executed by one or more processors.

Although a limited number of engines, components, and modules are depicted inFIGS.2A and2BandFIG.3B, there may be any number of engines, components, and modules or the like. Further, individual engines, components, and modules may perform any number of functions, including functions of multiple modules as described herein. Moreover, although the virtual auditory display system102, the first ear-worn device102a, the second ear-worn device102b, and the generation system380may be depicted as having a single one of several engines, components, or modules, the virtual auditory display system102, the first ear-worn device102a, the second ear-worn device102b, and the generation system380may have multiple engines, components, modules, or the like that perform a particular function. For example, the first ear-worn device102ais depicted as having a single one of the audio DSP260, but the first ear-worn device102amay include multiple of the audio DSP260.

FIG.3Ais a block diagram of a method300of personalizing, generating and applying digital filters in some embodiments. A generation system380(seeFIG.3B) may perform the generation of digital filters (step304through step310), and the virtual auditory display system102may perform the personalization of digital filters (step302) and the application of digital filters (step312through step314).

A digital filter may be or include one or more parametric equalization (EQ) filters that allow for adjustment of parameters such as the center frequency, gain, quality (Q or q), cutoff frequency, slope, bandwidth and/or filter type. The parametric EQ filters may be or include biquad filters. The biquad filters may be or include peaking, low shelf, and high shelf filters. In some embodiments, the digital filters may be or include one or more finite impulse response (FIR) filters. The FIR filters may be generated from or based on one or more infinite impulse response (IIR) filters. In some embodiments, the digital filters may be or include one or more IIR filters, or any other suitable type of digital filter.

The digital filters that the generation system380generates may be organized into multiple groups. The groups of digital filters may include a group of notch filters, a group of head shadow filters, a group of shelf filters, a group of peak filters, a group of beam filters, a group of stereo filters, a group of rear filters, a group of top filters, a group of top transition filters, a group of bottom filters, a group of bottom transition filters, and a group of broadside filters. Other groups are possible. Certain digital filters or groups of digital filters may be utilized for purposes of setting the locations of sounds in virtual auditory space (for example groups of notch filters). Certain digital filters or groups of digital filters may be utilized for purposes of ensuring that sounds meet required thresholds of tonal quality, clarity, brightness, and the like.

A digital filter may be or include an algorithm and one or more parameters for the algorithm. For example, the algorithm may be or include a high shelf, a low shelf, and a peaking algorithm. The one or more parameters may be or include a center frequency, a quality (Q or q), a gain, and a sampling frequency. For example, a notch digital filter may specify a peaking algorithm, an initial center frequency of 6600 Hz, a Q of 15, and an initial gain of −85 decibels (dB). The one or more parameters may be modified. For example, an initial center frequency may be shifted to obtain a shifted center frequency and an initial gain may be modified by a parameter modifier (see, for example, the discussion with reference toFIGS.7A through7X) and a factor that has any value, such as a value between 0 and 1, inclusive. The digital filter may be or include the one or more parameters as modified.

Digital filters may be generated and utilized based on how the digital filters represent individuals' geometries interact with sound waves. For example, a digital filter having a high shelf algorithm may produce a high shelf that may be a virtual representation of how the geometry of individual's concha bowl interacts with sound waves.

The method300may include a step302of calibration and/or personalization. The virtual auditory display system102(for example, the calibration and personalization module208) may perform calibration of the IMUs and/or other sensors of the virtual auditory display device100using various devices and/or services326, such as one or both of the first ear-worn device102aand the second ear-worn device102b, a cloud-based computing service, and/or a peripheral to a computing device, such as a camera.

The virtual auditory display system102(for example, the calibration and personalization module208) may perform personalization of the virtual auditory display system using various methodologies and/or techniques, such as: 1) a user-directed action and/or perception of acoustic cues316; 2) acoustic quality user feedback318; 3) anatomical measurements320; 4) demographic information322; and 5) audiometric measurements324.

User-directed action and/or perception of acoustic cues316may include capturing responses of a user to locations of acoustic cues. Responses may be vocal responses of a user captured using a microphone of a computing device, gestures (for example, head and/or arm movements) of a user captured using a camera of a computing device and/or one or both of the first ear-worn device102aand the second ear-worn device102b, and user input captured via a graphical user interface (GUI) of a computing device.

Acoustic quality user feedback318may include user-directed feedback on acoustic quality (for example, responses to questions on quality metrics such as brightness, warmth, clarity, etc., responses to questions provided by a GUI or an Audio User Interface (AUI)), and observations of user behavior such as user song and/or notification preferences via, for example, a GUI or AUI.

Anatomical measurements320may include measurements of user anatomical features, such as the head, the pinna, and/or the concha, via scanning or prediction. Anatomical measurements320of one or more users may also include direct measurements (for example, via silicone ear impressions) and indirect measurements obtained via sensors and/or computer peripherals.

Demographic information322may include information provided by users such as user age or other demographics and a digital fingerprint of a user generated from one or more user features such as age, gender, and/or other user characteristics.

Audiometric measurements324may include those provided by user input and or obtained via acoustic measurements, such as in an anechoic chamber while audio signals are emitted from different known locations.

The method300may include a step304of the generation system380(for example, a model generation module386of the generation system380, seeFIG.3B) generating, modifying, and/or receiving multiple models370. The multiple models370may include one or more outer ear models354, which may include one or more pinna models356and one or more concha models358. The multiple models370may also include one or more head and torso models360and one or more canal models362. The generation system380may generate the multiple models370based on the calibration and/or personalization information obtained in step302.

For each model of the multiple models370, for each location in virtual auditory space, the generation system380may generate one or more first digital filters (for the left ear) and one or more second digital filters (for the right ear) based on the model. Accordingly, for the multiple models370, for each location in virtual auditory space, the generation system380may generate multiple first digital filters and multiple second digital filters.

For example, for the one or more head and torso models360, the generation system380may generate one or more first digital filters and one or more second digital filters that take into account shoulder width and/or breadth, head diameter, neck height, and other factors. For the one or more concha models358the generation system380may generate one or more first digital filters and one or more second digital filters that represent the acoustic effects of the physical features of the concha. These features include (but are not limited to), concha depth, width, and angle.

As another example, for the one or more pinna models356, the generation system380may generate one or more first digital filters and one or more second digital filters that represent the acoustic effects of the physical features of the pinna. These features include (but are not limited to), pinna height, width, depth, location on the head, and flare angle relative to head. For the one or more canal models362, the generation system380may generate one or more first digital filters and one or more second digital filters that take into account the physical proportions of the pinna, concha, and other ear components.

Also at the step304, for each location in virtual auditory space, the generation system380may sum, aggregate, or otherwise combine the multiple first digital filters into combined first digital filters, and may sum, aggregate, or otherwise combine the multiple second digital filters into combined second digital filters. The combined first digital filters may be or include one or more finite impulse response (FIR) filters. The combined second digital filters may also be or include one or more FIR filters. Accordingly, at the conclusion of the step304, for all the locations in virtual auditory space, there may be a set of combined first digital filters and a set of combined second digital filters.

At a step306, the generation system380may generate a mapping or association of the combined first digital filters to their corresponding locations in virtual auditory space for the left ear. The generation system380may also generate a mapping or association of the combined second digital filters to their corresponding locations in virtual auditory space for the right ear. The generation system380may utilize cartesian, polar, and/or spherical polar coordinates for the mapping or association.

At a step308, the generation system380may generate a file, a database, or other data structure that includes the mapping or association of the combined first digital filters to their corresponding locations in virtual auditory space and the mapping or association of the combined second digital filters to their corresponding locations in virtual auditory space.

At a step310, the generation system380may provide or store the file, the database, or other data structure on one or more non-transitory computer-readable media of a device. The device may be the first ear-worn device102aand/or the second ear-worn device102b, a mobile device such as a phone or a tablet, a laptop or desktop computer, another device, or any combination of the foregoing.

At a step312, the virtual auditory display system102may select the combined first digital filters and the combined second digital filters for use. After selection, at a step314, the virtual auditory display system102may utilize the combined first digital filters and the combined second digital filters in various applications, such as to render music. Various applications of the disclosed technology are discussed with reference to, for example,FIG.12.

In some embodiments, at step304, for each model of the multiple models370, the generation system380may generate one or more first digital filters and one or more second digital filters for each azimuth and elevation combination at locations in virtual auditory space of one degree increments of azimuth and elevation at a distance of one meter (1m) from a center point representing a virtual listener in virtual auditory space. The one degree increments of azimuth are from approximately negative 180 degrees, inclusive, to approximately positive 180 degrees, inclusive. The one degree increments of elevation are from approximately negative 90 degrees, inclusive, to approximately 90 degrees, inclusive. Accordingly, there are 65,160 combinations of azimuth and elevation, and therefore 65,160 locations in virtual auditory space, each location being at a distance of 1m from the center point. Therefore, the generation system380may generate 65,160 sets of one or more first digital filters and 65,160 sets of one or more second digital filters.

In some embodiments, the method300may include a step of the generation system380reducing the number of locations in virtual auditory space for which digital filters are generated or stored. For example, after step304, the generation system380may a select a proper subset from the set of combined first digital filters and a proper subset from the set of combined second digital filters.

In embodiments where there are 65,160 locations in virtual auditory space, the generation system380may select a proper subset from the set of combined first digital filters that includes approximately 7,000, such as 7,220, combined first digital filters. Similarly, the generation system380may select a proper subset from the set of combined second digital filters that includes approximately 7,000, such as 7,220, combined second digital filters.

The generation system380may select a proper subset that adequately represent locations in virtual auditory space, while reducing the amount of storage required for the sets of digital filters and reducing the amount of time to select and process digital filters. The generation system380may achieve these objectives in other ways, such as by generating mapping or associations for a reduced number of locations in virtual auditory space or storing the mapping or associations for a reduced number of locations in virtual auditory space.

In some embodiments, at step304the generation system380does not sum, aggregate, or otherwise combine the multiple first digital filters into combined first digital filters and the multiple second digital filters into combined second digital filters. Accordingly, at the conclusion of the step304, for all the locations in virtual auditory space, there may be a set of multiple first digital filters and a set of multiple second digital filters. A proper subset of the set of multiple first digital filters and a proper subset of the set of multiple second digital filters may be utilized as described herein.

In such embodiments, at step306the generation system380may instead generate a mapping or association of the multiple first digital filters to their corresponding locations in virtual auditory space for the left ear and generate a mapping or association of the multiple second digital filters to their corresponding locations in virtual auditory space for the right ear.

Further in such embodiments, at step308the generation system380may instead generate a file, a database, or other data structure that includes the mapping or association of the multiple first digital filters to their corresponding locations in virtual auditory space and the mapping or association of the multiple combined second digital filters to their corresponding locations in virtual auditory space.

In some embodiments, the generation system380generates multiple sets of digital filters for the locations in virtual auditory space. The generation system380may generate a first set of digital filters for the left ear and a first set of digital filters for the right ear as described herein. The generation system380may then generate one or more second sets of digital filters for the left ear and one or more second sets of digital filters for the right ear based on the first set of digital filters for the left ear and the first set of digital filters for the right ear. Each pair of sets may be for a different archetype representing a different user population or grouping of users.

The generation system380may generate the one or more second sets of digital filters for the left ear and the one or more second sets of digital filters for the right ear by modifying one or more parameters of the digital filters for the left ear and the digital filters for the right ear. For example, the generation system380may modify the center frequency of notch filters that are included in the first set of digital filters for the left ear and the first set of digital filters for the right ear. The generation system380may modify the center frequency of notch filters to personalize digital filters to a user, as described with reference to, for example,FIGS.15A through15F. The generation system380may do so to adjust for a delta between an actual location of a sound in virtual auditory space and the location of the sound the wearer perceives.

In some embodiments, the generation system380may generate a first set of digital filters for the left ear and a first set of digital filters for the right ear for a distance of 1m from a center point representing a virtual listener in virtual auditory space, as described herein. The generation system380may generate one or more second sets of digital filters for the left ear and the one or more second sets of digital filters for the right ear for other distances from the center point. The generation system380may generate one or more second sets of digital filters for the left ear based on the first set of digital filters for the left ear and one or more second sets of digital filters for the right ear based on the first set of digital filters for the right ear. For example, the generation system380may increase the gain of digital filters for distances closer than 1m from the center point and may decrease the gain of digital filters for distances further than 1m from the center point. Other methods will be apparent.

FIG.3Bis a block depicting components of the generation system380in some embodiments. The generation system380may include a communication module382, a filter generation module384, a model generation module386, a parameter generation module388, a parameter mask module390, a digital filter tuning module392, a user interface module394, and a data storage396.

The communication module382may send requests and/or data between components of the generation system380and any other systems, components or devices, such as the virtual auditory display system102. The communication module382may also receive requests and/or data between components of the generation system380and any other systems, components or devices.

The filter generation module384may generate digital filters and the acoustic environment digital filters. A filter may be or include one or more algorithms and, optionally, one or more parameters for the one or more algorithms.

The parameter mask module390may generate parameter modifier masks. The parameter mask module390may use image processing techniques to generate parameter modifier masks. The parameter mask module390may determine one or more parameter modifiers to one or more parameter of filters using the parameter modifier masks. The parameter mask module390may modify the one or more parameter using the one or more parameter modifiers.

The digital filter tuning module392may receive parameters for digital filters from users and modify digital filters based on the received parameters. The user interface module394may provide user interfaces that allow users to, among other things, listen to sound output from audio signals generated by application of digital filters and modify parameters of digital filters.

The data storage396may include data stored, accessed, and/or modified by any of the engines, components, modules or the like of the generation system380. The data storage396may include any number of data storage structures such as tables, databases, lists, and/or the like. The data storage396may include data that is stored in memory (for example, random access memory (RAM)), on disk, or some combination of in-memory and on-disk.

FIGS.4A-4Care graphs of frequency responses of digital audio signals in some embodiments.FIG.4Ais a graph400the frequency response for three audio signals. Each audio signal has a notch at a different center frequency. The center frequency of the notch is a factor in specifying the location in virtual auditory space of the sound corresponding to the audio signal, meaning where a user (for example, a wearer of the first ear-worn device102aand the second ear-worn device102b) perceives the location of the sound to be.

The first audio signal is for a first sound that has a first location in virtual auditory space at a distance of one (1) meter (m), zero degrees (0°) azimuth and zero degrees (0°) elevation. The second audio signal is for a second sound that has a second location in virtual auditory space at a distance of one (1) m, five degrees (5°) azimuth and zero degrees (0°) elevation The third audio signal is for a third sound that has a third location in virtual auditory space at a distance of one (1) m, ten degrees (10°) azimuth and zero degrees (0°) elevation.FIG.4Ashows the variation of the center frequency notch in the three signals due to the differences in locations in virtual auditory space. The virtual auditory display system102has applied a notch filter to the three audio signals produce each notch in each of the three frequency responses in order to produce the three sounds at the specified locations in virtual auditory space. The notch filter may be or include parametric EQ filters with the parameters being the center frequency, the gain, and the bandwidth.

FIG.4Bis a graph420of a frequency response of an audio signal to which digital filters have been applied according to some embodiments. The frequency response has three notches at three different center frequencies. The audio signal is for a sound that has a location in virtual auditory space at a distance of one (1) meter (m), zero degrees (0°) azimuth and zero degrees (0°) elevation. The virtual auditory display system102has applied three notch filters to produce the three notches in the frequency response of the audio signal. The notch filters may be or include parametric EQ filters with the parameters being the center frequency, the gain, and the bandwidth.

FIG.4Cis a graph440of frequency responses of two audio signals to which digital filters have been applied according to some embodiments. Each frequency response has a notch at a different center frequency. The virtual auditory display system102has applied three notch filters to produce the three notches in each frequency response. The notch filters may be or include parametric EQ filters with the parameters being the center frequency, the gain, and the bandwidth.

In the examples depicted inFIGS.4A-4C, the peak-to-trough decibel values of notches of the azimuthal values between negative ten degrees (−10°) to ninety-five degrees (95°) and the elevation values of between negative thirty degrees (−30°) to forty-five degrees (45°) reach<negative thirty (−30) decibels (dB). When a virtual sound-source exists within the proposed azimuth-elevation bounds, the peak-to-trough decibel value of −30 dB or more may be beneficial for producing accurate sound-source localization for the hearer.

FIG.5Adepicts a distribution500of center frequencies as a function of azimuth (x-axis) and elevation (y-axis) for the left ear.FIG.5Bdepicts a distribution550of center frequencies as a function of azimuth (x-axis) and elevation (y-axis) for the right ear. The distribution500and the distribution550indicate that, for any particular azimuth, the center frequencies follow a generally sigmoidal curve or have a generally sigmoidal shape or distribution. Similarly, for any particular elevation, the center frequencies follow a generally sigmoidal curve or have a generally sigmoidal shape or distribution.

For example,FIG.6Ais a graph600of a center frequency curve602as a function of elevation (x-axis) for the right ear where the azimuth is zero degrees (0°). The center frequency values range from about approximately 4900 Hz to about approximately 8700 Hz from negative 90 degrees elevation to 90 degrees elevation. The center frequency curve602has a generally sigmoidal shape or distribution.

Returning toFIG.5A and5B, the virtual auditory display system102may utilize the distribution500and/or the distribution550to determine the center frequencies for one or more notches in the frequency spectrums of audio signals. The virtual auditory display system102may determine the center frequences for the one or more notches based on the location of the sounds in virtual auditory space that the audio signal will cause the first ear-worn device102aand the second ear-worn device102bto produce.

That is, based on the location (as specified by, for example, azimuth and elevation) of the sounds in virtual auditory space, the virtual auditory display system102may determine the center frequencies for one or more notches in a frequency spectrum of the audio signals that cause the first ear-worn device102aand the second ear-worn device102bto produce the sounds. The virtual auditory display system102may determine the center frequencies of the first notches in the frequency spectrums of the audio signals by accessing the distribution500and the distribution550. The virtual auditory display system102may determine the center frequencies of the second notches and subsequent notches in the frequency spectrums of the audio signals based on the distribution500and the distribution550and on one or more shifts from the center frequencies obtained from the distribution500and the distribution550.

In some embodiments, in addition to or as an alternative to utilizing the distribution500and/or the distribution550, the virtual auditory display system102may utilize one or more center frequency curves, each of which may be for a different azimuth value, like the center frequency curve602ofFIG.6A, or a different elevation value. The virtual auditory display system102determines the center frequencies for one or more notches in a frequency spectrum of an audio signal, based on the location of the resulting sounds in virtual auditory space.

FIG.6Bis a graph650of user experience data of multiple trials with five different digital filters, which vary as a function of notch center frequency, in some embodiments. Each of the point652a, the point652b, the point652c, the point652d, and the point652eis the mean of 15 user trials that collected real-time user feedback on perceived sound location in virtual auditory space, for a total of 75 user trials. The bar654a, the bar654b, the bar654c, the bar654d, and the bar654eeach represent plus or minus one (1) standard deviation. The point652b, the point652c, the point652dand the point652edemonstrate that there is an observed delta of approximately 2.5° for each 150 Hz added to the notch center frequencies. The line656can be fit to the points652. The virtual auditory display system102may utilize the linear function that produced the line656to determine the center frequency to use for one or more notches based on the elevation of the sound to be produced. For example, for certain ranges of elevations (for example, between approximately zero degrees and approximately 50 degrees, or between approximately 10 degrees and approximately 40 degrees), the virtual auditory display system102may utilize the linear that produced the line656to determine one or more shifts from a center frequency in the ranges. The virtual auditory display system102may do so in addition to or as an alternative to utilizing the distribution500and/or the distribution550depicted inFIGS.5A and5B.

The parameter mask module390may use image processing algorithms to create continuous transitions of values between the particular region and the other regions to generate the parameter modifier masks with the parameter modifier values. For example, the parameter mask module390may use image processing algorithms such as a gaussian function, a sharpening function, a contrast adjustment function, a color correction function, a thresholding function, an edge detection function, and/or a segmentation function. In some embodiments, the parameter mask module390uses a gaussian blur mask to generate the parameter modifier values. The parameter mask module390may generate the parameter modifier mask for a right ear and then reflect the parameter modifier mask for the right ear about a vertical axis at an azimuth value of zero (0) to obtain the parameter modifier mask for the right ear.

The filter generation module384may utilize the parameter modifier masks depicted inFIGS.7A-7Xto select, based on a location for a sound in virtual auditory space, one or more parameter modifiers that the filter generation module384may use to modify one or more parameters to obtain one or more modified parameters. For example, the filter generation module384may utilize one or more parameter modifiers to modify the gain of digital filters. In some embodiments, the filter generation module384multiplies the one or more parameters by the one or more parameter modifiers to obtain the one or more modified parameters. Other uses of parameter modifiers will be apparent.

FIG.8Adepicts a gain distribution870for a head shadow for a left ear andFIG.8Bdepicts a gain distribution880for a head shadow for a right ear according to some embodiments. The gain distribution870depicts how a gain changes as a sound source transitions from a location872generally by the right ear to a location874generally in front of the wearer to a location876generally by the left ear. The gain distribution880depicts how a gain changes as a sound source transitions from a location882generally by the left ear to a location884generally in front of the wearer to a location886generally by the left ear.

FIG.8Cdepicts a gain distribution860of the application of digital filters to an audio signal according to some embodiments. The gain distribution890shows several notches864across a head shadow862. The several notches864are at center frequencies between 10{circumflex over ( )}3 Hz and 10{circumflex over ( )}4 Hz.

FIG.8Ddepicts user experience data for digital filters according to some embodiments and user experience data for a prior art head-related transfer function (HRTF). The prior art HRTF is used as a standard model for many past and present HRTF applications. Panel800ofFIG.8Dreports the difference between the user-perceived elevation of a virtual sound object and the actual elevation of the sound object for both the digital filters for 150 trials and the prior art HRTF for 150 trials. For the digital filters trials, point804is the mean user-perceived elevation and band802is the standard deviation of the user-perceived elevation. For the HRTF trials, point808is the mean user-perceived elevation and band806is the standard deviation of the user-perceived elevation.

Panel850ofFIG.8Dreports the difference between the user-perceived azimuth of a virtual sound object and the actual azimuth of the sound object for both the digital filters for 150 trials and the prior art HRTF for 150 trials. For the digital filters trials, point852is the mean user-perceived azimuth and band854is the standard deviation of the user-perceived azimuth. For the HRTF trials, point858is the mean user-perceived azimuth and band856is the standard deviation of the user-perceived azimuth.

For the elevation trials, the closer the elevation delta is to 0°, the more accurate the representation of the virtual sound object. Similarly, for the azimuth trials, the closer the elevation delta is to 0°, the more accurate the representation of the virtual sound object. The user experience data shows that the digital filters improve the elevation delta from a mean of approximately 30.19° with a standard deviation of approximately 12.54° to a mean of approximately −0.03° with a standard deviation of approximately 4.12°. The user experience data also shows the digital filters improves the azimuth delta from a mean of approximately −0.64° with a standard deviation of approximately 7.76° to a mean of approximately −0.02° with a standard deviation of approximately 2.04°. The data shows that the digital filters according to some embodiments improve the accuracy and precision of virtual sound objects.

FIG.9Adepicts a method900of generating digital filters according to some embodiments. The generation system380may perform the method900. The generation system380may perform the method900to generate a set of combined first digital filters and a set of combined second digital filters. The method900begins at a step902, where the generation system380(for example, the filter generation module384) may generate a generally sigmoidal distribution of center frequencies for the right ear (see, for example,FIG.5A) and a generally sigmoidal distribution of center frequencies for the left ear (see, for example,FIG.5B).

At a step906, for each location of multiple locations in virtual auditory space, the generation system380(for example, the parameter generation module388) may generate one or more first parameters for one or more first digital filters and one or more second parameters for one or more second digital filters. The one or more first parameters may include one or more first q's, one or more first gains, and one or more first center frequencies. The one or more second parameters may include one or more second q's, one or more second gains, and one or more second center frequencies.

The generation system380may utilize the generally sigmoidal distribution of center frequencies for the right ear and/or the generally sigmoidal distribution of center frequencies for the left ear to determine one or more center frequencies for one or more notches in the frequency spectrums of the audio signal for the right ear and the audio signal for the left ear. The generation system380may determine the center frequences for the one or more notches based on the location in virtual auditory space.

At a step908, for each location, the generation system380(for example, the filter generation module384) may generate one or more first digital filters including one or more first notch filters including the one or more first parameters. The generation system380may utilize the one or more first q's, the one or more first gains, and the one or more first center frequencies to generate the one or more first notch filters. The one or more first notch filters are configured to produce one or more first notches in a first frequency spectrum of a first audio signal according to the one or more first q's, the one or more first gains, and the one or more first center frequencies when the generation system380applies the one or more notch filters to an audio signal for the right ear.

At a step910, for each location, the generation system380(for example, the filter generation module384) may generate, based on the one or more first digital filters, one or more combined first digital filters for the location. In some embodiments, the one or more first digital filters are IIR filters, and the one or more combined first digital filters are FIR filters.

At a step912, for each location, the generation system380may store the one or more combined first digital filters in association with the location (in, for example, the data storage220).

At a step914, for each location, the generation system380(for example, the filter generation module384) may generate one or more second digital filters including one or more second notch filters including the one or more second parameters. The generation system380may utilize the one or more second q's, the one or more second gains, and the one or more second center frequencies to generate the one or more second notch filters. The one or more second notch filters are configured to produce one or more second notches in a second frequency spectrum of a second audio signal according to the one or more second q's, the one or more second gains, and the one or more second center frequencies when the generation system380applies the one or more notch filters to an audio signal for the left ear.

At a step916, for each location, the generation system380(for example, the filter generation module384) may generate, based on the one or more second digital filters, one or more combined second digital filters for the location. In some embodiments, the one or more second digital filters are IIR filters, and the one or more combined second digital filters are FIR filters.

At a step918, for each location, the generation system380may store the one or more combined second digital filters in association with the location (in, for example, the data storage220).

At a step920the generation system380tests to see if there are more locations for which the generation system380is to generate digital filters. If so, the method900returns to step906. The generation system380may perform the method900multiple times to generate multiple sets of combined first digital filters and multiple sets of combined second digital filters. Each pair of sets of digital filters may be for a different archetype

In some embodiments, the generation system380may perform the method900several times to generate multiple sets of combined first digital filters and combined second digital filters. Each pair of sets may be for a different archetype representing a different user population or grouping of users.

FIG.9Bdepicts a method950of generating digital filters in some embodiments. The generation system380may perform the method900. The method950includes certain steps that may be generally similar to certain steps of the method900. The generation system380(for example, various components of the generation system380) may perform the method950. The generation system380may perform the method950to generate a set of first digital filters and a set of second digital filters.

The method950begins at a step952, where the generation system380(for example, the parameter generation module388) may generate a first generally sigmoidal distribution of center frequencies and a second generally sigmoidal distribution of center frequencies. At a step954the generation system380(for example, the parameter mask module390) may generate first parameter modifier masks and second parameter modifier masks.

At a step956, for each of multiple virtual auditory space locations, the generation system380(for example, the parameter generation module388) may generate one or more first parameters for one or more first digital filters and one or more second parameters for one or more second digital filters. The one or more first parameters may include one or more first q's, one or more first gains, and one or more first center frequencies. The one or more second parameters may include one or more second q's, one or more second gains, and one or more second center frequencies.

At a step958, for each virtual auditory space location, the generation system380may generate one or more first digital filters including one or more first notch filters including the one or more first parameters. Step958is generally similar to step980of the method900.

At a step960, for each virtual auditory space location, the generation system380may store the one or more first digital filters in association with the virtual auditory space location. Step960is generally similar to step912of the method900.

At a step962, for each virtual auditory space location, the generation system380may generate one or more second digital filters including one or more second notch filters including the one or more second parameters. Step962is generally similar to step914of the method900.

At a step964, for each virtual auditory space location, the generation system380may store the one or more second digital filters in association with the virtual auditory space location. Step964is generally similar to step918of the method900.

At a step966the generation system380tests to see if there are more virtual auditory space locations for which the generation system380is to generate digital filters. If so, the method900returns to step956. The generation system380may perform the method950multiple times to generate multiple sets of one or more first digital filters and multiple sets of one or more second digital filters.

The method900and the method950may include additional steps. For example, the generation system380may provide for testing digital filters. The generation system380(for example, the user interface module394) may provide a user interface that allows for sound generated by audio signals to which digital filters have been applied to be played. A user may listen to the sounds and determine that one or more parameters of the digital filters should be modified. For example, the user may modify parameters of digital filters to ensure that sounds meet required thresholds of tonal quality, clarity, brightness, and the like. The generation system380may provide another user interface that allows the user to modify the one or more parameters of the digital filters. The generation system380(for example, the digital filter tuning module392) may receive the one or more parameters of the digital filters from users and modify digital filters based on the received one or more parameters.

FIG.10Adepicts a method1000of applying digital filters according to some embodiments. The virtual auditory display system102and the virtual auditory display device100may perform the method1000. The method1000begins at a step1002, where the virtual auditory display system102(for example, the binauralizer138) receives a set of combined first digital filters and a set of combined second digital filters. At a step1004the virtual auditory display system102(for example, the binauralizer138) receives an input audio signal that includes one or more audio sub-signals. Each audio sub-signal of the one or more audio sub-signals has a location in virtual auditory space.

While receiving the input audio signal, the virtual auditory display system102performs step1006through step1020of the method1000. At step1006one or both of the first ear-worn device102aand the second ear-worn device102bdetects a head orientation of the user wearing the first ear-worn device102aand the second ear-worn device102b. The first ear-worn device102aand/or the second ear-worn device102bprovide the head orientation to the virtual auditory display system102.

At a step1008, for each audio sub-signal of the one or more audio sub-signals, the virtual auditory display system102determines, based on the location of the audio sub-signal and the head orientation, a particular location in the virtual auditory space. At a step1010, for each audio sub-signal, the virtual auditory display system102selects, based on the particular location, particular one or more combined first digital filters and particular one or more combined second digital filters.

At a step1012, for each audio sub-signal, the virtual auditory display system102applies the particular one or more combined first digital filters to the audio sub-signal to obtain a first processed audio sub-signal. At a step1014, for each audio sub-signal, the virtual auditory display system102applies the particular one or more combined second digital filters to the audio sub-signal to obtain a second processed audio sub-signal.

At a step1016the virtual auditory display system102tests to see if there are more audio sub-signals to process. If so, the method1000returns to step1008. If not the method1000continues to step1018. After processing all the audio sub-signals the virtual auditory display system102obtains multiple first processed audio sub-signals and multiple second processed audio sub-signals.

At a step1018the virtual auditory display system102generates, based on the multiple first processed audio sub-signals, a first output audio signal for the left ear-worn device, and based on the multiple second processed audio sub-signals, a second output audio signal for the right ear-worn device. The virtual auditory display system102provides the first output audio signal to the first ear-worn device102aand the second output audio signal to the second ear-worn device102b.

At a step1020the first ear-worn device102aoutputs first sound based on the first output audio signal and the second ear-worn device102boutputs second sound based on the second output audio signal. The virtual auditory display system102may thus utilize the method1000to provide virtual auditory display sound based on an audio signal that may have multiple audio sub-signals (or channels) that would typically require multiple speakers to produce a surround sound effect. The virtual auditory display system102may provide the virtual auditory display sound to the user using only the first ear-worn device102aand the second ear-worn device102b.

FIG.10Bdepicts a method1050of applying digital filters according to some embodiments. The method1050includes certain steps that may be generally similar to certain steps of the method1000. The virtual auditory display system102may perform the method1050.

The method1050begins at a step1052, where the virtual auditory display system102(for example, the binauralizer138) receives a set of one or more first digital filters and a set of one or more second digital filters. At a step1054the virtual auditory display system102(for example, the binauralizer138) receives an audio signal that has one or more audio sub-signals. Each audio sub-signal of the one or more audio sub-signals is associated with a virtual auditory space location.

At a step1056the virtual auditory display system102receives a head orientation of a user. At a step1058, for each audio sub-signal of the one or more audio sub-signals, the virtual auditory display system102determines, based on the virtual auditory space location and the head orientation, a particular virtual auditory space location. At a step1060, for each audio sub-signal, the virtual auditory display system102selects, based on the virtual auditory space location or the particular virtual auditory space location, particular one or more first digital filters and particular one or more second digital filters.

At a step1062, for each audio sub-signal, the virtual auditory display system102applies the particular one or more first digital filters to the audio sub-signal to obtain a first processed audio sub-signal. At a step1064, for each audio sub-signal, the virtual auditory display system102applies the particular one or more second digital filters to the audio sub-signal to obtain a second processed audio sub-signal.

At a step1066the virtual auditory display system102tests to see if there are more audio sub-signals to process. If so, the method1050returns to step1058. If not the method1000continues to step1068. After processing all the audio sub-signals the virtual auditory display system102obtains multiple first processed audio sub-signals and multiple second processed audio sub-signals.

At a step1068the virtual auditory display system102generates, based on multiple first processed audio sub-signals, a first output audio signal for a first device, and based on multiple second processed audio sub-signals, a second output audio signal for a second device. The first device may be or include, for example, the first ear-worn device102a, and the second device may be or include, for example, the second ear-worn device102b. At a step1070the virtual auditory display system102provides the first output audio signal to the first device and the second output audio signal to the second device.

FIG.10Cdepicts a method1080of generating and applying virtual auditory display filters in some embodiments. The generation system380and the virtual auditory display system102may perform the method1000. The method1080begins at a step1082where the generation system380(for example, the parameter generation module388) may generate a first generally sigmoidal distribution of center frequencies and a second generally sigmoidal distribution of center frequencies. At a step1084the generation system380(for example, the parameter mask module390) may generate first parameter modifier masks and second parameter modifier masks, including a first notch parameter modifier mask and a second notch parameter modifier mask.

At a step1086the generation system380(for example, the filter generation module384) may generate a first virtual auditory display filter and a second virtual auditory display filter. The first virtual auditory display filter may include a first set of first functions. One or more first functions, when applied to a first audio signal having a first location in virtual auditory space, may generate a first processed audio signal having a first frequency response with one or more first notches at one or more first center frequencies that are based on the first location. The one or more first notches may have one or more first peak-to-trough depths of at most −10 dB (for example, approximately −30 dB).

The second virtual auditory display filter may include a second set of second functions. One or more second functions, when applied to the first audio signal, may generate a second processed audio signal having a second frequency response with one or more second notches at one or more second center frequencies that are based on the second location. The one or more second notches may have one or more second peak-to-trough depths of at most −10 dB (for example, approximately −30 dB).

At a step1088the virtual auditory display system102may receive an audio signal having a second location in the virtual auditory space. For example, the virtual auditory display system102may receive the audio signal from a digital device on which the virtual auditory display system102is executing. At a step1090the virtual auditory display system102may receive a head orientation of a user, for example, from the virtual auditory display device100that the user is utilizing.

At a step1092the virtual auditory display system102may apply the first virtual auditory display filter, including a first subset of first functions selected based on the second location, to the second audio signal to generate a third processed audio signal having a third frequency response. At a step1094the virtual auditory display system102may apply the second virtual auditory display filter, including a second subset of second functions selected based on the second location, to the second audio signal to generate a fourth processed audio signal having a fourth frequency response. At a step1094the virtual auditory display system102may provide the first processed audio signal to a first sound output device (for example, the first ear-worn device102a) and the second processed audio signal to a second sound output device (for example, the second ear-worn device102b).

The virtual auditory display system102may perform step1088through step1094of the method1080while the virtual auditory display system102is receiving an input audio signal that may correspond to, for example, a song file, an audio stream, a podcast, or any other audio.

The method1000, the method1050, and the method1080may include additional steps not illustrated inFIGS.10A through10C. For example, these methods may include a step of the virtual auditory display system102receiving a selection of an acoustic environment and a step of the virtual auditory display system102determining, based on the acoustic environment, a first acoustic environment digital filter and a second acoustic environment digital filter. The acoustic environment may be represented by one or more ambisonic arrays. The virtual auditory display system102may determine the acoustic environment digital filters based on the one or more ambisonics arrays. The virtual auditory display system102may apply the digital filters and the acoustic environment digital filters to obtain the processed audio sub-signals. Other modifications to these methods will be apparent.

FIG.2Cis a block diagram depicting a process290for generating acoustic environment digital filters in some embodiments. A first speaker292aand a second speaker292bmay be in a particular acoustic environment, such as a concert hall, a vehicle, a night club, or the like. Sound output by the first speaker292aand the second speaker292bare captured by a microphone294and converted into signals. The virtual auditory display system102(for example, the filter generation module384) generates one or more ambisonics digital filters296based on the signals. The one or more ambisonics digital filters296are a set of one or more acoustic environment digital filters298.

FIG.2Dis a block diagram depicting operations of a spatialization engine270of the virtual auditory display system102in some embodiments. The spatialization engine270may be part of the binauralizer138or a separate component. The spatialization engine270receives a user interface (UI) selected acoustic environment at a block272and determines acoustic environment digital filters based on the selected acoustic environment at a block274. The spatialization engine270receives coordinates and the input audio signal of decoded audio objects at a block276. At a block278the spatialization engine270matches an index of the acoustic environment digital filters to an input audio signal index. At a block280the spatialization engine270applies a convolution matrix to the output of the block278and the coordinates and the input audio signal.

At a block282the spatialization engine270receives the user head orientation and audio source distance signal. At a block284the spatialization engine performs an ambisonics to binaural conversion based on the output of the block280and the user head orientation and audio source distance signal, by applying digital filters to the audio signals received at a block276based on the locations of the audio signals in virtual auditory space. At a block286the spatialization engine270outputs the audio signal for the left ear and at a block288the spatialization engine270outputs the audio signal for the right ear.

FIGS.11A and11Bdepict an example user interface1100for displaying a representation of a virtual audio display in some embodiments. The virtual auditory display system102(for example, the user interface module210) may provide the user interface1100.FIGS.11A and11Bare described with reference to the virtual auditory display device100, but the virtual auditory display system102may provide the user interface1100for other devices.

The user interface1100includes an icon1104labeled “VAD” indicating that the virtual auditory display device100is connected to the virtual auditory display system102and an icon1102labeled “IMU” indicating that the IMU-based sensor systems of the virtual auditory display device100are calibrated. The user interface1100also includes an encoding representation dropdown1114that allows the wearer to select how the virtual auditory display system102should represent audio received by the virtual auditory display system102. Example encoding representations are mono (a single audio channel), stereo (two channels of audio), 5.1 5.1 (six channels of audio), 7.1 (eight channels of audio), 7.1.4 (12 channels of audio), and 9.1.6 (16 channels of audio).

The user interface1100also includes an acoustic environment dropdown1116that allows the wearer to select an acoustic environment in which the virtual auditory display system102should render the virtual auditory display. Example acoustic environments include a dry acoustic environment, a studio acoustic environment, a car acoustic environment, a phone acoustic environment, a club acoustic environment, and a headphone acoustic environment. The virtual auditory display system102may select an acoustic environment digital filter based on the selected acoustic environment and apply the acoustic environment digital filter along with the virtual auditory display filters. The virtual auditory display sounds will sound different for the wearer based on the selected acoustic environment. The user interface1100also includes an output volume slider1122allowing the wearer to adjust the volume of the sound output by the virtual auditory display device100.

The user interface1100also includes a representation1108of a virtual audio display. InFIG.11A, the virtual auditory display system102depicts the representation1108as a virtual sphere surrounding a head1112representing the head of the wearer from a top rear perspective. The user interface1100also displays sounds at their locations in virtual auditory space relative to the head of the wearer at corresponding locations relative to the head1112on the representation1108. For example, sounds1110aare depicted as to the left of, below, and to the rear of the head1112. This is because the actual sounds corresponding to the sounds1110ahave those locations in virtual auditory space. Sounds1110bare depicted as above, behind, and slightly to the left of the head1112, sounds1110care depicted as in front of and to the right of the head1112, and sounds1110dare depicted as to the right of, below, and to the rear of the head1112.

While outputting sounds, the virtual auditory display device100detects head orientations of the wearer and sends the head orientation to the virtual auditory display system102. The virtual auditory display system102updates the representation1108based on the detected head orientations. The virtual auditory display system102may move the head1112and the sounds1110based on the detected head orientations.

The user interface1100also includes a virtual auditory display representation dropdown1118that allows the wearer to select how the virtual auditory display system102provides the virtual auditory display representation. Example virtual auditory display representations include a custom representation (depicted inFIG.11A), which provides the wearer a top right rear perspective of the representation1108, a top representation (depicted inFIG.11B), which provides the wearer a top perspective of the representation1108(from the top of the sphere inFIG.11A), and a back representation, which provides the wearer a rear perspective of the representation1108(from the rear of the sphere inFIG.11A).

The user interface1100also includes a location button1120which, if selected by the wearer, may cause the virtual auditory display system102to change the representation1108such that the location specified by a certain coordinate (for example, zero degrees azimuth, zero degrees elevation) may be directly in front of the head1112. The user interface1100also includes a settings icon1106which, if selected by the wearer, may cause the virtual auditory display system102to provide an example user interface for adjusting settings for a virtual audio display.

FIG.11Cdepicts an example user interface1150for adjusting settings for a virtual audio display in some embodiments. The virtual auditory display system102(for example, the user interface module210) may provide the user interface1150. The user interface1150includes an icon1152labeled “IMU” indicating that the IMU-based sensor systems of the virtual auditory display device100are calibrated. The user interface1150also includes a button1154labeled “Recalibrate” that the wearer may select to have the virtual auditory display system102perform the calibration part of the calibration and/or personalization process.

The user interface1150also includes an icon1156labeled “VAD” indicating that the virtual auditory display device100is connected to the virtual auditory display system102, a recommendation1158of a virtual auditory display filter, and a button1160labeled “Personalize” that the wearer may select to have the virtual auditory display system102perform the personalization part of the calibration and/or personalization process. The user interface1150also indicates the spatialization precision estimate of the virtual auditory display of the wearer and a button1162labeled “Test” that the wearer may select to have the virtual auditory display system102provide a test procedure for the wearer to allow the wearer to test to see if the wearer can accurately locate virtual auditory display sounds.

The user interface1150also includes a group1164of icons (labeled “A” through “G”) that indicates the set of virtual auditory display filters that create the virtual auditory display for the wearer. As depicted, the current set of virtual auditory display filters is “VAD C.” The wearer may select a different set of virtual auditory display filters by selecting a different icon in the group1164. The wearer may then perform the calibration part of the calibration and/or personalization process by selecting the button1154and/or perform the personalization part of the calibration and/or personalization process by selecting the button1160.

The user interface1150also allows the wearer to select a custom set of digital filters for the virtual auditory display system102to use to generate the virtual auditory display. The wearer may do so by selecting a button1168labeled “Upload,” which allows the wearer to upload a file containing a custom set of digital filters to the virtual auditory display system102. The user interface element1166may then display the name of the file. This functionality may be desirable for wearers who already have a custom HRTF and want the virtual auditory display system102to utilize the custom HRTF.

FIG.12is multiple images1200depicting example use cases of virtual auditory display filter technology described in this application in some embodiments. One example use case is for improved virtual surround sound for television or movies using only a pair of speakers, as depicted in image1202. A group of example use cases relates to producing or listening to music. Image1210depicts an example use case of listening to music on headphones. The virtual auditory display filter technology may render music played on headphones as indistinguishable from music played using physical surround sound installations.

Image1218depicts the use of virtual auditory display filter technology in virtual monitors to provide noise isolation, sound quality, and virtualization to musicians. Image1204depicts the use of the virtual auditory display filter technology to mix music in any acoustic environment. Image1220depicts how the virtual auditory display filter technology may provide a listening experience that reinvigorates the music that listeners love for them. Image1212depicts using virtual auditory display filter technology in games to provide an immersive gaming experience. virtual auditory display filter technology may allow users to hear sounds emanating from locations that are not shown on users' displays and thus improve users' awareness.

Another group of example use cases relate to military, non-military (for example, first responders such as police and firefighters) and/or other organizational applications. For example, military personnel may use military radio systems to communicate with fellow soldiers, commanders, and other military personnel. The present technology may be utilized in scenarios including military operations, emergency services, aviation, marine and others.

Image1206depicts virtual auditory display filter technology providing improved voice pickup and voice display for organizational communications. Image1214depicts virtual auditory display filter technology providing augmentation of visual instrumentation with auditory signals in maritime operations. Image1222depicts virtual auditory display filter technology providing hyper-realistic virtual audio environments which facilitates virtual training for military and/or non-military personnel.

Image1208depicts virtual auditory display filter technology providing audio augmentation for orientation awareness for combat infantry situational awareness. Image1216depicts virtual auditory display filter technology providing audio augmentation for orientation awareness for air force orientation control. For example, pilots may use the localization of virtual beacons to assist in situational awareness. Image1224depicts virtual auditory display filter technology providing audio augmentation for hyper-situational awareness for unmanned aerial vehicle operations.

Another example use case of virtual auditory display filter technology involves phone calls or video conferences. For example, multiple people may talk at the same time in a phone call or video conference, making it difficult for listeners to focus on the speaker they want to hear, which may lead to confusion and misunderstandings. The present technology allows users to virtually select which talker they want to listen to through the simple movement of a head or other gesture. This attention selection mechanism may help avoid confusion and make meetings more productive.

As another example, air traffic controllers use radio messages to communicate with pilots. The air traffic controllers monitor the position, speed and altitude of aircraft in their assigned airspace visually and by radar and give directions to the pilots by radio. Often, air traffic controllers will need to communicate with multiple pilots simultaneously. Today, these situations requiring multiple pilot communications are addressed by physical switch boards that do not allow for user directed attention selection. The present technology may allow an air traffic controller to use gestures (for example, movement of a head or a hand) or other actions to localize radio communications so that there is seamless attention selection. In a simple example, multiple radio communication signals are statically arranged in unique virtual locations. The air traffic controller then looks at these predefined locations to hear the radio signal. In other examples, the radio communication signals are dynamically updated with the position, speed and altitude of the aircraft.

Other example use cases include virtual auditory display notifications to localize notifications such as voice, text-to-speech messages, email alerts, phone messages or other audio notifications, spatial navigation to use virtual auditory display cues to communicate direction and distance of virtual or real objects, which may also be used for wayfinding or orientation awareness, and spatial ambience to give a user a virtual sound environment that can be mixed with local or virtual sounds (for example, to experience music as if in a concert hall). Other example use cases of virtual auditory display filter technology are possible.

FIGS.13A and13Bare diagrams of a method of personalizing digital filters in some embodiments that involves providing an action (for example, playing a sound) and detecting a user perception of the action. As described herein, the user may indicate perception of the action in various ways, such as by pointing his or her head, making one or more gestures, indicating where the sound is on a graphical user interface, and the like.

FIG.13Adepicts a view1300indicating how the location of a sound in virtual auditory space as indicated by azimuth1314and elevation1316may be perceived by a user.FIG.13Bdepicts a view1350indicating how of how the location of a sound in virtual auditory space as indicated by distance1318and elevation1316may be perceived by a user. In both the view1300and the view1350, the user1302is wearing the first ear-worn device102aand the second ear-worn device102b(not illustrated inFIGS.13A and13B). The virtual auditory display system102may provide instructions to the user1302to follow a sound in virtual auditory space with the user's head as the user hears the sound. The virtual auditory display system102may then generate audio signals that cause the first ear-worn device102aand the second ear-worn device102bto output one or more sounds that have a location1304in virtual auditory space. As the user1302hears the one or more sounds, the user1302may point his or her head towards a perceived location1306the user perceives the sound to be coming from, which may be different from the location1304. In pointing his or her head towards the perceived location1306, it may appear as if the user1302is looking in the direction of where the user1302perceives the sound to be. Other gestures the user may make with his or her head include nodding, shaking, tilting, and turning. Other head gestures will be apparent.

As the user1302moves his or her head to point towards the perceived location1306of the one or more sounds, one or both of the first ear-worn device102aand the second ear-worn device102b(for example, using the IMU sensor system252and/or the magnetometer254) may detect a head orientation of the user1302. The virtual auditory display system102may utilize the detected head orientation to determine the perceived location1306. The virtual auditory display system102may then determine a delta1308between the location1304and the perceived location1306. The virtual auditory display system102may calculate the delta1308based on differences between the azimuth1314and/or elevation1316of the location1304and the azimuth1314and/or elevation1316of the perceived location1306.

The user1302may user other gestures to indicate distance, such as extending an arm a specified amount, and the virtual auditory display system102may use such gestures to determine the delta1308based on differences between the distance1318of the location1304and the distance1318of the perceived location1306.

The virtual auditory display system102may generate audio signals that cause the first ear-worn device102aand the second ear-worn device102bto output one or more subsequent sounds whose locations in virtual auditory space change, as indicated by solid line1310. The user1302may move his or her head to follow the movement of the one or more subsequent sounds, and the perceived locations of the one or more sounds may change, as indicated by dashed line1312.

One or both of the first ear-worn device102aand the second ear-worn device102bmay detect subsequent head orientations of the user1302as he or she moves his or her head. The virtual auditory display system102may utilize the detected subsequent head orientations to determine perceived locations of the one or more subsequent sounds. The virtual auditory display system102may then determine one or more subsequent deltas between the location of the one or more subsequent sounds and the perceived locations of the one or more subsequent sounds.

The virtual auditory display system102may store the deltas that the virtual auditory display system102determines and utilize the deltas to modify the digital filters so as to cause the user1302to perceive the location of sounds in virtual auditory space to be closer to the actual locations in virtual auditory space. In some embodiments, the virtual auditory display system102modifies the digital filters by selecting a different set of digital filters that the virtual auditory display system102determines will reduce or minimize the deltas for the user1302. The virtual auditory display system102may then use the different set of digital filters for the user1302.

In some embodiments, the virtual auditory display system102modifies the parameters of the digital filters. For example, the virtual auditory display system102may modify parameters such as center frequencies, gains, and/or q's. For example, the user1302may have an elevation delta of several degrees. The virtual auditory display system102may modify the center frequency of a notch, a pair of notches, or a group of notches (see, for example,FIG.6B) to modify the elevation of a virtual sound object and thereby reduce the elevation delta. The virtual auditory display system102may then utilize the modified digital filters during real-time audio playback.

The virtual auditory display system102may repeat the personalization procedure one or more times until the virtual auditory display system102determines that the deltas are within certain ranges or thresholds.

In some embodiments, the virtual auditory display system102and/or other devices may capture other actions that the user may make in response to the user hearing a sound in virtual auditory space to indicate where the user perceives the location of the sound to be. Example actions may include vocal responses by the user, gestures by the user using parts of the user's body other than the user's head (for example, pointing with a finger or an arm of the user, waving, clapping, tapping and hand signals). Such actions may be captured by a device connected to the virtual auditory display system102, such as a microphone, camera or a motion sensing device.

Other example actions include the user indicating the perceived location of the sound using a graphical user interface and/or user input devices of a digital device such as a phone, tablet or laptop or desktop computer. For example, the virtual auditory display system102may provide a graphical user interface that graphically represents virtual auditory space for the user, and the user may utilize an input device (mouse, keyboard, touchscreen, and/or voice command) to indicate the perceived location of the sound on the graphical representation of the virtual auditory space. It will be appreciated that there are various methods to capture user actions in response to the user perception of the location of the sound, and that the virtual auditory display system102, optionally in cooperation with other devices, may utilize the various methods.

FIG.14Adepicts a method1400of personalizing digital filters in some embodiments. The virtual auditory display system102may perform the method1400. The method1400begins at a step1402where the virtual auditory display system102(for example, the binauralizer138) receives a personalization audio signal that has a first location in virtual auditory space. At a step1404the virtual auditory display system102(for example, the binauralizer138) determines, based on the first location, a first particular first location in the virtual auditory space.

At a step1406the virtual auditory display system102selects, based on the first particular first location, particular one or more combined first digital filters from a first set of combined first digital filters and particular one or more combined second digital filters from a first set of combined second digital filters.

At a step1408the virtual auditory display system102applies the particular one or more combined first digital filters to the personalization audio signal to obtain a first processed personalization audio signal and the particular one or more combined second digital filters to the personalization audio signal to obtain a second processed personalization audio signal.

At a step1410the virtual auditory display system102generates, based on the first processed personalization audio signal, a first output audio signal for a left ear-worn device, and based on the second processed personalization audio signal, a second output audio signal for a right ear-worn device. At a step1412the left ear-worn device outputs first sound based on the first output audio signal and the right ear-worn device outputs second sound based on the second output audio signal.

At a step1414one or both of the left ear-worn device and the right ear-worn device detects a head orientation of a user wearing the left ear-worn device and the right ear-worn device. At a step1416the virtual auditory display system102determines, based on the head orientation, a second particular first location in the virtual auditory space.

At a step1418the virtual auditory display system102determines a delta between the first particular first location and the second particular first location. At a step1420the virtual auditory display system102selects, based on the delta, a second set of combined first digital filters and a second set of combined second digital filters. The virtual auditory display system102may use the second set of combined first digital filters and the second set of combined second digital filters while receiving a subsequent input audio signal.

FIG.14Bdepicts a method1450of personalizing digital filters in some embodiments. The method1450includes certain steps that may be generally similar to certain steps of the method1400. The virtual auditory display system102(for example, various components of the virtual auditory display system102) may perform the method1450.

At a step1452the virtual auditory display system102receives a set of multiple first digital filters. At a step1454the virtual auditory display system102receives a set of multiple second digital filters. There are one or more first digital filters and one or more second digital filters generated for each of multiple virtual auditory space locations.

At a step1456the virtual auditory display system102receives personalization information for a user. Personalization information may include user-directed action or perception of acoustic cues, acoustic quality information, user anatomical measurements, user demographic information, and/or user audiometric measurements.

At a step1458the virtual auditory display system102modifies, based on the personalization information for the user, the set of multiple first digital filters. At a step1460the virtual auditory display system102modifies, based on the personalization information for the user, the set of multiple second digital filters.

In some embodiments, modifying, based on the personalization information, the set of multiple first digital filters includes modifying one or more first center frequencies of the multiple first digital filters. Moreover, modifying, based on the personalization information, the set of multiple second digital filters includes modifying one or more second center frequencies of the multiple second digital filters.

In some embodiments, modifying, based on the personalization information, the set of multiple first digital filters includes selecting a different set of multiple first digital filters. Further, modifying, based on the personalization information, the set of multiple second digital filters includes selecting a different set of multiple second digital filters.

The virtual auditory display system102may provide a calibration and/or personalization process that allows a wearer of a virtual auditory display device to calibrate the virtual auditory display device and/or to personalize a virtual auditory display provided by the virtual auditory display device.

The calibration and/or personalization process may include a calibration part and a personalization part. The virtual auditory display device may include an inertial measurement unit (IMU). Calibrating the virtual auditory display device may refer to calibrating the IMU. Personalizing the virtual auditory display may refer to selecting a set of virtual auditory display filters for the wearer and/or modifying an existing set of virtual auditory display filters so that the virtual auditory display provided by the virtual auditory display device is customized to the wearer. The virtual auditory display system102may allow the wearer to perform both the calibration part and the personalization part of the calibration and/or personalization process, just the calibration part, or just the personalization part.

FIGS.15A through15Cdepict an example user interface1500for calibrating a virtual auditory display device in some embodiments. The virtual auditory display device may be the virtual auditory display device100which includes the first ear-worn device102aand the second ear-worn device102b.FIGS.15A through15Fare described with reference to the virtual auditory display device100, but other virtual auditory display devices may be calibrated and/or personalized.

The virtual auditory display system102(for example, the user interface module210) may provide the user interface1500. The wearer may start a calibration and/or personalization process by selecting a button labeled “Start” (not shown inFIGS.15A through15C) displayed by the virtual auditory display system102.FIG.15Adepicts the user interface1500providing a user interface element1502indicating the point in the calibration part of the calibration and/or personalization process at which the wearer is, and instructions1504for the wearer.

FIG.15Bdepicts the user interface1500providing a first circle1506aand a second circle1506b. The virtual auditory display system102may cause the first circle1506aand/or the second circle1506bto move up and down on the user interface1500and instruct the wearer to nod their head up and down to follow the first circle1506aand the second circle1506b.

FIG.15Cdepicts the user interface1500providing a circle1508. The virtual auditory display system102may cause the circle1508to move up and down on the user interface1500and instruct the wearer to nod their head up and down to follow the circle1508. Additionally or alternatively, the virtual auditory display system102may cause the circle1508to move from side to side on the user interface1500and instruct the wearer to move their head from side to side to follow the circle1508.

While the virtual auditory display system102performs the calibration part of the calibration and/or personalization process, the virtual auditory display system102may receive detections of head orientations of the head of the wearer from the virtual auditory display device100based on data obtained from the IMU-based sensor system and/or other sensors of the first ear-worn device102aand/or the second ear-worn device102b. The virtual auditory display system102may use the detections of head orientations and other factors, such as a known or estimated distance from a display providing the user interface1500, a width and height of the display, positions of the first circle1506a, the second circle1506b, and/or the circle1508, and/or other data from the IMU-based sensor system to calibrate the IMU-based sensor system.

FIGS.15D through15Fdepict an example user interface1550for personalizing a virtual auditory display provided by a virtual auditory display device in some embodiments. The virtual auditory display system102(for example, the user interface module210) may provide the user interface1550.

The wearer of the virtual auditory display device100may start the personalization part of the calibration and/or personalization process after completing the calibration part. The virtual auditory display system102may cause the virtual auditory display device100to play sounds at several locations (for example, five locations). The sounds may include, for example, sounds produced by objects that appear to the wearer as moving around his or her head, such as airplanes, helicopters, birds, and other flying creatures.FIG.15Ddepicts the user interface1550providing instructions1554instructing the wearer to point their nose at the source of each sound as the virtual auditory display device100plays the sound. The wearer may begin the personalization part of the calibration and/or personalization process by selecting the button1556labeled “Continue.”

FIG.15Edepicts the user interface1550providing a user interface element1552indicating the point in the personalization part of the calibration and/or personalization process at which the wearer is, and the instructions1554.FIG.15Fdepicts the user interface1550with the user interface element1552indicating that the wearer has located a sound that the virtual auditory display device100played at a first location. The virtual auditory display system102may cause the virtual auditory display device100to play sounds at subsequent locations and update the user interface1550accordingly.

While the virtual auditory display system102performs the personalization part of the calibration and/or personalization process, the virtual auditory display system102may receive detections of head orientations of the head of the wearer from the virtual auditory display device100based on data obtained from the IMU-based sensor system and/or other sensors of the first ear-worn device102aand/or the second ear-worn device102b. The virtual auditory display system102may use the detections of head orientations and the locations of the sounds generated by the virtual auditory display system102to calculate one or more deltas, as described with reference to, for example,FIGS.13A and13B. The virtual auditory display system102may use the calculated one or more deltas to select a set of virtual auditory display filters and estimate a spatialization precision of the virtual auditory display for the wearer.

FIGS.15G through15Jdepict an example user interface1570for providing information on calibration of a virtual auditory display device and personalization of a virtual auditory display of the virtual auditory display device in some embodiments. The user interface1570includes a recommendation1572of a set of virtual auditory display filters. In some embodiments, as described herein with reference to, for example,FIGS.13A and13B, the virtual auditory display system102may select a set of virtual auditory display filters from among multiple sets of virtual auditory display filters based on the results of the calibration and/or personalization process. The user interface1570also includes an estimate1574of a spatialization precision of the virtual auditory display for the wearer.

The virtual auditory display system102may categorize the spatialization precision of the virtual auditory display for the wearer based on the estimate1574, such as “Very Good” (FIG.15G), “Medium” (FIG.15H), and “Poor” (FIG.15I). The virtual auditory display system102may provide recommendations to redo the calibration part and/or the personalization part of the calibration and/or personalization process and/or to use custom filters. The user interface1570also includes a button1576labeled “Continue” that the wearer may select to return to the user interface1100depicted inFIGS.11A and11B.

Although the virtual auditory display system102is described as using circles, the virtual auditory display system102may utilize other visual user interface elements in the calibration and/or personalization process. Furthermore, although the virtual auditory display system102is described as receiving detections of head orientations from the virtual auditory display device100in the calibration and/or personalization process, the virtual auditory display system102may receive detections of head orientations from other devices connected to the virtual auditory display system102, such as cameras, motion sensing devices, virtual reality headsets, and the like.

One advantage of the calibration and/or personalization process is that the virtual auditory display system102may personalize a set of virtual auditory display filters for a wide range of individuals. The virtual auditory display system102may personalize the set of virtual auditory display filters by modifying the set of virtual auditory display filters. The virtual auditory display system102may have pre-configured multiple sets of virtual auditory display filters and may modify the set of virtual auditory display filters by selecting a different set of virtual auditory display filters based on the results of the calibration and/or personalization process for a user.

Additionally or alternatively, the virtual auditory display system102may modify the set of virtual auditory display filters by modifying the digital filters or functions included in the set of virtual auditory display filters. For example, where the set of virtual auditory display filters includes digital filters, the virtual auditory display system102may modify parameters of the digital filters, such as the center frequencies, gains, q's, algorithm type, or other parameters based on the results of the calibration and/or personalization process for a user.

Personalization of virtual auditory display filters allows a wide range of individuals to experience immersive, accurately rendered sound in a virtual auditory space. Moreover, such individuals would not have to have HRTFs generated for them using potentially difficult and/or unreliable physical measurement procedures. Such individuals could obtain a personalized set of virtual auditory display filters simply by having the virtual auditory display system102perform the calibration and/or personalization process for them. The modification of virtual auditory display filters may be performed at an initial setup procedure for the person and at any subsequent point during the person's use of the virtual auditory display system102and/or virtual auditory display device100.

One advantage of virtual auditory display filters is that sounds in far more locations in virtual auditory space may be rendered in comparison to existing technologies. For example, a 9.1.6 configuration has 16 virtual speakers and thus may be limited to accurately rendering sounds for only those 16 virtual speaker locations. Such configurations may render sounds from other locations by smearing sounds from virtual speaker locations to represent the other locations, but such artifacts may be noticeable to listeners.

In contrast, virtual auditory display filters may be able to render sound at far more locations. For example, using locations at one degree increments of azimuth and elevation results in 65,160 locations. However, the described technology may generate virtual auditory display filters at smaller increments, resulting in even more locations at which the virtual auditory display filters may render sound. Moreover, typical approaches render sound at a modeled distance of 1 m from a center point representing the listener. The described technology may generate virtual auditory display filters for any number of distances from the center point. Accordingly, the described technology may accurately render sounds at varying distances.

One advantage of the described technology is that the described technology accurately renders virtual auditory display sound in virtual auditory space, meaning that sound is perceived by a listener as coming from the location that the creator of the sound intended for the sound. Another advantage of the described technology is that the described technology may be utilized with any ear-worn device, such as headphones, headset, and earbuds. Another advantage is that the virtual auditory display sound is high-quality and clear. Another advantage is the described technology may emphasize or deemphasize sounds in certain regions or locations of virtual auditory space so as to focus a listener's attention on those certain regions or locations. Such an approach may increase the listener's hearing abilities and allow the listener to hear sounds that the listener would not otherwise hear.

Another advantage of the described technology is that any digital device with suitable storage and processing power may store and apply the virtual auditory display filters. As described herein, a general purpose computing device such as a laptop or desktop computer may store and apply the virtual auditory display filters to audio signals to generate processed audio signals. The laptop or desktop computer may then send the processed audio signals to ear-worn devices to generate sound based on the processed audio signals. Similarly, a digital device such as a phone, tablet, or a virtual reality headset may store and apply the virtual auditory display filters to audio signals to generate processed audio signals and send the processed audio signals to ear-worn devices.

Additionally or alternatively, ear-worn devices, such as the virtual auditory display device100described herein, may store and apply the virtual auditory display filters. The ear-worn devices may receive an input audio signal from, for example, a digital device with which the ear-worn devices are paired such as a phone or tablet, or from a cloud-based service. The ear-worn devices may apply the stored virtual auditory display filters to the input audio signal to generate processed audio signals and output virtual auditory display sound based on the processed audio signals. Another example is that a cloud-based service may store and apply the virtual auditory display filters to generate processed audio signals and send the processed audio signals to ear-worn devices. Other advantages will be apparent.

FIG.16depicts a block diagram of an example digital device1600according to some embodiments. The digital device1600is shown in the form of a general-purpose computing device. The digital device1600includes at least one processor1602, RAM1604, communication interface1606, input/output device1608, storage1610, and a system bus1612that couples various system components including storage1610to the at least one processor1602. A system, such as a computing system, may be or include one or more of the digital device1600.

The digital device1600typically includes a variety of computer system readable media, such as computer system readable storage media. Such media may be any available media that is accessible by any of the systems described herein and it includes both volatile and nonvolatile media, removable and non-removable media.

In some embodiments, the at least one processor1602is configured to execute executable instructions (for example, programs). In some embodiments, the at least one processor1602comprises circuitry or any processor capable of processing the executable instructions.

In some embodiments, RAM1604stores programs and/or data. In various embodiments, working data is stored within RAM1604. The data within RAM1604may be cleared or ultimately transferred to storage1610, such as prior to reset and/or powering down the digital device1600.

In some embodiments, the digital device1600is coupled to a network via communication interface1606. The digital device1600can communicate with one or more networks such as a local area network (LAN), a general wide area network (WAN), and/or a public network (for example, the Internet).

In some embodiments, storage1610can include computer system readable media in the form of non-volatile memory, such as read only memory (ROM), programmable read only memory (PROM), solid-state drives (SSD), flash memory, and/or cache memory. Storage1610may further include other removable/non-removable, volatile/non-volatile computer system storage media. By way of example only, storage1610can be provided for reading from and writing to a non-removable, non-volatile magnetic media. The storage1610may include a non-transitory computer-readable medium, or multiple non-transitory computer-readable media, which stores programs or applications for performing functions such as those described herein with reference to, for example,FIGS.2A,2B, and3B. Although not shown, a magnetic disk drive for reading from and writing to a removable, non-volatile magnetic disk (for example, a “floppy disk”), and an optical disk drive for reading from or writing to a removable, non-volatile optical disk such as a CDROM, DVD-ROM or other optical media can be provided. In such instances, each can be connected to system bus1612by one or more data media interfaces. As will be further depicted and described below, storage1610may include at least one program product having a set (for example, at least one) of program modules that are configured to carry out the functions of embodiments of the invention. In some embodiments, RAM1604is found within storage1610.

Programs/utilities, having a set (at least one) of program modules, such as the virtual auditory display system102, may be stored in storage1610by way of example, and not limitation, as well as an operating system, one or more application programs, other program modules, and program data. Each of the operating system, one or more application programs, other program modules, and program data or some combination thereof, may include an implementation of a networking environment. Program modules generally carry out the functions and/or methodologies of embodiments of the invention as described herein.

It should be understood that although not shown, other hardware and/or software components could be used in conjunction with the digital device1600. Examples include, but are not limited to microcode, device drivers, redundant processing units, and external disk drive arrays, RAID systems, tape drives, and data archival storage systems, etc.

A transitory computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof.

Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object-oriented programming language such as Java, Smalltalk, C++, Python, or the like and conventional procedural programming languages, such as the “C” programming language or similar programming languages. The computer program code may execute entirely on any of the systems described herein or on any combination of the systems described herein.

While specific examples are described above for illustrative purposes, various equivalent modifications are possible. For example, while processes or blocks are presented in a given order, alternative implementations may perform routines having steps, or employ systems having blocks, in a different order, and some processes or blocks may be deleted, moved, added, subdivided, combined, and/or modified to provide alternative or sub-combinations. Each of these processes or blocks may be implemented in a variety of different ways. Also, while processes or blocks are at times shown as being performed in series, these processes or blocks may instead be performed or implemented concurrently or in parallel or may be performed at different times. Further any specific numbers noted herein are only examples: alternative implementations may employ differing values or ranges.

Throughout this specification, plural instances may implement components, operations, or structures described as a single instance. Structures and functionality presented as separate components in example configurations may be implemented as a combined structure or component. Similarly, structures and functionality presented as a single component may be implemented as separate components. These and other variations, modifications, additions, and improvements fall within the scope of the subject matter herein. Furthermore, any specific numbers noted herein are only examples: alternative implementations may employ differing values or ranges.

Components may be described or illustrated as contained within or connected with other components. Such descriptions or illustrations are examples only, and other configurations may achieve the same or similar functionality. Components may be described or illustrated as “coupled,” “couplable,” “operably coupled,” “communicably coupled” and the like to other components. Such description or illustration should be understood as indicating that such components may cooperate or interact with each other, and may be in direct or indirect physical, electrical, or communicative contact with each other.

Components may be described or illustrated as “configured to,” “adapted to,” “operative to,” “configurable to,” “adaptable to,” “operable to” and the like. Such description or illustration should be understood to encompass components both in an active state and in an inactive or standby state unless required otherwise by context.

The use of “or” in this disclosure is not intended to be understood as an exclusive “or.” Rather, “or” is to be understood as including “and/or.” For example, the phrase “providing products or services” is intended to be understood as having several meanings: “providing products,” “providing services,” and “providing products and services.”

It may be apparent that various modifications may be made, and other embodiments may be used without departing from the broader scope of the discussion herein. For example, the virtual auditory display system102may utilize a group of FIR filters for each of certain locations in virtual auditory space and a group of IIR filters for each of other certain locations in virtual auditory space. As another example, the virtual auditory display system102may provide audio signals to any device capable of directing sound towards the ears of a listener. As another example, a virtual auditory display device may be any device or set of devices (such as a pair of speakers) capable of producing sound based on output audio signals generated by the virtual auditory display system102.

Therefore, these and other variations upon the example embodiments are intended to be covered by the disclosure herein.