Patent Description:
Humans derive spatial cues and balance from environmental sounds that travel through the air, bounce off the pinna and concha of the exterior ear, and enter the ear canal. The environmental sounds vibrate the tympanic membrane, causing nerve signals to travel to the brain. However, headphones or in-ear-monitors that block the ear canal and transmit sounds to a listener's ear can result in a reduction or loss of directional cues in the transmitted sounds. The reduction in directional cues can reduce the listener's situational awareness.

Losing the ability to derive situational cues from ambient sounds can lead to the listener experiencing dissatisfaction with the headphones or in-ear-monitor and lead the listener to stop wearing the devices. <CIT> discloses a digital hearing aid system. <CIT> relates to speech enhancement for headsets with in-ear microphones. Other prior art listening devices are disclosed in <CIT> and <CIT>.

Embodiments relate to a listening device for adjusting and transmitting environmental sounds to a user on-the-fly as the user is participating in an artificial reality experience. In one embodiment, the user wears the listening device for listening to artificial audio content in an artificial reality environment. The listening device includes a reference microphone positioned outside a blocked ear canal of a user wearing the listening device to receive the environmental sounds and generate first signals based in part on the environmental sounds. A loudspeaker is coupled to the reference microphone and positioned inside the ear canal. The loudspeaker generates internal sounds based in part on the first signals. An internal microphone is positioned inside the ear canal to receive the internal sounds from the loudspeaker and generate second signals based in part on the internal sounds. A controller is coupled to the internal microphone and the reference microphone. The controller computes a transfer function based in part on the first signals and the second signals. The transfer function describes a variation between the environmental sounds and the internal sounds. The variation may be caused by the listening device blocking the ear canal and the internal sounds bouncing off the surfaces of the ear canal and the ear. This unwanted variation may add a bias to the reproduced environmental sounds as perceived by the user. The controller adjusts, based on the transfer function, the internal sounds to mitigate the variation.

Some embodiments describe a method for receiving environmental sounds by a reference microphone positioned outside a blocked ear canal of a user wearing a listening device. First signals are generated based in part on the environmental sounds. Internal sounds are generated, based in part on the first signals, by a loudspeaker coupled to the reference microphone and positioned inside the ear canal of the user. The internal sounds are received from the loudspeaker by an internal microphone positioned inside the ear canal of the user. Second signals are generated based in part on the internal sounds. A transfer function is computed based in part on the first signals and the second signals. The transfer function describes a variation between the environmental sounds and the internal sounds caused by the listening device blocking the ear canal of the user. Based in part on the transfer function, the internal sounds are adjusted to mitigate the variation.

Embodiments according to the invention are in particular disclosed in the attached claims directed to a listening device, a method, and a storage medium, wherein any feature mentioned in one claim category, e.g. listening device, can be claimed in another claim category, e.g. method, system, computer program product, as well.

In an embodiment, there is provided a listening device as recited in claim <NUM>.

The controller may be configured to compute the transfer function by executing steps to:.

The controller may be configured to adjust the internal sounds by performing steps to:.

The controller may comprise an adaptive filter configured to filter, based in part on an inverse of the transfer function, the first signals to mitigate effects of the transfer function from the internal sounds.

The listening device may be configured to adjust the environmental sounds relative to the received artificial audio content by increasing or decreasing a level of the environmental sounds relative to a level of the received artificial audio content.

In an embodiment, a listening device may comprise a second internal microphone positioned inside the ear canal of the user and configured to receive the internal sounds from the loudspeaker, wherein the listening device may be configured to determine a first acoustic pressure of the environmental sounds received by the reference microphone and a second acoustic pressure of the internal sounds received by the second internal microphone.

The controller may be configured to adjust the internal sounds to mitigate the variation between the first acoustic pressure and the second acoustic pressure by performing steps to:.

The controller may comprise an adaptive filter configured to filter, based in part on the variation between the first acoustic pressure and the second acoustic pressure, the first signals to mitigate the variation between the first acoustic pressure and the second acoustic pressure.

There is also provided a method according to claim <NUM>.

The computing of the transfer function may comprise:.

The adjusting of the internal sounds may comprise: generating correction signals based in part on an inverse of the transfer function; and adjusting the first signals, based in part on the correction signals, to mitigate effects of the transfer function from the internal sounds.

The adjusting of the internal sounds may comprise filtering, by an adaptive filter, based in part on an inverse of the transfer function, the first signals to mitigate effects of the transfer function from the internal sounds.

In an embodiment, a method may comprise:.

The adjusting of the internal sounds to mitigate the variation between the first acoustic pressure and the second acoustic pressure may comprise:.

The adjusting of the internal sounds to mitigate the variation between the first acoustic pressure and the second acoustic pressure may comprise filtering, by an adaptive filter, based in part on the variation between the first acoustic pressure and the second acoustic pressure, the first signals to mitigate the variation between the first acoustic pressure and the second acoustic pressure.

There is also provided a computer-readable medium according to claim <NUM>.

In a further embodiment of the invention, one or more computer-readable non-transitory storage media embody software that is operable when executed to perform in a listening device according to the invention or any of the above mentioned embodiments.

In a further embodiment of the invention, a computer-implemented method uses a listening device according to the invention or any of the above mentioned embodiments.

In a further embodiment of the invention, a computer program product, preferably comprising a computer-readable non-transitory storage media, is used in a listening device according to the invention or any of the above mentioned embodiments.

Embodiments of the invention may include or be implemented in conjunction with an artificial reality system. Artificial reality is a form of reality that has been adjusted in some manner before presentation to a user, which may include, e.g., a virtual reality (VR), an augmented reality (AR), a mixed reality (MR), a hybrid reality, or some combination and/or derivatives thereof. Artificial reality content may include completely generated content or generated content combined with captured (e.g., real-world) content. The artificial reality content may include video, audio, haptic feedback, or some combination thereof, and any of which may be presented in a single channel or in multiple channels (such as stereo video that produces a three-dimensional effect to the viewer). Additionally, in some embodiments, artificial reality may also be associated with applications, products, accessories, services, or some combination thereof, that are used to, e.g., create content in an artificial reality and/or are otherwise used in (e.g., perform activities in) an artificial reality. The artificial reality system that provides the artificial reality content may be implemented on various platforms, including an HMD connected to a host computer system, a standalone HMD, a mobile device or computing system, or any other hardware platform capable of providing artificial reality content to one or more viewers.

An artificial reality system may present artificial audio content to a user using a listening device such that the user experiences an artificial reality environment. The listening device may partially or fully block the ear or ear canal of the user to present a more realistic sound environment or simply because of the manner in which the listening device is designed. The embodiments described herein adjust and transmit environmental sounds received by the listening device on-the-fly to the user while artificial audio content is being presented to the user. In an embodiment, the listening device may transmit only the environmental sounds to the user or adjust the environmental sounds relative to the received artificial audio content. The listening device may mix the environmental sounds with the received artificial audio content. The listening device may increase or decrease a level of the environmental sounds relative to a level of the received artificial audio content. The listening device may also block the environmental sounds and transmit only the received artificial audio content to the user.

<FIG> is an example view of a listening device <NUM> within a user's ear <NUM> for mitigating a variation between environmental sounds <NUM> and internal sounds caused by the listening device <NUM> blocking an ear canal <NUM> of the user, in accordance with one or more embodiments. The listening device <NUM> is positioned within the user's ear <NUM> for transmitting hybrid audio content including adjusted environmental sounds and artificial reality audio content to the user, in accordance with an embodiment. The listening device <NUM> may be worn by itself on the user's ear <NUM>, or as part of a set of headphones or head-mounted display (HMD) worn on the user's head. Such an HMD may also reflect projected images and allow the user to see through it, display computer-generated imagery (CGI), live imagery from the physical world, or may allow CGI to be superimposed on a real-world view (referred to as augmented reality or mixed reality).

<FIG> shows the ear <NUM> of the user. The ear <NUM> includes a pinna <NUM>, the ear canal <NUM>, and an eardrum <NUM>. The pinna <NUM> is the part of the user's ear <NUM> made of cartilage and soft tissue so that it keeps a particular shape but is also flexible. The ear canal <NUM> is a passage comprised of bone and skin leading to the eardrum <NUM>. The ear canal <NUM> functions as an entryway for sound waves, which get propelled toward the eardrum <NUM>. The eardrum <NUM>, also called the tympanic membrane, is a thin membrane that separates the external ear from the middle ear (not shown in <FIG>). The function of the eardrum <NUM> is to transmit sounds (e.g., the environmental sounds <NUM>) from the air to the cochlea by converting and amplifying vibrations in air to vibrations in fluid.

The listening device <NUM> of <FIG> adjusts the environmental sounds <NUM>, and transmits the adjusted environmental sounds and received artificial audio content to the user. The listening device <NUM> is intended to be placed or inserted into the ear <NUM> in a manner to block the ear canal <NUM>. For example, the listening device <NUM> may block the ear canal <NUM> to isolate received artificial audio content provided by an artificial reality system coupled to the listening device <NUM> using a wired connection or a wireless connection. The listening device <NUM> includes a reference microphone <NUM>, a loudspeaker <NUM>, one or more internal microphones <NUM> and/or <NUM>, and a controller <NUM>. In other embodiments, the listening device <NUM> may include additional or fewer components than those described herein. Similarly, the functions can be distributed among the components and/or different entities in a different manner than is described here.

The reference microphone <NUM> receives the environmental sounds <NUM> and generates first signals (e.g., electrical signals or some other transducer signals) based in part on the environmental sounds <NUM>. The reference microphone <NUM> is positioned outside the blocked ear canal <NUM> of the user wearing the listening device <NUM>. The reference microphone <NUM> may include a transducer that converts air pressure variations of the environmental sounds <NUM> to the first signals. For example, the reference microphone <NUM> may include a coil of wire suspended in a magnetic field, a vibrating diaphragm, a crystal of piezoelectric material, some other transducer, or a combination thereof. The first signals generated by the reference microphone <NUM> are processed by the listening device <NUM> to transmit the internal sounds into the ear canal <NUM> and towards the eardrum <NUM>.

The loudspeaker <NUM> receives the first signals (e.g., electrical signals) from the reference microphone and generates the internal sounds based in part on the first signals. The loudspeaker <NUM> also transmits artificial audio content received by the listening device <NUM> to the user. The loudspeaker <NUM> may be coupled to the reference microphone <NUM> using a wired connection or a wireless connection. The loudspeaker <NUM> is positioned inside the ear canal <NUM> of the user. The loudspeaker <NUM> may include an electroacoustic transducer to generate the internal sounds based in part on the first signals and the received artificial audio content. For example, the loudspeaker <NUM> may include a voice coil, a piezoelectric speaker, a magnetostatic speaker, some other mechanism to convert the first signals and the received artificial audio content to the internal sounds, or a combination thereof. The internal sounds generated by the loudspeaker <NUM> are transmitted to the eardrum <NUM>.

The internal microphone <NUM> acts as a monitor by receiving the internal sounds from the loudspeaker and generating second signals (e.g., electrical signals or some other transducer signals) based in part on the internal sounds. The second signals are used by the listening device <NUM> to monitor and correct for variations between the environmental sounds <NUM> received by the reference microphone <NUM> at the entrance of the user's ear <NUM> and the internal sounds generated by the loudspeaker <NUM>. The internal microphone <NUM> is also positioned inside the ear canal <NUM> of the user. The internal microphone <NUM> may include a transducer to convert the internal sounds to the second signals by any of the several methods described above with respect to the reference microphone <NUM>.

The internal microphone <NUM> may be sensitive to changes in position within the ear canal <NUM>, e.g., when the user tilts or moves her head or moves the listening device <NUM>. To correct for this sensitivity to changes in position of the internal microphone <NUM>, the optional second internal microphone <NUM> may be used to determine an acoustic pressure of the internal sounds received by the second internal microphone <NUM> and correct for variations between the acoustic pressure of the internal sounds and an acoustic pressure of the environmental sounds <NUM> received by the reference microphone <NUM>.

The controller <NUM> uses a combination of acoustic measurement and model fitting to correct for variations between the environmental sounds <NUM> received at the entrance of the user's ear <NUM> and the internal sounds generated by the loudspeaker near the eardrum <NUM>. The controller <NUM> may be an analog or digital circuit, a microprocessor, an application-specific integrated circuit, some other implementation, or a combination thereof. The controller <NUM> may be implemented in hardware, software, firmware, or a combination thereof. The controller <NUM> is coupled to the internal microphone <NUM> and the reference microphone <NUM>. The controller <NUM> may be coupled to the reference microphone <NUM>, the loudspeaker <NUM>, and the internal microphones <NUM> and/or <NUM> using wired and/or wireless connections. In embodiment, the controller <NUM> may be located external to the ear canal <NUM>. For example, the controller <NUM> may be located behind the pinna <NUM>, on an HMD, on a mobile device, on an artificial reality console, etc..

The mechanical shape and/or the electrical and acoustic transmission properties of the listening device <NUM>, and the sounds bouncing off the user's ear canal <NUM> may add a bias to the environmental sounds <NUM> when they are reproduced by the loudspeaker <NUM> as internal sounds and received by the internal microphone <NUM>. This bias may be represented as a transfer function between the internal sounds and the environmental sounds <NUM>. The transfer function results from the shape and sound reflection properties of the components of the listening device <NUM> and the ear <NUM> (including ear canal <NUM>). The transfer function is personal to each user based on her personal ear characteristics. The transfer function alters the environmental sounds <NUM> so that the user hears a distorted version of the environmental sounds <NUM>. In other words, the listening device <NUM> converts the received environmental sounds <NUM> to the internal sounds based in part on the transfer function. The transfer function may be represented in the form of a mathematical function H(s) relating the output or response (e.g., the internal sounds) to the input or stimulus (e.g., the environmental sounds <NUM>).

In one embodiment, the transfer function H(s) describes a variation between the environmental sounds <NUM> and the internal sounds. The variation is caused by the listening device blocking <NUM> the ear canal <NUM> of the user. The variation may be based in part on the mechanical shape and electrical and acoustic transmission properties of the listening device <NUM>, and the shape and sound reflection properties of the ear <NUM> (including ear canal <NUM>). The internal sounds that reach the user may therefore mask the situational cues present in the environmental sounds <NUM>, or provide incorrect or inadequate spatial cues and situational awareness to the user when she is wearing the listening device <NUM>.

The controller <NUM> corrects for the bias in the internal sounds by computing the transfer function H(s) based in part on the first signals and the second signals. The controller <NUM> uses the computed transfer function H(s) to pre-process the first signals (e.g., by using an inverse of the computed transfer function) to mitigate effects of the transfer function H(s) from the internal sounds. In an embodiment, the controller <NUM> may use the second internal microphone <NUM> to perform acoustic outlier measurement with particle blocking at the entrance to the eardrum <NUM> to replicate the acoustic pressure field observed at the reference microphone <NUM> to account for sub-mm differences in placement of the internal microphone <NUM>. In this embodiment, the controller <NUM> may adjust the internal sounds to mitigate variations between the acoustic pressure of the environmental sounds <NUM> received by the reference microphone <NUM> and the acoustic pressure of the internal sounds.

The benefits and advantages of the embodiments disclosed are that the listening device <NUM> may be positioned in the blocked ear canal <NUM> to encode the environmental sounds <NUM> and determine a personalized audio fingerprint of the user for localization, such that the user retains auditory situational awareness. The loudspeaker <NUM> and the internal microphones <NUM> and <NUM> are deeply seated in the ear canal <NUM> to reproduce the internal sounds captured at the ear canal <NUM> and remove the transfer function effect of the listening device <NUM> by calibration of the internal sounds individually to each user.

<FIG> is an example architectural block diagram of a listening device <NUM> using a controller <NUM> for mitigating a variation between environmental sounds (e.g., <NUM>) and internal sounds <NUM> caused by the listening device <NUM> blocking an ear canal (e.g., <NUM>) of the user, in accordance with one or more embodiments. The listening device <NUM> may be an embodiment of the listening device <NUM> shown in <FIG> and the controller <NUM> may be an embodiment of the controller <NUM> shown in <FIG>. The listening device <NUM> includes a reference microphone (e.g., <NUM>), the controller <NUM>, a loudspeaker (e.g., <NUM>), one or more internal microphones <NUM>, and a summer <NUM>. The internal microphones <NUM> may be an embodiment of the one or more internal microphones <NUM> and/or <NUM>. In other embodiments, the listening device <NUM> comprises additional or fewer components than those described herein. Similarly, the functions can be distributed among the components and/or different entities in a different manner than is described here.

The reference microphone receives the environmental sounds <NUM> at the entrance to the user's ear (e.g., <NUM>) and generates first signals <NUM> (e.g., electrical signals or some other transducer signals) based in part on the environmental sounds <NUM>. The reference microphone <NUM> is positioned outside the blocked ear canal <NUM> of the user wearing the listening device <NUM>. The first signals <NUM> may be electrical signals (e.g., voltage, current, digital signals, or a combination thereof) generated by the reference microphone <NUM> by any of the methods described above with reference to <FIG>.

The loudspeaker <NUM> generates the internal sounds <NUM> based in part on the first signals <NUM> (as adjusted by the controller <NUM>) to transmit the internal sounds <NUM> to the eardrum <NUM>. The loudspeaker <NUM> is positioned inside the ear canal <NUM> of the user. The loudspeaker <NUM> may be coupled to the reference microphone <NUM> and the controller <NUM> using a wired connection or a wireless connection.

The internal microphones <NUM> are used to determine and correct for variations between the environmental sounds <NUM> and the internal sounds <NUM> captured by the internal microphones <NUM>. The internal sounds <NUM> are transmitted along the ear canal <NUM> to the eardrum <NUM> for sound perception. The internal microphones <NUM> are also positioned inside the ear canal <NUM> of the user and may be coupled to the controller <NUM> using a wired or wireless connection. At least one of the internal microphones <NUM> receives the internal sounds <NUM> from the loudspeaker <NUM> and generates second signals <NUM> based in part on the internal sounds <NUM>.

A second one of the internal microphones <NUM> is used to perform acoustic power correction. The acoustic power of the environmental sounds <NUM> may be determined as acoustic power = acoustic pressure × particle velocity. The acoustic power of the internal sounds <NUM> may be similarly determined. The acoustic power is invariant to small changes in position of the internal microphone <NUM> while the acoustic pressure may vary with the physical position of the internal microphone <NUM> and the characteristics of the ear canal <NUM>. When only a single internal microphone <NUM> is used to compute the transfer function between the internal sounds <NUM> and the environmental sounds <NUM>, the transfer function computed may be sensitive to small changes in the physical position of the internal microphone <NUM> relative to the ear canal <NUM>. The transfer function is therefore individualized per user and may act like an acoustic fingerprint. The second one of the internal microphones <NUM> is therefore used to correct the internal sounds <NUM> to reproduce the same acoustic pressure at the eardrum <NUM> that is observed at the reference microphone <NUM> when the user is in a particular environment.

The controller <NUM> is used to monitor the first signals <NUM> and the second signals <NUM>, and correct for variations between the environmental sounds <NUM> and the internal sounds <NUM>. The controller <NUM> may include an optional adaptive filter <NUM> to filter the first signals <NUM> to correct for the variations between the environmental sounds <NUM> and the internal sounds <NUM>. The controller may be coupled to the reference microphone <NUM>, the loudspeaker <NUM>, and the internal microphones <NUM> using wired connections and/or wireless connections.

The controller <NUM> receives and may sample the first signals <NUM> and the second signals <NUM>. For example, the controller <NUM> may analyze the behavior of the first signals <NUM> and the second signals <NUM> with respect to how they vary with respect to time. The controller <NUM> computes a transfer function (e.g., H(s)) based in part on the first signals <NUM> and the second signals <NUM>. The transfer function H(s) describes a variation between the environmental sounds <NUM> and the internal sounds <NUM>. The controller <NUM> may compute the transfer function H(s) using a domain transform based on the second signals <NUM> and the first signals <NUM>. For example, if the continuous-time input signal x(t) represents the first signals <NUM> and the continuous-time output y(t) represents the second signals <NUM>, the controller <NUM> may map the Laplace transform of the second signals Y(s)=L{y(t)} to the Laplace transform of the first signals X(s)=L{x(t)}. The transfer function may therefore computed as H(s) = Y(s)/X(s). In other embodiments, other domain transforms such as Fourier transforms, Fast Fourier transforms, Z transforms, some other domain transform, or a combination thereof may be used.

The controller <NUM> adjusts the first signals <NUM> based on the transfer function H(s) to generate adjusted first signals <NUM> to mitigate the variation between the environmental sounds <NUM> and the internal sounds <NUM>. In one embodiment, the controller <NUM> adjusts the first signals <NUM> by generating correction signals <NUM>. The correction signals <NUM> may be electrical signals (e.g., voltage, current, digital signals, or a combination thereof). The correction signals <NUM> may be based in part on an inverse I(s) of the transfer function H(s). The controller <NUM> may transmit the correction signals <NUM> to the summer <NUM> to adjust the first signals <NUM> to mitigate effects of the transfer function H(s) from the internal sounds.

The summer <NUM> adjusts the first signals <NUM> to generate the adjusted first signals <NUM>. The adjusted first signals <NUM> may be a voltage, current, digital signal, or a combination thereof. The summer may subtract the correction signals <NUM> from the first signals <NUM> to generate the adjusted first signals <NUM>. For example, if C(s) represents the correction signals <NUM>, the adjusted first signals <NUM> may be represented as X(s) - C(s). The correction signals <NUM> may instruct the summer to adjust certain frequencies, amplitudes, some other characteristics, or a combination thereof, of the first signals <NUM>. The correction signals <NUM> are used to adjust the first signals <NUM> (and the internal sounds <NUM>) such that the user perceives the internal sounds <NUM> as being closer to the original environmental sounds <NUM>.

In alternative embodiments, the controller <NUM> may adjust the internal sounds <NUM> by transmitting correction signals (e.g., corresponding to an inverse I(s) of the transfer function H(s)) to the loudspeaker <NUM> to mitigate effects of the transfer function H(s) from the internal sounds <NUM>. These correction signals may be electrical signals (e.g., voltage, current, digital signals, or a combination thereof) to instruct the loudspeaker <NUM> to adjust certain frequencies, amplitudes, some other characteristics, or a combination thereof, of the internal sounds <NUM> to more closely match the environmental sounds <NUM>.

In an embodiment, the controller <NUM> may perform acoustic power correction of the internal sounds <NUM> by adjusting the internal sounds <NUM> such that the acoustic pressure of the environmental sounds <NUM> observed at the reference microphone <NUM> is reproduced at the eardrum <NUM>. In this embodiment, the controller <NUM> may determine a first acoustic pressure of the environmental sounds <NUM> observed by the reference microphone <NUM> (e.g., based on the first signals <NUM>). The controller <NUM> may determine a second acoustic pressure of the internal sounds <NUM> observed by the internal microphones <NUM> (e.g., based on the second signals <NUM>). The controller <NUM> may adjust the internal sounds <NUM> (using the adjusted first signals <NUM>) to mitigate a variation between the first acoustic pressure and the second acoustic pressure. For example, the first signals <NUM> may be adjusted such that acoustic pressures corresponding to different frequency components of the internal sounds <NUM> are increased or decreased, acoustic pressures corresponding to amplitudes of the internal sounds <NUM> at different times are increased or decreased, etc. In this manner, unwanted bias effects of the transfer function H(s) may be mitigated from the internal sounds <NUM> while matching the second acoustic pressure of the internal sounds <NUM> to the first acoustic pressure of the environmental sounds <NUM> more closely.

In one embodiment, the optional adaptive filter <NUM> may adaptively filter the first signals <NUM> to correct for the effects of the transfer function H(s). The adaptive filter <NUM> may be implemented in software, hardware, firmware, or a combination thereof. As shown in <FIG>, the adaptive filter <NUM> may reside within the controller <NUM>. In an embodiment (not illustrated in <FIG>), the adaptive filter <NUM> may lie outside the controller <NUM>.

The adaptive filter <NUM> may filter, using an inverse I(s) of the transfer function H(s), the first signals <NUM> to mitigate effects of the transfer function H(s) from the internal sounds <NUM>. The adaptive filter <NUM> may adaptively filter the first signals <NUM> to mitigate the variation between the first signals <NUM> and the second signals <NUM>. The adaptive filter <NUM> may be a linear filter having an internal transfer function controlled by variable parameters and a means to adjust those parameters according to an optimization algorithm. The benefits and advantages of using the adaptive filter <NUM> are that certain parameters (e.g., x(t) and y(t), or the position and orientation of the listening device <NUM>) may not be known in advance or may be changing. Thus the adaptive filter <NUM> may use feedback in the form of an internal error signal to adaptively refine its filter function.

In one embodiment, the controller <NUM> may adjust the received environmental sounds <NUM> (first signals <NUM>) relative to artificial audio content <NUM> received from an artificial reality system coupled to the listening device <NUM>, a virtual reality audio device, a smartphone, some other device, or a combination thereof. The artificial audio content <NUM> may be test sounds intended to calibrate the listening device <NUM>, immersive VR cinematic sound, channel-based surround sound, some other audio content, or a combination thereof. The controller <NUM> may combine the adjusted environmental sounds <NUM> (the adjusted first signals <NUM>) with the received artificial audio content <NUM> to generate the internal sounds <NUM>. For example, the controller <NUM> may combine the adjusted environmental sounds <NUM> with the artificial audio content <NUM> to construct and present an audio portion of an immersive artificial reality experience so that what the user hears matches what the user is seeing and interacting with. In embodiment, immersive 3D audio techniques, including binaural recordings and object-based audio, may thus be applied using the listening device <NUM>.

The benefits and advantages, among others, of the embodiments disclosed herein are that the listening device <NUM> is able to transmit corrected environmental sounds including inherent spatial cues as well as music and speech content during normal usage of the listening device <NUM> in an artificial reality environment. The ongoing correction by the adaptive filter <NUM> may be used to adjust the internal sounds <NUM> as the user walks around a room or moves her jaw, etc. Disruptions to the external portion of the user's ear (e.g., <NUM>) are reduced and normal spatial cues that users use to infer and interpret the external sound field are transmitted to the user. The user can keep the listening device <NUM> in her ear <NUM> for long periods of time because the normal listening function is not disrupted.

<FIG> is an example architectural block diagram of a controller <NUM> for mitigating a variation between environmental sounds (e.g., <NUM>) and internal sounds (e.g., <NUM>) caused by a listening device (e.g., <NUM>) blocking an ear canal of the user, in accordance with one or more embodiments. The controller <NUM> may be an embodiment of the controller <NUM> shown in <FIG> or the controller <NUM> shown in <FIG>. The controller <NUM> includes a transfer function computation module <NUM>, an acoustic pressure computation module <NUM>, a correction signals generator <NUM>, an optional adaptive filter (e.g., <NUM>), and an audio content mixer <NUM>. In other embodiments, the controller <NUM> may include additional or fewer components than those described herein. Similarly, the functions can be distributed among the components and/or different entities in a different manner than is described here.

The transfer function computation module <NUM> computes a transfer function (e.g., H(s)) based in part on first signals (e.g., <NUM>) and second signals (e.g., <NUM>). The first signals <NUM> may be generated by a reference microphone (e.g., <NUM>) positioned outside a blocked ear canal (e.g., <NUM>) of a user wearing the listening device <NUM> based in part on the environmental sounds <NUM>. The second signals <NUM> may be generated by an internal microphone (e.g., <NUM>) positioned inside the ear canal <NUM> of the user and configured to receive the internal sounds <NUM> from a loudspeaker (e.g., <NUM>) and generate the second signals <NUM>.

The transfer function H(s) describes the variation between the environmental sounds <NUM> and the internal sounds <NUM> caused by the listening device <NUM> blocking the ear canal <NUM> of the user. In one embodiment, the transfer function computation module <NUM> computes the transfer function H(s) by performing perform spectral estimation on the first signals <NUM> and the second signals <NUM> to generate a frequency distribution. For example, the transfer function computation module <NUM> may perform spectrum analysis, also referred to as frequency domain analysis or spectral density estimation, to decompose the first signals <NUM> and the second signals <NUM> into individual frequency components X(s) and Y(s). The transfer function computation module <NUM> may further quantify the various amounts present in the signals <NUM> and <NUM> (e.g., amplitudes, powers, intensities, or phases) versus frequency. The transfer function computation module <NUM> may perform spectral estimation on the entirety of the first signals <NUM> and the second signals <NUM> or the signals <NUM> and <NUM> may be broken into samples, and spectral estimation may be applied to the individual samples.

The transfer function computation module <NUM> may compute the transfer function H(s) based in part on the frequency distribution obtained from the spectral estimation. For example, the transfer function computation module <NUM> may use linear operations on X(s) and Y(s) in the frequency domain to compute the transfer function H(s) as H(s)=Y(s)/X(s).

The acoustic pressure computation module <NUM> determines the first acoustic pressure of the environmental sounds <NUM> observed by the reference microphone <NUM> (e.g., based on the first signals <NUM>). The first acoustic pressure (or sound pressure) of the environmental sounds <NUM> received by the reference microphone <NUM> is the local pressure deviation from the ambient atmospheric pressure caused by the environmental sounds <NUM>. The first acoustic pressure may be recorded and analyzed by the acoustic pressure computation module <NUM> to determine information about the nature of the path the environmental sounds <NUM> took from the source to the reference microphone <NUM>. The first acoustic pressure depends on the environment, reflecting surfaces, the distance of the reference microphone <NUM>, ambient sounds, etc..

In an embodiment, the acoustic pressure computation module <NUM> may determine the first acoustic pressure p1 of the environmental sounds <NUM> (based in part on the first signals <NUM>) as the local pressure deviation from the ambient pressure caused by sound waves of the environmental sounds <NUM>. The first acoustic pressure p1 may be measured in units of pascals. The acoustic pressure computation module <NUM> may determine a first particle velocity v1 of the environmental sounds <NUM> that is the velocity of a particle in a medium as it transmits the environmental sounds <NUM>. The first particle velocity v1 may be expressed in units of meter per second. The acoustic pressure computation module <NUM> may determine a first acoustic intensity I1 of the environmental sounds <NUM> as I1 = p1 × v1. The first acoustic intensity I1 is the power carried by sound waves of the environmental sounds <NUM> per unit area in a direction perpendicular to that area. The first acoustic intensity I1 may be expressed in watt per square meter.

The acoustic pressure computation module <NUM> may also determine the second acoustic pressure p2 of the internal sounds <NUM> observed by the internal microphones <NUM> (e.g., based on the second signals <NUM>). The user's auditory system analyses the second acoustic pressure for sound localization and spatial cues using directional and loudness evaluation. However, variations in the second acoustic pressure from the first acoustic pressure can lead to unstable directional cues because there may be a mix of sounds reflected by the listening device <NUM> and the ear canal <NUM>. Therefore, the controller <NUM> uses the acoustic pressure computation module <NUM> to adjust the internal sounds <NUM> such that the acoustic pressure of the internal sounds <NUM> reaching the eardrum <NUM> is closer to the acoustic pressure of the environmental sounds <NUM> received by the reference microphone <NUM>.

In an embodiment, the acoustic pressure computation module <NUM> may determine a second particle velocity v2 of the internal sounds <NUM> and a second acoustic intensity I2 of the internal sounds <NUM> as I2 = p2 × v2. The acoustic pressure computation module <NUM> may determine variations between p2 and p1 caused by positional changes of the internal microphone <NUM>. However, the second acoustic intensity I2 of the internal sounds <NUM> is invariant from the first acoustic intensity I1 of the environmental sounds <NUM>. Therefore, the internal sounds <NUM> may be adjusted to correct for the variations between p2 and p1.

The correction signals generator <NUM> generates correction signals (e.g., <NUM>) to adjust the first signals <NUM> to mitigate effects of the transfer function H(s) from the internal sounds <NUM>. In one embodiment, the correction signals generator <NUM> generates the correction signals <NUM> based in part on an inverse I(s) of the transfer function H(s). The correction signals <NUM> therefore enable the reference microphone <NUM> and listening device <NUM> to adjust its performance to meet the desired output response (environmental sounds <NUM>). In one embodiment, the correction signals generator <NUM> generates the correction signals <NUM> to adjust the internal sounds <NUM> to mitigate a variation between the first acoustic pressure p1 and the second acoustic pressure p2.

The correction signals <NUM> may be negative feedback correction signals that correspond to a variation between a domain transform of the first signals X(s) and a domain transform of the second signals Y(s). When the correction signals (e.g., E(s)) are transmitted to a summer (e.g., <NUM>), a negative feedback loop is created that adjusts the internal sounds (Y(s)) to be closer to the environmental sounds (X(s)). The following equations may be used to determine the corrected internal sounds: C(s) = X(s) - Y(s) and Yc(s) = H(s) × Xc(s), where Yc(s) refers to the adjusted internal sounds and Xc(s) refers to the adjusted first signals <NUM>. Similar determinations may be made to adjust the signals to account for variations in acoustic pressure.

The optional adaptive filter <NUM> filters the first signals <NUM> to mitigate effects of the transfer function H(s) from the internal sounds <NUM>. The adaptive filter <NUM> changes its filter parameters (coefficients) over time to adapt to changing signal characteristics of the first signals <NUM> and the second signals <NUM> by self-learning. As the first signals <NUM> are received by the adaptive filter <NUM>, the adaptive filter <NUM> adjusts its coefficients to achieve the desired result (i.e., adjusting the first signals <NUM> and the internal sounds <NUM> to be closer to the environmental sounds <NUM>).

To define the adaptive filtering process, an adaptive algorithm may be selected to mitigate the error between the signal y(t) (internal sounds <NUM>) and a desired signal d(t) (adjusted internal sounds). For example, the adaptive filter <NUM> may use an adaptive algorithm such as least mean squares (LMS), recursive least squares (RLS), lattice filtering, filtering that operates in the frequency domain, or a combination thereof. In one embodiment, when the LMS performance criterion for an internal error signal between the first signals <NUM> and the second signals <NUM> has achieved its minimum value through the iterations of the adaptive algorithm, the adaptive filter <NUM>'s coefficients may converge to a solution. The output from the adaptive filter may now be closer to the desired signal d(t). When the input data characteristics of the environmental sounds <NUM> change, the adaptive filter <NUM> adapts by generating a new set of coefficients for the new signal characteristics.

In one embodiment, the adaptive filter <NUM> filters, using an inverse I(s) of the transfer function H(s), the first signals <NUM> to mitigate a variation between the first acoustic pressure p1 and the second acoustic pressure p2. By placing the adaptive filter <NUM> in series with the forward path of the listening device <NUM> as shown in <FIG>, the adaptive filter <NUM> adapts to the inverse I(s) of the transfer function H(s) to mitigate the variation between the first acoustic pressure p1 and the second acoustic pressure p2.

The audio content mixer <NUM> may combine the received environmental sounds <NUM> with received artificial audio content (e.g., <NUM>) to generate the internal sounds <NUM>. The audio content mixer <NUM> may mix ambient sounds with sounds corresponding to an artificial reality display. In one embodiment, the listening device <NUM> may have a sliding control for blocking part of the environmental sounds <NUM> or part of the artificial audio content <NUM> to varying degrees, e.g., <NUM>% ambient sound, <NUM>% ambient sound + <NUM>% artificial audio content, etc. The audio content mixer <NUM> may receive information in the form of a signal from the sliding control to control the environmental sounds <NUM>, the received artificial audio content <NUM>, or both.

The audio content mixer <NUM> may adjust the environmental sounds <NUM> relative to the artificial audio content <NUM>. The audio content mixer <NUM> may adjust the environmental sounds <NUM> by increasing or decreasing a level of the environmental sounds <NUM> relative to a level of the artificial audio content <NUM> to generate the internal sounds <NUM>. For example, the volume level, frequency content, dynamics, and panoramic position of the environmental sounds <NUM> may be manipulated and or enhanced. The levels of speech (dialogue, voice-overs, etc.), ambient noise, sound effects, and music in the artificial audio content <NUM> may be increased or decreased relative to the environmental sounds <NUM>.

The audio content mixer <NUM> may combine the adjusted environmental sounds <NUM> with the artificial audio content <NUM> into one or more channels. For example, the adjusted environmental sounds <NUM> and the artificial audio content <NUM> may be electrically blended together to include sounds from instruments, voices, and pre-recorded material. Either the environmental sounds <NUM> or the artificial audio content <NUM> or both may be equalized and/or amplified and reproduced via the loudspeaker <NUM>.

<FIG> is an example process for mitigating a variation between environmental sounds (e.g., <NUM>) and internal sounds (e.g., <NUM>) caused by a listening device (e.g., <NUM>) blocking an ear canal (e.g., <NUM>) of a user, in accordance with one or more embodiments. In one embodiment, the process of <FIG> is performed by a listening device (e.g., <NUM>). Other entities (e.g., an HMD) may perform some or all of the steps of the process in other embodiments. Likewise, embodiments may include different and/or additional steps, or perform the steps in different orders.

The listening device <NUM> receives <NUM> the environmental sounds <NUM> using a reference microphone (e.g., <NUM>). The reference microphone <NUM> is positioned outside a blocked ear canal of a user wearing the listening device <NUM>.

The listening device <NUM> generates <NUM> first signals (e.g., <NUM>) based in part on the environmental sounds <NUM>. The first signals <NUM> may be electrical signals (e.g., voltage, current, digital signals, or a combination thereof. ) The reference microphone <NUM> may include a transducer that converts air pressure variations of the environmental sounds <NUM> to the first signals <NUM>. For example, the reference microphone <NUM> may include a coil of wire suspended in a magnetic field, a vibrating diaphragm, a crystal of piezoelectric material, some other transducer, or a combination thereof.

The listening device <NUM> generates <NUM> the internal sounds <NUM> based in part on the first signals <NUM> by a loudspeaker (e.g., <NUM>) that is coupled to the reference microphone <NUM>. The loudspeaker <NUM> may include an electroacoustic transducer to convert the first signals <NUM> to the internal sounds <NUM>. The loudspeaker <NUM> may include a voice coil, a piezoelectric speaker, a magnetostatic speaker, some other mechanism to convert the first signals <NUM> to the internal sounds <NUM>, or a combination thereof.

The listening device <NUM> receives <NUM> the internal sounds <NUM> using an internal microphone (e.g., <NUM>). The internal microphone <NUM> is also positioned inside the ear canal <NUM> of the user.

The listening device <NUM> generates <NUM> second signals (e.g., <NUM>) corresponding to the internal sounds <NUM>. The second signals <NUM> may be electrical signals (e.g., voltage, current, digital signals, or a combination thereof. ) The internal microphone <NUM> may generate the second signals <NUM> in a manner described above with respect to the reference microphone <NUM>.

The listening device <NUM> computes <NUM> a transfer function (e.g., H(s)) based in part on the first signals <NUM> and the second signals <NUM>. The transfer function H(s) describes a variation between the environmental sounds <NUM> and the internal sounds <NUM>. For example, the variation may be caused by the listening device <NUM> blocking the ear canal <NUM> of the user. The listening device <NUM> may perform spectral estimation on the first signals <NUM> and the second signals <NUM> to generate a frequency distribution. The listening device <NUM> may compute the transfer function H(s) from the frequency distribution.

The listening device <NUM> adjusts <NUM>, based on the transfer function H(s), the internal sounds <NUM> to mitigate the variation. The listening device <NUM> may adjust the internal sounds <NUM> by using a controller (e.g., <NUM>) to generate correction signals (e.g., <NUM>) based on an inverse I(s) of the transfer function H(s). The controller <NUM> may use the correction signals <NUM> to adjust the first signals <NUM> to mitigate effects of the transfer function H(s) from the internal sounds <NUM>. In one embodiment, an adaptive filter (e.g., <NUM>) may filter, using an inverse I(s) of the transfer function H(s), the first signals <NUM> to mitigate effects of the transfer function H(s) from the internal sounds <NUM>.

The listening device (e.g., <NUM>) may be part of an HMD coupled to an artificial reality system, including base stations to provide audio content, and a console. In embodiments, a part of the functionality of the controller (e.g., <NUM>) may be performed by a console to which the listening device <NUM> is coupled. One or more base stations may further include a depth camera assembly to determine depth information describing a position of the listening device <NUM> or HMD in the local area relative to the locations of the base stations.

The HMD may further include an inertial measurement unit (IMU) including one or more position sensors to generate signals in response to motion of the HMD. Examples of position sensors include: accelerometers, gyroscopes, magnetometers, another suitable type of sensor that detects motion, a type of sensor used for error correction of the IMU, or some combination thereof. The audio content (e.g., <NUM>) and environmental sounds (e.g., <NUM>) may be further adjusted based on the signals corresponding to motion of the user.

The artificial reality system may provide video content to the user via the HMD, where the audio content (e.g., <NUM>) corresponds to the video content, and the video content corresponds to the position of the listening device <NUM> or HMD to provide an immersive artificial reality experience.

The foregoing description of the embodiments of the disclosure has been presented for the purpose of illustration; it is not intended to be exhaustive or to limit the disclosure to the precise forms disclosed. Persons skilled in the relevant art can appreciate that many modifications and variations are possible in light of the above disclosure.

Some portions of this description describe the embodiments of the disclosure in terms of algorithms and symbolic representations of operations on information. These algorithmic descriptions and representations are commonly used by those skilled in the data processing arts to convey the substance of their work effectively to others skilled in the art. These operations, while described functionally, computationally, or logically, are understood to be implemented by computer programs or equivalent electrical circuits, microcode, or the like. Furthermore, it has also proven convenient at times, to refer to these arrangements of operations as modules, without loss of generality. The described operations and their associated modules may be embodied in software, firmware, hardware, or any combinations thereof.

Embodiments of the disclosure may also relate to an apparatus for performing the operations herein. This apparatus may be specially constructed for the required purposes, and/or it may comprise a general-purpose computing device selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a non-transitory, tangible computer readable storage medium, or any type of media suitable for storing electronic instructions, which may be coupled to a computer system bus. Furthermore, any computing systems referred to in the specification may include a single processor or may be architectures employing multiple processor designs for increased computing capability.

Embodiments of the disclosure may also relate to a product that is produced by a computing process described herein. Such a product may comprise information resulting from a computing process, where the information is stored on a non-transitory, tangible computer readable storage medium and may include any embodiment of a computer program product or other data combination described herein.

Claim 1:
A listening device (<NUM>) to be coupled to an artificial reality system, the listening device comprising:
a reference microphone (<NUM>) positioned outside a blocked ear canal (<NUM>) of a user wearing the listening device and configured to receive environmental sounds (<NUM>) and generate first signals based in part on the environmental sounds;
a loudspeaker (<NUM>) coupled to the reference microphone and positioned inside the ear canal of the user, the loudspeaker configured to generate internal sounds based in part on the first signals;
an internal microphone (<NUM>) positioned inside the ear canal of the user and configured to receive the internal sounds from the loudspeaker and generate second signals based in part on the internal sounds; and
a controller (<NUM>) coupled to the internal microphone, to the loudspeaker and to the reference microphone and configured to:
compute a transfer function based in part on the first signals and the second signals, the transfer function describing a variation between the environmental sounds and the internal sounds caused by the listening device blocking the ear canal of the user,
adjust the first signals based on the transfer function,
receive artificial audio content from the artificial reality system,
combine the adjusted first signals with the received artificial audio content to obtain hybrid audio content, and
transmit the hybrid audio content to the loudspeaker.