A system reduces sound coloration caused by rendering of a 3D audio signal. The system renders the 3D audio signal including a plurality of channels using the input audio signal. Input spectra data defining spectral information of the input audio signal is computed. 3D spectra data defining spectral information of a single channel representation of the 3D audio signal is computed. The system generates a tonal balance filter based on the input spectral data and the 3D spectral data. The tonal balance filter, when applied to the 3D audio signal, reduces sound coloration caused by the rendering of the 3D audio signal. The tonal balance filter is applied to the 3D audio signal to generate an output audio signal and the output audio signal is presented via a speaker array.

FIELD OF THE INVENTION

This disclosure relates generally to 3-dimensional (3D) audio rendering, and more specifically to reducing sound coloration for 3D audio rendering.

BACKGROUND

3D audio systems typically suffer from sound coloration, which may be defined as a negative perceptual audio experience elicited by spectral distortion introduced by the 3D audio system that are audible but cannot be associated to 3D sound cues. It is desirable to reduce sound coloration while preserving the spectral information of the original audio content.

SUMMARY

Embodiments relate to reducing sound coloration caused rendering of a 3D audio signal from an input audio signal. Some embodiments include a method performed by one or more processors. The method includes rendering a 3D audio signal including a plurality of channels using an input audio signal. Input spectra data defining spectral information of the input audio signal is computed. 3D spectra data defining spectral information of a single channel representation of the 3D audio signal is computed. The method further includes generating a tonal balance filter based on the input spectral data and the 3D spectral data. The tonal balance filter, when applied to the 3D audio signal, reduces sound coloration caused by the rendering of the 3D audio signal. The method further includes applying the tonal balance filter the 3D audio signal to generate an output audio signal and presenting, via a speaker array, the output audio signal.

Some embodiments include a device. The device includes a speaker array, one or more processors, and a memory. The memory stores program code, when executed by the one or more processors, configures the one or more processors for: rendering a 3D audio signal including a plurality of channels using an input audio signal; computing input spectral data defining spectral information of the input audio signal; computing a 3D spectral data defining spectral information of a single channel representation of the 3D audio signal; generating a tonal balance filter based on the input spectral data and the 3D spectral data, the tonal balance filter configured to, when applied to the 3D audio signal, reduce sound coloration caused by the rendering of the 3D audio signal; applying the tonal balance filter to the 3D audio signal to generate an output audio signal; and presenting, via the speaker array, the output audio signal.

Some embodiments include a non-transitory computer-readable medium including stored program code that, when executed by one or more processors of an audio system, configures the audio system to: render a 3D audio signal including a plurality of channels using an input audio signal; compute input spectral data defining spectral information of the input audio signal; compute a 3D spectral data defining spectral information of a single channel representation of the 3D audio signal; generate a tonal balance filter based on the input spectral data and the 3D spectral data, the tonal balance filter configured to, when applied to the 3D audio signal, reduce sound coloration caused by the rendering of the 3D audio signal; apply the tonal balance filter to the 3D audio signal to generate an output audio signal; and present, via a speaker array, the output audio signal.

DETAILED DESCRIPTION

Embodiments relate to reducing sound coloration caused by rendering of a 3D audio signal from an input audio signal. Generic and pseudo-personalized 3D-audio systems can suffer from sound coloration. Sound coloration refers to a negative perceptual audio experience elicited by spectral distortion introduced by 3D audio rendering that are audible but cannot be associated to 3D-sound cues. Fixed equalization filters have been used to reduce sound coloration but are sub-optimal. These fixed equalization filters include free-field equalizers and diffuse-field equalizers. Free field equalization filters compensate for the effect of a single direction, typically the front direction. Diffuse field equalization filters compensate for the effect of all possible directions. Since the sound coloration introduced by 3D audio rendering often originates from the simultaneous contribution of multiple directions, free-field equalization (single direction) and diffuse-field equalization (all possible directions) may be considered as the extremes of a continuum in the equalization space, which is not always optimal.

Embodiments relate to monitoring the sound coloration introduced by a 3D audio rendering of an input audio signal and correcting the sound coloration with regards to the spectral information that exists at the input audio signal. A 3D audio signal including multiple channels is rendered using an input audio signal. The input audio signal may include one or more channels. To reduce sound coloration caused by the rendering of the 3D audio signal, the difference between the spectral data of the 3D audio signal and the spectral data of the input audio signal are used to generate a tonal balance filter that is applied to the 3D audio signal. For example, input spectral data defining spectral data of a single channel representation of the input audio signal is computed. 3D spectral data defining spectral data of a single channel representation of the 3D audio signal is computed. The tonal balance filter is generated based on the input spectral data and the 3D spectral data, and the tonal balance filter is applied to each channel of the 3D audio signal to generate an output audio signal. The generation of the tonal balance filter may be performed in the time-domain or the frequency-domain. In some embodiments, a sound coloration model can be used to modify the tonal balance filter. The sound coloration model is used to inform the equalization procedure on the sufficient amount of correction required to reduce over-equalization or under-equalization based on perceptual considerations.

The sound coloration reduction discussed herein removes sound coloration while preserving the spectral information of the original audio material. It provides equalization (and in some cases optimal equalization) for the coloration introduced by the 3D-audio renderer at any given time, allowing for equalizations that lie somewhere in between free-field (or any single-direction based equalization) and diffuse field.

FIG.1Ais a perspective view of a headset100implemented as an eyewear device, in accordance with one or more embodiments. In some embodiments, the eyewear device is a near eye display (NED). In general, the headset100may be worn on the face of a user such that content (e.g., media content) is presented using a display assembly and/or an audio system. However, the headset100may also be used such that media content is presented to a user in a different manner. Examples of media content presented by the headset100include one or more images, video, audio, or some combination thereof. The headset100includes a frame, and may include, among other components, a display assembly including one or more display elements120, a depth camera assembly (DCA), an audio system, and a position sensor190. WhileFIG.1Aillustrates the components of the headset100in example locations on the headset100, the components may be located elsewhere on the headset100, on a peripheral device paired with the headset100, or some combination thereof. Similarly, there may be more or fewer components on the headset100than what is shown inFIG.1A.

The frame110holds the other components of the headset100. The frame110includes a front part that holds the one or more display elements120and end pieces (e.g., temples) to attach to a head of the user. The front part of the frame110bridges the top of a nose of the user. The length of the end pieces may be adjustable (e.g., adjustable temple length) to fit different users. The end pieces may also include a portion that curls behind the ear of the user (e.g., temple tip, ear piece).

The one or more display elements120provide light to a user wearing the headset100. As illustrated the headset includes a display element120for each eye of a user. In some embodiments, a display element120generates image light that is provided to an eyebox of the headset100. The eyebox is a location in space that an eye of user occupies while wearing the headset100. For example, a display element120may be a waveguide display. A waveguide display includes a light source (e.g., a two-dimensional source, one or more line sources, one or more point sources, etc.) and one or more waveguides. Light from the light source is in-coupled into the one or more waveguides which outputs the light in a manner such that there is pupil replication in an eyebox of the headset100. In-coupling and/or outcoupling of light from the one or more waveguides may be done using one or more diffraction gratings. In some embodiments, the waveguide display includes a scanning element (e.g., waveguide, mirror, etc.) that scans light from the light source as it is in-coupled into the one or more waveguides. Note that in some embodiments, one or both of the display elements120are opaque and do not transmit light from a local area around the headset100. The local area is the area surrounding the headset100. For example, the local area may be a room that a user wearing the headset100is inside, or the user wearing the headset100may be outside and the local area is an outside area. In this context, the headset100generates VR content. Alternatively, in some embodiments, one or both of the display elements120are at least partially transparent, such that light from the local area may be combined with light from the one or more display elements to produce AR and/or MR content.

In some embodiments, a display element120does not generate image light, and instead is a lens that transmits light from the local area to the eyebox. For example, one or both of the display elements120may be a lens without correction (non-prescription) or a prescription lens (e.g., single vision, bifocal and trifocal, or progressive) to help correct for defects in a user's eyesight. In some embodiments, the display element120may be polarized and/or tinted to protect the user's eyes from the sun.

In some embodiments, the display element120may include an additional optics block (not shown). The optics block may include one or more optical elements (e.g., lens, Fresnel lens, etc.) that direct light from the display element120to the eyebox. The optics block may, e.g., correct for aberrations in some or all of the image content, magnify some or all of the image, or some combination thereof.

The DCA determines depth information for a portion of a local area surrounding the headset100. The DCA includes one or more imaging devices130and a DCA controller (not shown inFIG.1A), and may also include an illuminator140. In some embodiments, the illuminator140illuminates a portion of the local area with light. The light may be, e.g., structured light (e.g., dot pattern, bars, etc.) in the infrared (IR), IR flash for time-of-flight, etc. In some embodiments, the one or more imaging devices130capture images of the portion of the local area that include the light from the illuminator140. As illustrated,FIG.1Ashows a single illuminator140and two imaging devices130. In alternate embodiments, there is no illuminator140and at least two imaging devices130.

The DCA controller computes depth information for the portion of the local area using the captured images and one or more depth determination techniques. The depth determination technique may be, e.g., direct time-of-flight (ToF) depth sensing, indirect ToF depth sensing, structured light, passive stereo analysis, active stereo analysis (uses texture added to the scene by light from the illuminator140), some other technique to determine depth of a scene, or some combination thereof.

The audio system provides audio content. The audio system includes a transducer array, a sensor array, and an audio controller150. However, in other embodiments, the audio system may include different and/or additional components. Similarly, in some cases, functionality described with reference to the components of the audio system can be distributed among the components in a different manner than is described here. For example, some or all of the functions of the controller may be performed by a remote server.

The transducer array presents sound to user. The transducer array includes a plurality of transducers. A transducer may be a speaker160or a tissue transducer170(e.g., a bone conduction transducer or a cartilage conduction transducer). Although the speakers160are shown exterior to the frame110, the speakers160may be enclosed in the frame110. In some embodiments, instead of individual speakers for each ear, the headset100includes a speaker array comprising multiple speakers integrated into the frame110to improve directionality of presented audio content. The tissue transducer170couples to the head of the user and directly vibrates tissue (e.g., bone or cartilage) of the user to generate sound. The number and/or locations of transducers may be different from what is shown inFIG.1A.

The sensor array detects sounds within the local area of the headset100. The sensor array includes a plurality of acoustic sensors180. An acoustic sensor180captures sounds emitted from one or more sound sources in the local area (e.g., a room). Each acoustic sensor is configured to detect sound and convert the detected sound into an electronic format (analog or digital). The acoustic sensors180may be acoustic wave sensors, microphones, sound transducers, or similar sensors that are suitable for detecting sounds.

In some embodiments, one or more acoustic sensors180may be placed in an ear canal of each ear (e.g., acting as binaural microphones). In some embodiments, the acoustic sensors180may be placed on an exterior surface of the headset100, placed on an interior surface of the headset100, separate from the headset100(e.g., part of some other device), or some combination thereof. The number and/or locations of acoustic sensors180may be different from what is shown inFIG.1A. For example, the number of acoustic detection locations may be increased to increase the amount of audio information collected and the sensitivity and/or accuracy of the information. The acoustic detection locations may be oriented such that the microphone is able to detect sounds in a wide range of directions surrounding the user wearing the headset100.

The audio controller150provides for 3D audio rendering for an input audio signal. The audio controller150may comprise one or more processors and a computer-readable storage medium. The computer-readable storage medium includes instructions that, when executed by the one or more processors, configures the one or more processors to perform the functionality discussed herein by the audio controller150. In some embodiments, the audio controller150may include an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or some other type of processing circuitry.

The audio controller150receives the input audio signal and renders a 3D audio signal using the input audio signal. The input audio signal may be a single channel or multiple channels. The 3D audio signal includes multiple channels and provides 3D-sound cues when rendered by the transducer array. The audio controller150uses a tonal balance filter to reduce sound coloration caused by the 3D audio rendering. The 3D audio rendering and tonal balance filtering may be performed in the time or frequency domains. The audio controller150may use a time-varying algorithm for always-on compensation of the sound coloration.

For example, the audio controller150computes input spectral data defining spectral information of the input audio signal. The audio controller150also computes 3D spectral data defining spectral information of the 3D audio signal. The audio controller150generates the tonal balance filter based on the input spectral data and the 3D spectral data. For processing in the time-domain, the input spectral data and 3D spectral data may each be represented by a spectral curve. The tonal balance filter may be generated using a convolution of the spectral curve of the input audio signal and an inverse of the spectral curve of the 3D audio signal. the tonal balance filter, when applied to the 3D audio signal, reduces sound coloration caused by the rendering of the 3D audio signal For processing in the frequency-domain, the input spectral data and 3D spectral data may each be represented by frequency magnitude vectors. The tonal balance filter may be generated using a ratio between the frequency magnitude vectors of the input audio signal and the frequency magnitude vectors of the 3D audio signal.

In some embodiments, the audio controller150uses a sound coloration model to inform or modify the tonal balance filter. The sound coloration model may also be used to generate the input spectral data and 3D spectral data.

The audio controller150applies the tonal balance filter to the 3D audio signal to generate an output audio signal and provides the output audio signal to a speaker array. The application of the tonal balance filter to the 3D audio signal results in the output audio signal having reduced sound coloration relative to the 3D audio signal. Additional details regarding sound coloration reduction for 3D audio signals are discussed below in connection withFIGS.2through5.

The audio controller150processes information from the sensor array that describes sounds detected by the sensor array. The audio controller150may be configured to generate direction of arrival (DOA) estimates, generate acoustic transfer functions (e.g., array transfer functions and/or head-related transfer functions), track the location of sound sources, form beams in the direction of sound sources, classify sound sources, generate sound filters for the speakers160, or some combination thereof.

The position sensor190generates one or more measurement signals in response to motion of the headset100. The position sensor190may be located on a portion of the frame110of the headset100. The position sensor190may include an inertial measurement unit (IMU). Examples of position sensor190include: one or more accelerometers, one or more gyroscopes, one or more magnetometers, another suitable type of sensor that detects motion, a type of sensor used for error correction of the IMU, or some combination thereof. The position sensor190may be located external to the IMU, internal to the IMU, or some combination thereof.

In some embodiments, the headset100may provide for simultaneous localization and mapping (SLAM) for a position of the headset100and updating of a model of the local area. For example, the headset100may include a passive camera assembly (PCA) that generates color image data. The PCA may include one or more RGB cameras that capture images of some or all of the local area. In some embodiments, some or all of the imaging devices130of the DCA may also function as the PCA. The images captured by the PCA and the depth information determined by the DCA may be used to determine parameters of the local area, generate a model of the local area, update a model of the local area, or some combination thereof. Furthermore, the position sensor190tracks the position (e.g., location and pose) of the headset100within the room. Additional details regarding the components of the headset100are discussed below in connection withFIG.6.

FIG.1Bis a perspective view of a headset105implemented as an HMD, in accordance with one or more embodiments. In embodiments that describe an AR system and/or a MR system, portions of a front side of the HMD are at least partially transparent in the visible band (˜380 nm to 750 nm), and portions of the HMD that are between the front side of the HMD and an eye of the user are at least partially transparent (e.g., a partially transparent electronic display). The HMD includes a front rigid body115and a band175. The headset105includes many of the same components described above with reference toFIG.1A, but modified to integrate with the HMD form factor. For example, the HMD includes a display assembly, a DCA, an audio system, and a position sensor190.FIG.1Bshows the illuminator140, a plurality of the speakers160, a plurality of the imaging devices130, a plurality of acoustic sensors180, and the position sensor190. The speakers160may be located in various locations, such as coupled to the band175(as shown), coupled to front rigid body115, or may be configured to be inserted within the ear canal of a user.

FIG.2is a block diagram of an audio system200, in accordance with one or more embodiments. The audio system inFIG.1AorFIG.1Bmay be an embodiment of the audio system200. The audio system200generates one or more acoustic transfer functions for a user. The audio system200may then use the one or more acoustic transfer functions to generate audio content for the user. In the embodiment ofFIG.2, the audio system200includes a transducer array210, a sensor array220, and an audio controller230. Some embodiments of the audio system200have different components than those described here. Similarly, in some cases, functions can be distributed among the components in a different manner than is described here.

The transducer array210is configured to present audio content. The transducer array210includes a plurality of transducers. A transducer is a device that provides audio content. A transducer may be, e.g., a speaker (e.g., the speaker160), a tissue transducer (e.g., the tissue transducer170), some other device that provides audio content, or some combination thereof. A tissue transducer may be configured to function as a bone conduction transducer or a cartilage conduction transducer. The transducer array210may present audio content via air conduction (e.g., via one or more speakers), via bone conduction (via one or more bone conduction transducer), via cartilage conduction audio system (via one or more cartilage conduction transducers), or some combination thereof. In some embodiments, the transducer array210may include one or more transducers to cover different parts of a frequency range. For example, a piezoelectric transducer may be used to cover a first part of a frequency range and a moving coil transducer may be used to cover a second part of a frequency range.

The bone conduction transducers generate acoustic pressure waves by vibrating bone/tissue in the user's head. A bone conduction transducer may be coupled to a portion of a headset, and may be configured to be behind the auricle coupled to a portion of the user's skull. The bone conduction transducer receives vibration instructions from the audio controller230, and vibrates a portion of the user's skull based on the received instructions. The vibrations from the bone conduction transducer generate a tissue-borne acoustic pressure wave that propagates toward the user's cochlea, bypassing the eardrum.

The cartilage conduction transducers generate acoustic pressure waves by vibrating one or more portions of the auricular cartilage of the ears of the user. A cartilage conduction transducer may be coupled to a portion of a headset, and may be configured to be coupled to one or more portions of the auricular cartilage of the ear. For example, the cartilage conduction transducer may couple to the back of an auricle of the ear of the user. The cartilage conduction transducer may be located anywhere along the auricular cartilage around the outer ear (e.g., the pinna, the tragus, some other portion of the auricular cartilage, or some combination thereof). Vibrating the one or more portions of auricular cartilage may generate: airborne acoustic pressure waves outside the ear canal; tissue born acoustic pressure waves that cause some portions of the ear canal to vibrate thereby generating an airborne acoustic pressure wave within the ear canal; or some combination thereof. The generated airborne acoustic pressure waves propagate down the ear canal toward the ear drum.

The transducer array210(also referred to as a speaker array) generates audio content in accordance with instructions from the audio controller230. In some embodiments, the audio content is spatialized. Spatialized audio content is audio content that appears to originate from a particular direction and/or target region (e.g., an object in the local area and/or a virtual object). For example, spatialized audio content can make it appear that sound is originating from a virtual singer across a room from a user of the audio system200. The transducer array210may be coupled to a wearable device (e.g., the headset100or the headset105). In alternate embodiments, the transducer array210may be a plurality of speakers that are separate from the wearable device (e.g., coupled to an external console).

The sensor array220detects sounds within a local area surrounding the sensor array220. The sensor array220may include a plurality of acoustic sensors that each detect air pressure variations of a sound wave and convert the detected sounds into an electronic format (analog or digital). The plurality of acoustic sensors may be positioned on a headset (e.g., headset100and/or the headset105), on a user (e.g., in an ear canal of the user), on a neckband, or some combination thereof. An acoustic sensor may be, e.g., a microphone, a vibration sensor, an accelerometer, or any combination thereof. In some embodiments, the sensor array220is configured to monitor the audio content generated by the transducer array210using at least some of the plurality of acoustic sensors. Increasing the number of sensors may improve the accuracy of information (e.g., directionality) describing a sound field produced by the transducer array210and/or sound from the local area.

The audio controller230controls operation of the audio system200. In the embodiment ofFIG.2, the audio controller230includes a data store235, a DOA estimation module240, a transfer function module250, a tracking module260, a beamforming module270, and a sound filter module280. The audio controller230may be located inside a headset, in some embodiments. Some embodiments of the audio controller230have different components than those described here. Similarly, functions can be distributed among the components in different manners than described here. For example, some functions of the controller may be performed external to the headset. The user may opt in to allow the audio controller230to transmit data captured by the headset to systems external to the headset, and the user may select privacy settings controlling access to any such data.

The data store235stores data for use by the audio system200. Data in the data store235may include sounds recorded in the local area of the audio system200, audio content such as input audio signals and 3D audio signals, sound coloration models, tonal balance filters, input spectral data, 3D spectral data, output audio data generated using tonal balance filters, head-related transfer functions (HRTFs), transfer functions for one or more sensors, array transfer functions (ATFs) for one or more of the acoustic sensors, sound source locations, virtual model of local area, direction of arrival estimates, sound filters, and other data relevant for use by the audio system200, or any combination thereof.

The DOA estimation module240is configured to localize sound sources in the local area based in part on information from the sensor array220. Localization is a process of determining where sound sources are located relative to the user of the audio system200. The DOA estimation module240performs a DOA analysis to localize one or more sound sources within the local area. The DOA analysis may include analyzing the intensity, spectra, and/or arrival time of each sound at the sensor array220to determine the direction from which the sounds originated. In some cases, the DOA analysis may include any suitable algorithm for analyzing a surrounding acoustic environment in which the audio system200is located.

For example, the DOA analysis may be designed to receive input signals from the sensor array220and apply digital signal processing algorithms to the input signals to estimate a direction of arrival. These algorithms may include, for example, delay and sum algorithms where the input signal is sampled, and the resulting weighted and delayed versions of the sampled signal are averaged together to determine a DOA. A least mean squared (LMS) algorithm may also be implemented to create an adaptive filter. This adaptive filter may then be used to identify differences in signal intensity, for example, or differences in time of arrival. These differences may then be used to estimate the DOA. In another embodiment, the DOA may be determined by converting the input signals into the frequency domain and selecting specific bins within the time-frequency (TF) domain to process. Each selected TF bin may be processed to determine whether that bin includes a portion of the audio spectrum with a direct path audio signal. Those bins having a portion of the direct-path signal may then be analyzed to identify the angle at which the sensor array220received the direct-path audio signal. The determined angle may then be used to identify the DOA for the received input signal. Other algorithms not listed above may also be used alone or in combination with the above algorithms to determine DOA.

In some embodiments, the DOA estimation module240may also determine the DOA with respect to an absolute position of the audio system200within the local area. The position of the sensor array220may be received from an external system (e.g., some other component of a headset, an artificial reality console, a mapping server, a position sensor (e.g., the position sensor190), etc.). The external system may create a virtual model of the local area, in which the local area and the position of the audio system200are mapped. The received position information may include a location and/or an orientation of some or all of the audio system200(e.g., of the sensor array220). The DOA estimation module240may update the estimated DOA based on the received position information.

The transfer function module250is configured to generate one or more acoustic transfer functions. Generally, a transfer function is a mathematical function giving a corresponding output value for each possible input value. Based on parameters of the detected sounds, the transfer function module250generates one or more acoustic transfer functions associated with the audio system. The acoustic transfer functions may be array transfer functions (ATFs), head-related transfer functions (HRTFs), other types of acoustic transfer functions, or some combination thereof. An ATF characterizes how the microphone receives a sound from a point in space.

An ATF includes a number of transfer functions that characterize a relationship between the sound source and the corresponding sound received by the acoustic sensors in the sensor array220. Accordingly, for a sound source there is a corresponding transfer function for each of the acoustic sensors in the sensor array220. And collectively the set of transfer functions is referred to as an ATF. Accordingly, for each sound source there is a corresponding ATF. Note that the sound source may be, e.g., someone or something generating sound in the local area, the user, or one or more transducers of the transducer array210. The ATF for a particular sound source location relative to the sensor array220may differ from user to user due to a person's anatomy (e.g., ear shape, shoulders, etc.) that affects the sound as it travels to the person's ears. Accordingly, the ATFs of the sensor array220are personalized for each user of the audio system200.

In some embodiments, the transfer function module250determines one or more HRTFs for a user of the audio system200. The HRTF characterizes how an ear receives a sound from a point in space. The HRTF for a particular source location relative to a person is unique to each ear of the person (and is unique to the person) due to the person's anatomy (e.g., ear shape, shoulders, etc.) that affects the sound as it travels to the person's ears. In some embodiments, the transfer function module250may determine HRTFs for the user using a calibration process. In some embodiments, the transfer function module250may provide information about the user to a remote system. The user may adjust privacy settings to allow or prevent the transfer function module250from providing the information about the user to any remote systems. The remote system determines a set of HRTFs that are customized to the user using, e.g., machine learning, and provides the customized set of HRTFs to the audio system200.

The tracking module260is configured to track locations of one or more sound sources. The tracking module260may compare current DOA estimates and compare them with a stored history of previous DOA estimates. In some embodiments, the audio system200may recalculate DOA estimates on a periodic schedule, such as once per second, or once per millisecond. The tracking module may compare the current DOA estimates with previous DOA estimates, and in response to a change in a DOA estimate for a sound source, the tracking module260may determine that the sound source moved. In some embodiments, the tracking module260may detect a change in location based on visual information received from the headset or some other external source. The tracking module260may track the movement of one or more sound sources over time. The tracking module260may store values for a number of sound sources and a location of each sound source at each point in time. In response to a change in a value of the number or locations of the sound sources, the tracking module260may determine that a sound source moved. The tracking module260may calculate an estimate of the localization variance. The localization variance may be used as a confidence level for each determination of a change in movement.

The beamforming module270is configured to process one or more ATFs to selectively emphasize sounds from sound sources within a certain area while de-emphasizing sounds from other areas. In analyzing sounds detected by the sensor array220, the beamforming module270may combine information from different acoustic sensors to emphasize sound associated from a particular region of the local area while deemphasizing sound that is from outside of the region. The beamforming module270may isolate an audio signal associated with sound from a particular sound source from other sound sources in the local area based on, e.g., different DOA estimates from the DOA estimation module240and the tracking module260. The beamforming module270may thus selectively analyze discrete sound sources in the local area. In some embodiments, the beamforming module270may enhance a signal from a sound source. For example, the beamforming module270may apply sound filters which eliminate signals above, below, or between certain frequencies. Signal enhancement acts to enhance sounds associated with a given identified sound source relative to other sounds detected by the sensor array220.

The sound filter module280provides for 3D audio rendering for input audio signals. The 3D audio signals include multiple channels of audio that produce 3D-sound cues when rendered by the transducer array. The sound filter module280uses a tonal balance filter to reduce sound coloration caused by the 3D audio rendering. Additional details regarding the sound filter module280are discussed below in connection withFIG.3.

The sound filter module280may also determine sound filters for the transducer array210. The sound filters cause the audio content to be spatialized, such that the audio content appears to originate from a target region. The sound filter module280may use HRTFs and/or acoustic parameters to generate the sound filters. The acoustic parameters describe acoustic properties of the local area. The acoustic parameters may include, e.g., a reverberation time, a reverberation level, a room impulse response, etc. In some embodiments, the sound filter module280calculates one or more of the acoustic parameters. In some embodiments, the sound filter module280requests the acoustic parameters from a mapping server (e.g., as described below with regard toFIG.6).

The sound filter module280provides audio content processed using the sound filters to the transducer array210. In some embodiments, the sound filters may cause positive or negative amplification of sounds as a function of frequency.

FIG.3is a block diagram describing some operations of a sound filter module280, in accordance with one or more embodiments. The sound filter module280includes a 3D audio rendering module310, an input spectral data module320, a 3D spectral data module330, and a tonal balance filter module350. Some embodiments of the sound filter module280may have different components than those described here. Similarly, in some cases, functions can be distributed among the components in a different manner than is described here.

The 3D spectral data module330computes 3D spectral data366defining spectral information of the 3D audio signal362. The 3D spectral data366may be computed using a single channel representation of the multiple channels of the 3D audio signal362. For example, the 3D spectral data module330may compute a sum across the channels of the 3D audio signal362. The 3D spectral data366is determined using the single channel representation in either the time-domain or frequency-domain, as discussed in greater detail below.

The tonal balance filter module350generates a tonal balance filter based on the input spectral data364and the 3D spectral data366. The tonal balance filter module350uses a difference between the spectral content represented by the input spectral data364and the 3D spectral data366, which may be compared against the sound coloration model340, to generate the tonal balance filter such that the tonal balance filter efficiently compensates for this difference.

The tonal balance filter module350applies the tonal balance filter to the 3D audio signal362to generate the output audio signal368. For example, the tonal balance filter may be applied to each channel of the 3D audio signal362to equalize these channels and increase tonal balance. By applying the same tonal balance filter to all channels, no differences are introduced across channels thus faithfully preserving inter-channel 3D-audio information.

To ensure that silences or low-energy frequency regions in the 3D audio signal362generated by the 3D audio rendering module310do not create large amplifications, typically greater than 15 or 20 dB, in the output audio signal368, the frequency-dependent gain of the tonal balance filter may be limited. The limitation may be imposed by choosing a constant threshold across frequency or by choosing a frequency-dependent threshold, more amplification control may be required at low-frequencies than at high frequencies due to power constraints.

The processing by the input spectral data module320, 3D spectral data module330and tonal balance filter module350to generate the tonal balance filter may be performed in the time-domain or the frequency-domain. For time-domain processing, the input spectral data364generated by the input spectral data module320and the 3D spectral data366generated by the 3D spectral data module330are each represented by a spectral envelope. The spectral envelope of the input spectra data364is also referred to as an input spectral envelope and the spectral envelope of the 3D spectral data366is also referred to as a 3D spectral envelope. The spectral envelopes may be generated in various ways, such as using linear predictive coding or autoregressive modeling. For autoregressive modeling, an all-pole infinite impulse response (IIR) filter representation of a selected order may be used. To control the amount of detail encoded in the spectral envelopes, the input spectral data module320and 3D spectral data module330may use the sound coloration model340to determine the order of the autoregressive model. The tonal balance filter module350generates the tonal balance filter based on a convolution between the input spectral envelope and an inverse of the 3D spectral envelope. The inverse of the 3D spectral envelope represents the inverse of the autoregressive model used by the 3D spectral data module330. Because the autoregressive model is an all-pole model, its inverse is an all-zero model (or finite impulse response (FIR) filter), thus providing for stability of the tonal balance filter. Once the tonal balance filter is computed in350, the filter is convolved with the 3D audio signal362. This process may be implemented as convolution in time domain or complex multiplication in frequency domain. If the filtering is applied in the frequency domain as described below in more detail below, an inverse Fourier Transform is used to generate the 3D audio rendering time-domain signal. If the output of the 3D audio rendering module310is multichannel the same tonal balance filter may be equally applied to all channels.

For frequency-domain processing, the input spectral data364generated by the input spectral data module320and the 3D spectral data366generated by the 3D spectral data module330are each represented by frequency magnitude vectors. The frequency magnitude vectors of the input spectra data364are also referred to as input frequency magnitude vectors and the frequency magnitude vectors of the 3D spectra data366are also referred to as 3D frequency magnitude vectors. Each of the input frequency magnitude vectors and 3D frequency magnitude vectors may be computed using a subband processing based on, for example, an analysis filter bank implementation. The level of detail in the frequency magnitude vectors may be controlled by the frequency resolution of the filter bank. The tonal balance filter module350generates the tonal balance filter based on a ratio between the input frequency magnitude vectors (e.g., in the numerator) and 3D frequency magnitude vectors (e.g., in the denominator). To generate the output audio signal, the tonal balance filter module350multiplies the tonal balance filter with each of the channels of the 3D audio signal. The tonal balance filter module350may also convert the output audio signal to the time domain, such as by using a synthesis filter bank to transform the subband output audio signal to a time-domain representation of the output audio signal.

In some embodiments, the tonal balance filter module350uses the sound coloration model340to inform or modify the tonal balance filter. The sound coloration model340may be used with either the time-domain or frequency-domain processing. The sound coloration model340provides for perceptually-motivated spectral tuning of the shape of the tonal balance filter. The sound coloration model340may be generated based on perceptual metrics such as spectral profile analysis.

In some embodiments, the sound filter module280applies a time-varying algorithm for always-on compensation of the sound coloration. The tonal balance filter may be updated over time as defined by an update rate between new and old filter coefficients. Each update of the tonal balance filter may be calculated by analyzing the spectral data within a particular time period. For example, within a time period t1, the sound filter module280receives the input audio signal360for the time period t1and stores the input audio signal360for the time period t1within a buffer. The input audio signal360for time period t1is used to render a 3D audio signal362for the time period t1, which may also be stored in the buffer. The input spectral data364for time period t1is calculated using the input audio signal360for time period t1and the 3D spectral data366for time period t1is calculated using the 3D audio signal362for time period t1. The tonal balance filter module350generates the tonal balance filter using the input spectral data364for time period t1and the 3D spectral data366for time period t1. The tonal balance filter may be applied to the 3D audio signal362for at least a portion of the time period t1, such as the time remaining in time period t1after creation of the tonal balance filter. While the tonal balance filter is being applied to the 3D audio signal362for time period t1, an updated tonal balance filter is determined using the audio signals for a time period t2. The updated tonal balance filter is applied to the 3D audio signal362for at least a portion of the time period t2, and so forth. This process may be repeated for multiple time periods to provide the time-varying, always-on compensation of the sound coloration. The length of the time periods may be adjustable and may depend on an adjustable update rate for the tonal balance filter.

FIG.4is a graph400of sound coloration models, in accordance with one or more embodiments. The graph400shows amplitudes (dB) as a function of frequency (Hz). Tolerance curve402is a frequency-dependent tolerance curve from a coloration model based on average coloration sensitivity. Tolerance curve404is a frequency-dependent tolerance curve from a coloration model based on high coloration sensitivity. The tolerance curves402and404are determined based on audibility thresholds at different frequencies. A family of curves may be constructed in between the tolerance curves402and404to fit different sound coloration control requirements.

The sound coloration model340provides frequency-dependent subjective thresholds for the perception of coloration based on profile analysis, i.e. the ability of human subjects to discriminate between spectral shapes. The average-sensitivity tolerance curve402and high-sensitivity tolerance curve404may be provided by this model. The difference between the input spectral data364and 3D spectral data module366may be compared against the selected tolerance curve in the sound coloration model340. For differences that fall below the tolerance curve, no coloration correction may be required, and the tonal balance filter may be built based on differences above the tolerance curve.

FIG.5is a flowchart illustrating an example of a process500for reducing sound coloration in a 3D audio signal, in accordance with one or more embodiments. The process shown inFIG.5may be performed by components of an audio system (e.g., audio system200). Other entities may perform some or all of the steps inFIG.5in other embodiments. Embodiments may include different and/or additional steps, or perform the steps in different orders.

The audio system renders510a 3D audio signal using an input audio signal. The 3D audio signal includes multiple channels. Rendering the 3D audio signal may include using ambisonics, stereo panning, or HRTFs that are generic or specific to a user to the input audio signal to spatialize the input audio signal.

The audio system computes520input spectral data defining spectral information of the input audio signal. The input audio signal may include one or more channels. For multiple channels, a single channel representation of the input audio signal may be generated by summing the channels, and the input spectral data is computed using the single channel representation.

For time-domain processing, the input spectral data is represented by an input spectral curve. The input spectral curve may be computed using an autoregressive model or a linear predictive coding. In some embodiments, a sound coloration model is used to determine an order of the autoregressive model.

For frequency-domain processing, the input spectral data is represented by input frequency magnitude vectors. The input frequency magnitude vectors may be computed using a subband processing, such as based on an analysis filter bank implementation. The level of the detail in the input frequency magnitude vectors may be controlled using a frequency resolution of the analysis filter bank.

The audio system computes5303D spectral data defining spectral information of a single channel representation of the 3D audio signal. The single channel representation of the 3D audio signal may be generated by summing the channels. The 3D spectral data may be generated using the same type of processing used to generate the input spectral data.

For time-domain processing, the 3D spectral data is represented by a 3D spectral curve. The 3D spectral curve may be computed using an autoregressive model or a linear predictive coding. In some embodiments, a sound coloration model is used to determine an order of the autoregressive model.

For frequency-domain processing, the 3D spectral data is represented by 3D frequency magnitude vectors. The 3D frequency magnitude vectors may be computed using a subband processing, such as based on an analysis filter bank implementation. The level of the detail in the input frequency magnitude vectors may be controlled using a frequency resolution of the analysis filter bank.

The audio system generates540a tonal balance filter based on the input spectral data and the 3D spectral data. When applied to the 3D audio signal, the tonal balance filter reduces or removes the sound coloration introduced by the 3D-audio rendering, while preserving the original spectral content of the input audio signal. In some embodiments, the tonal balance filter is modified using a sound coloration model. For example, the difference between the input spectral data and 3D spectral data may be compared against a selected tolerance curve in the sound coloration model. For differences that fall below the tolerance curve, no coloration correction may be required, and the tonal balance filter may be built based on differences above the tolerance curve.

For time-domain processing, generating the tonal balance filter may include determining a convolution between the input spectral curve and an inverse of the 3D spectral curve. The autoregressive model of the 3D spectral curve is an all-pole model and thus the inverse of the 3D spectral curve includes an all-zero model. This results in stability of the tonal balance filter. For frequency-domain processing, generating the tonal balance filter may include determining a ratio between the input frequency magnitude vectors and the 3D frequency magnitude vectors.

The audio system applies550the tonal balance filter to the 3D audio signal to generate an output audio signal. The tonal balance filter may be applied to each channel of the 3D audio signal to generate the output audio signal. Application of the tonal balance filter to the 3D audio signal may be performed in the time-domain or frequency-domain. In the time domain, the tonal balance filter is convolved with the 3D audio signal. In the frequency domain, the tonal balance filter is multiplied with subband representations of the channels of the 3D audio signal to generate subband outputs. The subband outputs are transformed to the time-domain to generate the output audio signal, such as by using a synthesis filter bank.

The audio system presents560, via a speaker array, the output audio signal. For example, each channel of the output audio signal may be provided to a respective speaker of the speaker array. The speaker array may include two or more speakers. The 3D audio signal and the output audio signal may include two or more channels, each corresponding to one of the speakers of the speaker array. The speaker array may include various types of speakers, including stereo speakers, speakers on a headset (e.g., headset100or105), headphones, loudspeakers, surround sound (e.g.,5.1audio system) speakers, smart speakers, etc.

FIG.6is a system600that includes a headset605, in accordance with one or more embodiments. In some embodiments, the headset605may be the headset100ofFIG.1Aor the headset105ofFIG.1B. The system600may operate in an artificial reality environment (e.g., a virtual reality environment, an augmented reality environment, a mixed reality environment, or some combination thereof). The system600shown byFIG.6includes the headset605, an input/output (I/O) interface610that is coupled to a console615, the network620, and the mapping server625. WhileFIG.6shows an example system600including one headset605and one I/O interface610, in other embodiments any number of these components may be included in the system600. For example, there may be multiple headsets each having an associated I/O interface610, with each headset and I/O interface610communicating with the console615. In alternative configurations, different and/or additional components may be included in the system600. Additionally, functionality described in conjunction with one or more of the components shown inFIG.6may be distributed among the components in a different manner than described in conjunction withFIG.6in some embodiments. For example, some or all of the functionality of the console615may be provided by the headset605.

The headset605includes the display assembly630, an optics block635, one or more position sensors640, and the DCA645. Some embodiments of headset605have different components than those described in conjunction withFIG.6. Additionally, the functionality provided by various components described in conjunction withFIG.6may be differently distributed among the components of the headset605in other embodiments, or be captured in separate assemblies remote from the headset605.

The display assembly630displays content to the user in accordance with data received from the console615. The display assembly630displays the content using one or more display elements (e.g., the display elements120). A display element may be, e.g., an electronic display. In various embodiments, the display assembly630comprises a single display element or multiple display elements (e.g., a display for each eye of a user). Examples of an electronic display include: a liquid crystal display (LCD), an organic light emitting diode (OLED) display, an active-matrix organic light-emitting diode display (AMOLED), a waveguide display, some other display, or some combination thereof. Note in some embodiments, the display element120may also include some or all of the functionality of the optics block635.

The optics block635may magnify image light received from the electronic display, corrects optical errors associated with the image light, and presents the corrected image light to one or both eyeboxes of the headset605. In various embodiments, the optics block635includes one or more optical elements. Example optical elements included in the optics block635include: an aperture, a Fresnel lens, a convex lens, a concave lens, a filter, a reflecting surface, or any other suitable optical element that affects image light. Moreover, the optics block635may include combinations of different optical elements. In some embodiments, one or more of the optical elements in the optics block635may have one or more coatings, such as partially reflective or anti-reflective coatings.

Magnification and focusing of the image light by the optics block635allows the electronic display to be physically smaller, weigh less, and consume less power than larger displays. Additionally, magnification may increase the field of view of the content presented by the electronic display. For example, the field of view of the displayed content is such that the displayed content is presented using almost all (e.g., approximately 110 degrees diagonal), and in some cases, all of the user's field of view. Additionally, in some embodiments, the amount of magnification may be adjusted by adding or removing optical elements.

In some embodiments, the optics block635may be designed to correct one or more types of optical error. Examples of optical error include barrel or pincushion distortion, longitudinal chromatic aberrations, or transverse chromatic aberrations. Other types of optical errors may further include spherical aberrations, chromatic aberrations, or errors due to the lens field curvature, astigmatisms, or any other type of optical error. In some embodiments, content provided to the electronic display for display is pre-distorted, and the optics block635corrects the distortion when it receives image light from the electronic display generated based on the content.

The position sensor640is an electronic device that generates data indicating a position of the headset605. The position sensor640generates one or more measurement signals in response to motion of the headset605. The position sensor190is an embodiment of the position sensor640. Examples of a position sensor640include: one or more IMUS, one or more accelerometers, one or more gyroscopes, one or more magnetometers, another suitable type of sensor that detects motion, or some combination thereof. The position sensor640may include multiple accelerometers to measure translational motion (forward/back, up/down, left/right) and multiple gyroscopes to measure rotational motion (e.g., pitch, yaw, roll). In some embodiments, an IMU rapidly samples the measurement signals and calculates the estimated position of the headset605from the sampled data. For example, the IMU integrates the measurement signals received from the accelerometers over time to estimate a velocity vector and integrates the velocity vector over time to determine an estimated position of a reference point on the headset605. The reference point is a point that may be used to describe the position of the headset605. While the reference point may generally be defined as a point in space, however, in practice the reference point is defined as a point within the headset605.

The DCA645generates depth information for a portion of the local area. The DCA includes one or more imaging devices and a DCA controller. The DCA645may also include an illuminator. Operation and structure of the DCA645is described above with regard toFIG.1A.

The audio system650provides audio content to a user of the headset605. The audio system650is substantially the same as the audio system200describe above. For example, the audio system650generates a 3D audio signal from an input audio signal and applies a tonal balance filer to the 3D audio signal to reduce sound coloration caused by the 3D audio rendering. The audio system650may comprise one or more acoustic sensors, one or more transducers, and an audio controller. The audio system650may provide spatialized audio content to the user. In some embodiments, the audio system650may request acoustic parameters from the mapping server625over the network620. The acoustic parameters describe one or more acoustic properties (e.g., room impulse response, a reverberation time, a reverberation level, etc.) of the local area. The audio system650may provide information describing at least a portion of the local area from e.g., the DCA645and/or location information for the headset605from the position sensor640. The audio system650may generate one or more sound filters using one or more of the acoustic parameters received from the mapping server625, and use the sound filters to provide audio content to the user.

The I/O interface610is a device that allows a user to send action requests and receive responses from the console615. An action request is a request to perform a particular action. For example, an action request may be an instruction to start or end capture of image or video data, or an instruction to perform a particular action within an application. The I/O interface610may include one or more input devices. Example input devices include: a keyboard, a mouse, a game controller, or any other suitable device for receiving action requests and communicating the action requests to the console615. An action request received by the I/O interface610is communicated to the console615, which performs an action corresponding to the action request. In some embodiments, the I/O interface610includes an IMU that captures calibration data indicating an estimated position of the I/O interface610relative to an initial position of the I/O interface610. In some embodiments, the I/O interface610may provide haptic feedback to the user in accordance with instructions received from the console615. For example, haptic feedback is provided when an action request is received, or the console615communicates instructions to the I/O interface610causing the I/O interface610to generate haptic feedback when the console615performs an action.

The console615provides content to the headset605for processing in accordance with information received from one or more of: the DCA645, the headset605, and the I/O interface610. In the example shown inFIG.6, the console615includes an application store655, a tracking module660, and an engine665. Some embodiments of the console615have different modules or components than those described in conjunction withFIG.6. Similarly, the functions further described below may be distributed among components of the console615in a different manner than described in conjunction withFIG.6. In some embodiments, the functionality discussed herein with respect to the console615may be implemented in the headset605, or a remote system.

The application store655stores one or more applications for execution by the console615. An application is a group of instructions, that when executed by a processor, generates content for presentation to the user. Content generated by an application may be in response to inputs received from the user via movement of the headset605or the I/O interface610. Examples of applications include: gaming applications, conferencing applications, video playback applications, or other suitable applications.

The tracking module660tracks movements of the headset605or of the I/O interface610using information from the DCA645, the one or more position sensors640, or some combination thereof. For example, the tracking module660determines a position of a reference point of the headset605in a mapping of a local area based on information from the headset605. The tracking module660may also determine positions of an object or virtual object. Additionally, in some embodiments, the tracking module660may use portions of data indicating a position of the headset605from the position sensor640as well as representations of the local area from the DCA645to predict a future location of the headset605. The tracking module660provides the estimated or predicted future position of the headset605or the I/O interface610to the engine665.

The engine665executes applications and receives position information, acceleration information, velocity information, predicted future positions, or some combination thereof, of the headset605from the tracking module660. Based on the received information, the engine665determines content to provide to the headset605for presentation to the user. For example, if the received information indicates that the user has looked to the left, the engine665generates content for the headset605that mirrors the user's movement in a virtual local area or in a local area augmenting the local area with additional content. Additionally, the engine665performs an action within an application executing on the console615in response to an action request received from the I/O interface610and provides feedback to the user that the action was performed. The provided feedback may be visual or audible feedback via the headset605or haptic feedback via the I/O interface610.

The network620couples the headset605and/or the console615to the mapping server625. The network620may include any combination of local area and/or wide area networks using both wireless and/or wired communication systems. For example, the network620may include the Internet, as well as mobile telephone networks. In one embodiment, the network620uses standard communications technologies and/or protocols. Hence, the network620may include links using technologies such as Ethernet, 802.11, worldwide interoperability for microwave access (WiMAX), 2G/3G/4G mobile communications protocols, digital subscriber line (DSL), asynchronous transfer mode (ATM), InfiniBand, PCI Express Advanced Switching, etc. Similarly, the networking protocols used on the network620can include multiprotocol label switching (MPLS), the transmission control protocol/Internet protocol (TCP/IP), the User Datagram Protocol (UDP), the hypertext transport protocol (HTTP), the simple mail transfer protocol (SMTP), the file transfer protocol (FTP), etc. The data exchanged over the network620can be represented using technologies and/or formats including image data in binary form (e.g. Portable Network Graphics (PNG)), hypertext markup language (HTML), extensible markup language (XML), etc. In addition, all or some of links can be encrypted using conventional encryption technologies such as secure sockets layer (SSL), transport layer security (TLS), virtual private networks (VPNs), Internet Protocol security (IPsec), etc.

The mapping server625may include a database that stores a virtual model describing a plurality of spaces, wherein one location in the virtual model corresponds to a current configuration of a local area of the headset605. The mapping server625receives, from the headset605via the network620, information describing at least a portion of the local area and/or location information for the local area. The user may adjust privacy settings to allow or prevent the headset605from transmitting information to the mapping server625. The mapping server625determines, based on the received information and/or location information, a location in the virtual model that is associated with the local area of the headset605. The mapping server625determines (e.g., retrieves) one or more acoustic parameters associated with the local area, based in part on the determined location in the virtual model and any acoustic parameters associated with the determined location. The mapping server625may transmit the location of the local area and any values of acoustic parameters associated with the local area to the headset605.

One or more components of system600may contain a privacy module that stores one or more privacy settings for user data elements. The user data elements describe the user or the headset605. For example, the user data elements may describe a physical characteristic of the user, an action performed by the user, a location of the user of the headset605, a location of the headset605, an HRTF for the user, etc. Privacy settings (or “access settings”) for a user data element may be stored in any suitable manner, such as, for example, in association with the user data element, in an index on an authorization server, in another suitable manner, or any suitable combination thereof.

A privacy setting for a user data element specifies how the user data element (or particular information associated with the user data element) can be accessed, stored, or otherwise used (e.g., viewed, shared, modified, copied, executed, surfaced, or identified). In some embodiments, the privacy settings for a user data element may specify a “blocked list” of entities that may not access certain information associated with the user data element. The privacy settings associated with the user data element may specify any suitable granularity of permitted access or denial of access. For example, some entities may have permission to see that a specific user data element exists, some entities may have permission to view the content of the specific user data element, and some entities may have permission to modify the specific user data element. The privacy settings may allow the user to allow other entities to access or store user data elements for a finite period of time.

The privacy settings may allow a user to specify one or more geographic locations from which user data elements can be accessed. Access or denial of access to the user data elements may depend on the geographic location of an entity who is attempting to access the user data elements. For example, the user may allow access to a user data element and specify that the user data element is accessible to an entity only while the user is in a particular location. If the user leaves the particular location, the user data element may no longer be accessible to the entity. As another example, the user may specify that a user data element is accessible only to entities within a threshold distance from the user, such as another user of a headset within the same local area as the user. If the user subsequently changes location, the entity with access to the user data element may lose access, while a new group of entities may gain access as they come within the threshold distance of the user.

The system600may include one or more authorization/privacy servers for enforcing privacy settings. A request from an entity for a particular user data element may identify the entity associated with the request and the user data element may be sent only to the entity if the authorization server determines that the entity is authorized to access the user data element based on the privacy settings associated with the user data element. If the requesting entity is not authorized to access the user data element, the authorization server may prevent the requested user data element from being retrieved or may prevent the requested user data element from being sent to the entity. Although this disclosure describes enforcing privacy settings in a particular manner, this disclosure contemplates enforcing privacy settings in any suitable manner.

ADDITIONAL CONFIGURATION INFORMATION

The foregoing description of the embodiments has been presented for illustration; it is not intended to be exhaustive or to limit the patent rights to the precise forms disclosed. Persons skilled in the relevant art can appreciate that many modifications and variations are possible considering the above disclosure.