Patent Description:
In recent years, there is an increasing interest in Augmented Reality (AR), Virtual Reality (VR), and Mixed Reality (MR) technologies. Advances to image processing and computer vision technologies in the wireless space, have led to better rendering and computational resources allocated to improving the visual quality and immersive visual experience of these technologies.

In VR technologies, virtual information may be presented to a user using a head-mounted display such that the user may visually experience an artificial world on a screen in front of their eyes. In AR technologies, the real-world is augmented by visual objects that are super-imposed, or, overlaid on physical objects in the real-world. The augmentation may insert new visual objects or mask visual objects to the real-world environment. In MR technologies, the boundary between what's real or synthetic/virtual and visually experienced by a user is becoming difficult to discern. <CIT> discloses a method and system for generating view adaptive spatial audio. The method includes facilitating receipt of a spatial audio. The spatial audio comprises a plurality of audio adaptation sets, each audio adaptation set associated with a region among a plurality of regions, each audio adaptation set comprising one or more audio signals encoded at one or more bit rates, each of the one or more audio signals segmented into a plurality of audio segments. The method includes detecting a change in region from a source region to a destination region associated with a change in a head orientation of a user. The source region and the destination region are from among the plurality of regions. Further, the method includes facilitating a playback of the spatial audio by at least in part performing crossfading between at least one audio segment each of the source region and the destination region. <CIT> discloses a method that improves performance of a computer that provides binaural sound to a listener. A memory stores coordinate locations that follow a path of how the head of the listener moves. This path is retrieved in anticipation of subsequent head movements of the listener to improve computer performance of executing binaural sound. <CIT> discloses the technology allowing for a wearable display device, such as a head-mounted display, to be tracked within a 3D space by dynamically generating 6DoF data associated with an orientation and location of the display device within the 3D space. The 6DoF data is generated dynamically, in real time, by combining of 3DoF location information and 3DoF orientation information within a user-centered coordinate system. The 3DoF location information may be retrieved from depth maps acquired from a depth sensitive device, while the 3DoF orientation information may be received from the display device equipped with orientation and motion sensors. The dynamically generated 6DoF data can be used to provide <NUM>-degree virtual reality simulation, which may be rendered and displayed on the wearable display device. <CIT> discloses an audio signal processing device for processing an audio signal. The audio signal processing device includes a receiving unit configured to receive the audio signal; a processor configured to determine whether to render the audio signal by reflecting a location of a sound image simulated by the audio signal on the basis of metadata for the audio signal, and render the audio signal according to a result of the determination; and an output unit configured to output the rendered audio signal. <CIT> discloses an audio signal processing device. The audio signal processing device includes a receiving unit configured to receive an ambisonic signal and an object signal, a processor configured to modify a magnitude of a specific directional component of the ambisonic signal based on a location of an object simulated by the object signal, and render a signal generated based on the object signal and the ambisonic signal having a magnitude-modified specific directional component, and an output unit configured to output the rendered signal. SUMMARY[<NUM>] This disclosure relates generally to auditory aspects of the user experience of computer-mediated reality systems, including virtual reality (VR), mixed reality (MR), augmented reality (AR), computer vision, and graphics systems. More specifically, the techniques may enable rendering of audio data for VR, MR, AR, etc. that accounts for five or more degrees of freedom on devices or systems that support fewer than five degrees of freedom. As one example, the techniques may enable rendering of audio data that accounts for six degrees of freedom (yaw, pitch, and roll plus x, y, and z translation of the user in space) on devices or systems that only support three degrees of freedom (yaw, pitch, and roll) in terms of head movements or on devices or systems that support zero degrees of freedom.

Optional features are defined in the dependent claims.

The details of one or more examples of this disclosure are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of various aspects of the techniques will be apparent from the description and drawings, and from the claims.

Particular implementations of the present disclosure are described below with reference to the drawings. In the description, common features are designated by common reference numbers throughout the drawings. As used herein, various terminology is used for the purpose of describing particular implementations only and is not intended to be limiting. For example, the singular forms "a,", "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It may be further understood that the terms "comprise," "comprises," and "comprising" may be used interchangeably with "include," "includes," or "including. " Additionally, it will be understood that the term "wherein" may be used interchangeably with "where. " As used herein, "exemplary" may indicate an example, an implementation, and/or an aspect, and should not be construed as limiting or as indicating a preference or a preferred implementation. As used herein, an ordinal term (e.g., "first," "second," "third," etc.) used to modify an element, such as a structure, a component, an operation, etc., does not by itself indicate any priority or order of the element with respect to another element, but rather merely distinguishes the element from another element having a same name (but for use of the ordinal term). As used herein, the term "set" refers to a grouping of one or more elements, and the term "plurality" refers to multiple elements.

As used herein "coupled" may include "communicatively coupled," "electrically coupled," or "physically coupled," and may also (or alternatively) include any combinations thereof. Two devices (or components) may be coupled (e.g., communicatively coupled, electrically coupled, or physically coupled) directly or indirectly via one or more other devices, components, wires, buses, networks (e.g., a wired network, a wireless network, or a combination thereof), etc. Two devices (or components) that are electrically coupled may be included in the same device or in different devices and may be connected via electronics, one or more connectors, or inductive coupling, as illustrative, non-limiting examples. In some implementations, two devices (or components) that are communicatively coupled, such as in electrical communication, may send and receive electrical signals (digital signals or analog signals) directly or indirectly, such as via one or more wires, buses, networks, etc. As used herein, "directly coupled" may include two devices that are coupled (e.g., communicatively coupled, electrically coupled, or physically coupled) without intervening components.

As used herein, "integrated" may include "manufactured or sold with". A device may be integrated if a user buys a package that bundles or includes the device as part of the package. In some descriptions, two devices may be coupled, but not necessarily integrated (e.g., different peripheral devices may not be integrated to a command device, but still may be "coupled"). Another example may be that any of the transceivers or antennas described herein that may be "coupled" to a processor, but not necessarily part of the package that includes an AR, VR or MR device. Other examples may be inferred from the context disclosed herein, including this paragraph, when using the term "integrated".

As used herein "a wireless" connection between devices may be based on various wireless technologies, such as Bluetooth, Wireless-Fidelity (Wi-Fi) or variants of Wi-Fi (e.g. Wi-Fi Direct. Devices may be "wirelessly connected" based on different cellular communication systems, such as, a Long-Term Evolution (LTE) system, a Code Division Multiple Access (CDMA) system, a Global System for Mobile Communications (GSM) system, a wireless local area network (WLAN) system, or some other wireless system. A CDMA system may implement Wideband CDMA (WCDMA), CDMA 1X, Evolution-Data Optimized (EVDO), Time Division Synchronous CDMA (TD-SCDMA), or some other version of CDMA. In addition, when two devices are within line of sight, a "wireless connection" may also be based on other wireless technologies, such as ultrasound, infrared, pulse radio frequency electromagnetic energy, structured light, or directional of arrival techniques used in signal processing (e.g. audio signal processing or radio frequency processing).

As used herein A "and/or" B may mean that either "A and B", or "A or B", or both "A and B" and "A or B" are applicable or acceptable.

The term "computing device" is used generically herein to refer to any one or all of servers, personal computers, laptop computers, tablet computers, mobile devices, cellular telephones, smartbooks, ultrabooks, palm-top computers, personal data assistants (PDA's), wireless electronic mail receivers, multimedia Internet-enabled cellular telephones, Global Positioning System (GPS) receivers, wireless gaming controllers, and similar electronic devices which include a programmable processor and circuitry for wirelessly sending and/or receiving information.

There are various 'surround-sound' channel-based formats in the market, ranging, for example, from the <NUM> home theatre system (which has been the most successful in terms of making inroads into living rooms beyond stereo) to the <NUM> system developed by NHK (Nippon Hoso Kyokai or Japan Broadcasting Corporation). Content creators (e.g., Hollywood studios) would like to produce the soundtrack for a movie once and not spend effort to remix the soundtrack for each potential speaker configuration. A Moving Pictures Expert Group (MPEG) has released a standard allowing for soundfields to be represented using a hierarchical set of elements (e.g., Higher-Order Ambisonic - HOA - coefficients) that can be rendered to speaker feeds for most speaker configurations, including <NUM> and <NUM> configuration whether in location defined by various standards or in non-uniform locations.

MPEG released the standard as MPEG-H 3D Audio standard, formally entitled "Information technology - High efficiency coding and media delivery in heterogeneous environments - Part <NUM>: 3D audio," set forth by ISO/IEC JTC <NUM>/SC <NUM>, with document identifier ISO/IEC DIS <NUM>-<NUM>, and dated July <NUM>, <NUM>. MPEG also released a second edition of the 3D Audio standard, entitled "Information technology - High efficiency coding and media delivery in heterogeneous environments - Part <NUM>: 3D audio, set forth by ISO/IEC JTC <NUM>/SC <NUM>, with document identifier ISO/IEC <NUM>-<NUM>:201x(E), and dated October <NUM>, <NUM>. Reference to the "3D Audio standard" in this disclosure may refer to one or both of the above standards.

As noted above, one example of a hierarchical set of elements is a set of spherical harmonic coefficients (SHC). The following expression demonstrates a description or representation of a soundfield using SHC: <MAT>.

The expression shows that the pressure pi at any point {rr, θr, φr} of the soundfield, at time t, can be represented uniquely by the SHC, <MAT>. Here, <MAT>, c is the speed of sound (~<NUM>/s), {rr, θr, φr} is a point of reference (or observation point), jn(·) is the spherical Bessel function of order n, and <MAT> are the spherical harmonic basis functions (which may also be referred to as a spherical basis function) of order n and suborder m. It can be recognized that the term in square brackets is a frequency-domain representation of the signal (i.e., S(ω, rr, θr, φr)) which can be approximated by various time-frequency transformations, such as the discrete Fourier transform (DFT), the discrete cosine transform (DCT), or a wavelet transform. Other examples of hierarchical sets include sets of wavelet transform coefficients and other sets of coefficients of multiresolution basis functions.

<FIG> is a diagram illustrating spherical harmonic basis functions from the zero order (n = <NUM>) to the fourth order (n = <NUM>). As can be seen, for each order, there is an expansion of suborders m which are shown but not explicitly noted in the example of <FIG> for ease of illustration purposes.

The SHC <MAT> can either be physically acquired (e.g., recorded) by various microphone array configurations or, alternatively, they can be derived from channel-based or object-based descriptions of the soundfield. The SHC (which also may be referred to as higher order ambisonic - HOA - coefficients) represent scene-based audio, where the SHC may be input to an audio encoder to obtain encoded SHC that may promote more efficient transmission or storage. For example, a fourth-order representation involving (<NUM>+<NUM>)<NUM> (<NUM>, and hence fourth order) coefficients may be used.

As noted above, the SHC may be derived from a microphone recording using a microphone array. Various examples of how SHC may be derived from microphone arrays are described in <NPL>.

To illustrate how the SHCs may be derived from an object-based description, consider the following equation. The coefficients <MAT> for the soundfield corresponding to an individual audio object may be expressed as: <MAT> where i is <MAT> is the spherical Hankel function (of the second kind) of order n, and {rs, θs, φs} is the location of the object. Knowing the object source energy g(ω) as a function of frequency (e.g., using time-frequency analysis techniques, such as performing a fast Fourier transform on the PCM stream) allows us to convert each PCM object and the corresponding location into the SHC <MAT>. Further, it can be shown (since the above is a linear and orthogonal decomposition) that the <MAT> coefficients for each object are additive. In this manner, a number of PCM objects can be represented by the <MAT> coefficients (e.g., as a sum of the coefficient vectors for the individual objects). Essentially, the coefficients contain information about the soundfield (the pressure as a function of 3D coordinates), and the above represents the transformation from individual objects to a representation of the overall soundfield, in the vicinity of the observation point {rr, θr, φr}. The remaining figures are described below in the context of SHC-based audio coding.

<FIG> and <FIG> are diagrams illustrating systems that may perform various aspects of the techniques described in this disclosure. As shown in the example of <FIG>, system <NUM> includes a source device <NUM> and a content consumer device <NUM>. While described in the context of the source device <NUM> and the content consumer device <NUM>, the techniques may be implemented in any context in which any representation of a soundfield (including scene-based audio data - such as HOA coefficients, object-based audio data, and channel-based audio data) is encoded to form a bitstream representative of the audio data.

Moreover, the source device <NUM> may represent any form of computing device capable of generating a representation of a soundfield. Source device <NUM> is generally described herein in the context of being a VR content creator device although source device <NUM> may take other forms. Likewise, the content consumer device <NUM> may represent any form of computing device capable of implementing the techniques described in this disclosure as well as audio playback. Content consumer device <NUM> is generally described herein in the context of being a VR client device but may take other forms.

The source device <NUM> may be operated by an entertainment company or other entity that generates multi-channel audio content for consumption by operators of content consumer devices, such as the content consumer device <NUM>. In many VR scenarios, the source device <NUM> generates audio content in conjunction with video content. The source device <NUM> includes a content capture device <NUM> and a content capture assistant device <NUM>. The content capture device <NUM> may be configured to interface or otherwise communicate with a microphone <NUM>. The microphone <NUM> may represent an Eigenmike® or other type of 3D audio microphone capable of capturing and representing the soundfield as audio data <NUM>.

The content capture device <NUM> may, in some examples, include an integrated microphone <NUM> that is integrated into the housing of the content capture device <NUM>. The content capture device <NUM> may interface wirelessly or via a wired connection with the microphone <NUM>. Rather than capture, or in conjunction with capturing, audio data via microphone <NUM>, the content capture device <NUM> may process the audio data <NUM> after the audio data <NUM> are input via some type of removable storage, wirelessly and/or via wired input processes. As such, various combinations of the content capture device <NUM> and the microphone <NUM> are possible in accordance with this disclosure.

The content capture device <NUM> may also be configured to interface or otherwise communicate with the soundfield representation generator <NUM>. The soundfield representation generator <NUM> may include any type of hardware device capable of interfacing with the content capture device <NUM>. The soundfield representation generator <NUM> may the use audio data <NUM> provided by the content capture device <NUM> to generate various representations of the same soundfield represented by the audio data <NUM>. For instance, to generate the different representations of the soundfield using the audio data <NUM>, soundfield representation generator <NUM> may use a coding scheme for ambisonic representations of a soundfield, referred to as Mixed Order Ambisonics (MOA) as discussed in more detail in <CIT>, and granted on September <NUM>, <NUM>.

To generate a particular MOA representation of the soundfield, the soundfield representation generator <NUM> may generate a partial subset of a full set of HOA coefficients. For instance, each MOA representation generated by the soundfield representation generator <NUM> may provide precision with respect to some areas of the soundfield, but less precision in other areas. In one example, an MOA representation of the soundfield may include eight (<NUM>) uncompressed HOA coefficients, while the third order HOA representation of the same soundfield may include sixteen (<NUM>) uncompressed HOA coefficients. As such, each MOA representation of the soundfield that is generated as a partial subset of the HOA coefficients may be less storage-intensive and less bandwidth intensive (if and when transmitted as part of the bitstream <NUM> over the illustrated transmission channel) than the corresponding third order HOA representation of the same soundfield generated from the HOA coefficients.

Although described with respect to MOA representations, the techniques of this disclosure may also be performed with respect to full-order ambisonic (FOA) representations in which all of the HOA coefficients for a given order N are used to represent the soundfield. In other words, rather than represent the soundfield using a partial, non-zero subset of the audio data <NUM>, the soundfield representation generator <NUM> may represent the soundfield using all of the audio data <NUM> for a given order N, resulting in a total of HOA coefficients equaling (N+<NUM>)<NUM>.

In this respect, the higher order ambisonic audio data <NUM> may include higher order ambisonic coefficients <NUM> associated with spherical basis functions having an order of one or less (which may be referred to as "<NUM>st order ambisonic audio data <NUM>"), higher order ambisonic coefficients associated with spherical basis functions having a mixed order and suborder (which may be referred to as the "MOA representation" discussed above), or higher order ambisonic coefficients associated with spherical basis functions having an order greater than one (which is referred to above as the "FOA representation").

The content capture device <NUM> may, in some examples, be configured to wirelessly communicate with the soundfield representation generator <NUM>. In some examples, the content capture device <NUM> may communicate, via one or both of a wireless connection or a wired connection, with the soundfield representation generator <NUM>. Via the connection between the content capture device <NUM> and the soundfield representation generator <NUM>, the content capture device <NUM> may provide content in various forms of content, which, for purposes of discussion, are described herein as being portions of the audio data <NUM>.

In some examples, the content capture device <NUM> may leverage various aspects of the soundfield representation generator <NUM> (in terms of hardware or software capabilities of the soundfield representation generator <NUM>). For example, the soundfield representation generator <NUM> may include dedicated hardware configured to (or specialized software that when executed causes one or more processors to) perform psychoacoustic audio encoding (such as a unified speech and audio coder denoted as "USAC" set forth by the Motion Picture Experts Group (MPEG) or the MPEG-H 3D audio coding standard). The content capture device <NUM> may not include the psychoacoustic audio encoder dedicated hardware or specialized software and instead provide audio aspects of the content <NUM> in a non-psychoacoustic-audio-coded form. The soundfield representation generator <NUM> may assist in the capture of content <NUM> by, at least in part, performing psychoacoustic audio encoding with respect to the audio aspects of the content <NUM>.

The soundfield representation generator <NUM> may also assist in content capture and transmission by generating one or more bitstreams <NUM> based, at least in part, on the audio content (e.g., MOA representations and/or third order HOA representations) generated from the audio data <NUM>. The bitstream <NUM> may represent a compressed version of the audio data <NUM> and any other different types of the content <NUM> (such as a compressed version of spherical video data, image data, or text data).

The soundfield representation generator <NUM> may generate the bitstream <NUM> for transmission, as one example, across a transmission channel, which may be a wired or wireless channel, a data storage device, or the like. The bitstream <NUM> may represent an encoded version of the audio data <NUM> and may include a primary bitstream and another side bitstream, which may be referred to as side channel information. In some instances, the bitstream <NUM> representing the compressed version of the audio data may conform to bitstreams produced in accordance with the MPEG-H 3D audio coding standard.

The content consumer device <NUM> may be operated by an individual and may represent a VR client device. Although described with respect to a VR client device, content consumer device <NUM> may represent other types of devices, such as an augmented reality (AR) client device, a mixed reality (MR) client device, a standard computer, a headset, headphones, or any other device capable of tracking head movements and/or general translational movements of the individual operating the client consumer device <NUM>. As shown in the example of <FIG>, the content consumer device <NUM> includes an audio playback system <NUM>, which may refer to any form of audio playback system capable of rendering audio data, including one or more of SHC (whether in form of third order HOA representations and/or MOA representations), audio objects, and audio channels, for playback as multi-channel audio content.

While shown in <FIG> as being directly transmitted to the content consumer device <NUM>, the source device <NUM> may output the bitstream <NUM> to an intermediate device positioned between the source device <NUM> and the content consumer device <NUM>. The intermediate device may store the bitstream <NUM> for later delivery to the content consumer device <NUM>, which may request the bitstream. The intermediate device may comprise a file server, a web server, a desktop computer, a laptop computer, a tablet computer, a mobile phone, a smart phone, or any other device capable of storing the bitstream <NUM> for later retrieval by an audio decoder. The intermediate device may reside in a content delivery network capable of streaming the bitstream <NUM> (and possibly in conjunction with transmitting a corresponding video data bitstream) to subscribers, such as the content consumer device <NUM>, requesting the bitstream <NUM>.

Alternatively, the source device <NUM> may store the bitstream <NUM> to a storage medium, such as a compact disc, a digital video disc, a high definition video disc or other storage media, most of which are capable of being read by a computer and therefore may be referred to as computer-readable storage media or non-transitory computer-readable storage media. In this context, the transmission channel may refer to the channels by which content stored to the mediums are transmitted (and may include retail stores and other store-based delivery mechanism). In any event, the techniques of this disclosure should not therefore be limited in this respect to the example of <FIG>.

As noted above, the content consumer device <NUM> includes the audio playback system <NUM>. The audio playback system <NUM> may represent any system capable of playing back channel-based audio data. The audio playback system <NUM> may include a number of different renderers <NUM>. The renderers <NUM> may each provide for a different form of rendering, where the different forms of rendering may include one or more of the various ways of performing vector-base amplitude panning (VBAP), and/or one or more of the various ways of performing soundfield synthesis. As used herein, "A and/or B" means "A or B", or both "A and B".

The audio playback system <NUM> may further include an audio decoding device <NUM>. The audio decoding device <NUM> may represent a device configured to decode bitstream <NUM> to output audio data <NUM> (which again, as one example, may include HOA that form the full third order HOA representation or a subset thereof that forms an MOA representation of the same soundfield or decompositions thereof, such as the predominant audio signal, ambient HOA coefficients, and the vector based signal described in the MPEG-H 3D Audio Coding Standard). As such, the audio data <NUM> may be similar to a full set or a partial subset of HOA coefficients, but may differ due to lossy operations (e.g., quantization) and/or transmission via the transmission channel. The audio playback system <NUM> may, after decoding the bitstream <NUM> to obtain the audio data <NUM>, render the audio data <NUM> to output speaker feeds <NUM>. The speaker feeds <NUM> may drive one or more speakers (which are not shown in the example of <FIG> for ease of illustration purposes). Ambisonic representations of a soundfield may be normalized in a number of ways, including N3D, SN3D, FuMa, N2D, or SN2D.

To select the appropriate renderer or, in some instances, generate an appropriate renderer, the audio playback system <NUM> may obtain loudspeaker information <NUM> indicative of a number of loudspeakers and/or a spatial geometry of the loudspeakers. In some instances, the audio playback system <NUM> may obtain the loudspeaker information <NUM> using a reference microphone and driving the loudspeakers in such a manner as to dynamically determine the loudspeaker information <NUM>. In other instances, or in conjunction with the dynamic determination of the loudspeaker information <NUM>, the audio playback system <NUM> may prompt a user to interface with the audio playback system <NUM> and input the loudspeaker information <NUM>.

The audio playback system <NUM> may select one of the audio renderers <NUM> based on the loudspeaker information <NUM>. In some instances, the audio playback system <NUM> may, when none of the audio renderers <NUM> are within some threshold similarity measure (in terms of the loudspeaker geometry) to the loudspeaker geometry specified in the loudspeaker information <NUM>, generate the one of audio renderers <NUM> based on the loudspeaker information <NUM>. The audio playback system <NUM> may, in some instances, generate one of the audio renderers <NUM> based on the loudspeaker information <NUM> without first attempting to select an existing one of the audio renderers <NUM>.

When outputting the speaker feeds <NUM> to headphones, the audio playback system <NUM> may utilize one of the renderers <NUM> that provides for binaural rendering using head-related transfer functions (HRTF) or other functions capable of rendering to left and right speaker feeds <NUM> for headphone speaker playback. The terms "speakers" or "transducer" may generally refer to any speaker, including loudspeakers, headphone speakers, etc. One or more speakers may then playback the rendered speaker feeds <NUM>.

Although described as rendering the speaker feeds <NUM> from the audio data <NUM>', reference to rendering of the speaker feeds <NUM> may refer to other types of rendering, such as rendering incorporated directly into the decoding of the audio data <NUM> from the bitstream <NUM>. An example of the alternative rendering can be found in Annex G of the MPEG-H 3D audio coding standard, where rendering occurs during the predominant signal formulation and the background signal formation prior to composition of the soundfield. As such, reference to rendering of the audio data <NUM> should be understood to refer to both rendering of the actual audio data <NUM> or decompositions or representations thereof of the audio data <NUM> (such as the above noted predominant audio signal, the ambient HOA coefficients, and/or the vector-based signal - which may also be referred to as a V-vector).

As described above, the content consumer device <NUM> may represent a VR device in which a human wearable display is mounted in front of the eyes of the user operating the VR device. <FIG> is a diagram illustrating an example of a VR device <NUM> worn by a user <NUM>. The VR device <NUM> is coupled to, or otherwise includes, headphones <NUM>, which may reproduce a soundfield represented by the audio data <NUM>' through playback of the speaker feeds <NUM>. The speaker feeds <NUM> may represent an analog or digital signal capable of causing a membrane within the transducers of headphones <NUM> to vibrate at various frequencies, where such process is commonly referred to as driving the headphones <NUM>.

Video, audio, and other sensory data may play important roles in the VR experience. To participate in a VR experience, the user <NUM> may wear the VR device <NUM> (which may also be referred to as a VR headset <NUM>) or other wearable electronic device. The VR client device (such as the VR headset <NUM>) may track head movement of the user <NUM>, and adapt the video data shown via the VR headset <NUM> to account for the head movements, providing an immersive experience in which the user <NUM> may experience a virtual world shown in the video data in visual three dimensions.

While VR (and other forms of AR and/or MR) may allow the user <NUM> to reside in the virtual world visually, often the VR headset <NUM> may lack the capability to place the user in the virtual world audibly. In other words, the VR system (which may include a computer responsible for rendering the video data and audio data - that is not shown in the example of <FIG> for ease of illustration purposes, and the VR headset <NUM>) may be unable to support full three dimension immersion audibly.

Audio has conventionally provided a user with zero degrees of freedom (<NUM> DOF), meaning that user movement does not change the audio rendering. VR, however, can provide users with some degrees of freedom, meaning the audio rendering can change based on user movement. The audio aspects of VR have been classified into three separate categories of immersion. The first category provides the lowest level of immersion and is referred to as three degrees of freedom (3DOF). 3DOF refers to audio rendering that accounts for movement of the head in the three degrees of freedom (yaw, pitch, and roll), thereby allowing the user to freely look around in any direction. 3DOF, however, cannot account for translational head movements in which the head is not centered on the optical and acoustical center of the soundfield.

The second category, referred to 3DOF plus (3DOF+), provides for the three degrees of freedom (yaw, pitch, and roll) in addition to limited spatial translational movements due to the head movements away from the optical center and acoustical center within the soundfield. 3DOF+ may provide support for perceptual effects such as motion parallax, which may strengthen the sense of immersion.

The third category, referred to as six degrees of freedom (6DOF), renders audio data in a manner that accounts for the three degrees of freedom in term of head movements (yaw, pitch, and roll) but also accounts for translation of the user in space (x, y, and z translations). The spatial translations may be induced by sensors tracking the location of the user in the physical world or by way of an input controller.

<FIG> is a diagram illustrating a six degree-of-freedom (<NUM>-DOF) head movement scheme for AVR and/or AR applications. As shown in <FIG>, the <NUM>-DOF scheme includes three additional movement lines beyond the <NUM>-DOF scheme. More specifically, the <NUM>-DOF scheme of <FIG> includes, in addition to the rotation axes discussed above, three lines along which the user's head position may translationally move or actuate. The three translational directions are left-right (L/R), up-down (U/D), and forward-backward (F/B). An audio encoding device of source device <NUM> and/or the audio decoding device <NUM> may implement parallax handling to address the three translational directions. For instance, the audio decoding device <NUM> may apply one or more transmission factors to adjust the energies and/or directional information of various foreground audio objects to implement parallax adjustments based on the <NUM>-DOF range of motion of a VR/AR user.

According to one example of this disclosure, source device <NUM> may generate audio data representative of a soundfield captured at a plurality of capture locations, metadata that enables the audio data to be rendered to support at least five degrees of freedom, and adaptation metadata that enables the audio data to be rendered to support fewer than five degrees of freedom. Content consumer device <NUM> may receive and store audio data representative of the soundfield captured at the plurality of capture locations, the metadata that enables the audio data to be rendered to support at least five degrees of freedom, and the adaptation metadata that enables the audio data to be rendered to support fewer than five degrees of freedom. Content consumer device <NUM> may adapt, based on the adaptation metadata, the audio data to provide fewer than five degrees of freedom, and audio renderers <NUM> may generate speaker feeds based on the adapted audio data.

According to another example of this disclosure, source device <NUM> may generate audio data representative of a soundfield captured at a plurality of capture locations, metadata that enables the audio data to be rendered to support six degrees of freedom, and adaptation metadata that enables the audio data to be rendered to support fewer than six degrees of freedom. Content consumer device <NUM> may receive and store audio data representative of the soundfield captured at the plurality of capture locations, the metadata that enables the audio data to be rendered to support six degrees of freedom, and the adaptation metadata that enables the audio data to be rendered to support fewer than six degrees of freedom. Content consumer device <NUM> may adapt, based on the adaptation metadata, the audio data to provide fewer than six degrees of freedom, and audio renderers <NUM> may generate speaker feeds based on the adapted audio data.

According to this disclosure, content consumer device <NUM> may store audio data representative of a soundfield captured at a plurality of capture locations; determine a user location; adapt, based on the user location, the audio data to provide M degrees of freedom. wherein M comprise an integer value; and generate speaker feeds based on the adapted audio data. To determine the user location, content consumer device <NUM> displays a plurality of user locations and receive, from a user, an input indicative of one of the plurality of locations. To determine the user location, content consumer device <NUM> displays a trajectory; receive, from a user, an input indicative of a position on the trajectory; and select based on the position of the trajectory, one of a plurality of locations as the user location. To determine the user location, content consumer device <NUM> may detect a movement of a user and select a location based on the movement. Content consumer device <NUM> may select the location based on the movement from a plurality of locations.

According to another example of this disclosure, content consumer device <NUM> may store audio data representative of a soundfield captured at a plurality of capture locations; adapt the audio data to provide M degrees of freedom; generate speaker feeds based on the adapted audio data with the M degrees of freedom; adapt the audio data to provide N degrees of freedom; and generate speaker feeds based on the adapted audio data with the N degrees of freedom. Content consumer device <NUM> may be further configured to switch between generating speaker feeds based on the adapted audio data with the M degrees of freedom and the adapted audio data with the N degrees of freedom. Content consumer device <NUM> may, for example, perform such switching in response to a user input or a user movement.

Although described with respect to a VR device as shown in the example of <FIG>, the techniques may be performed by other types of wearable devices, including watches (such as so-called "smart watches"), glasses (such as so-called "smart glasses"), headphones (including wireless headphones coupled via a wireless connection, or smart headphones coupled via wired or wireless connection), and any other type of wearable device. As such, the techniques may be performed by any type of wearable device by which a user may interact with the wearable device while worn by the user.

<FIG> is a block diagram illustrating another example system <NUM> configured to perform various aspects of the techniques described in this disclosure. The system <NUM> is similar to the system <NUM> shown in <FIG>, except that the audio renderers <NUM> shown in <FIG> are replaced with a binaural renderer <NUM> capable of performing binaural rendering using one or more HRTFs or the other functions capable of rendering to left and right speaker feeds <NUM>.

The audio playback system <NUM> may output the left and right speaker feeds <NUM> to headphones <NUM>, which may represent another example of a wearable device and which may be coupled to additional wearable devices to facilitate reproduction of the soundfield, such as a watch, the VR headset noted above, smart glasses, smart clothing, smart rings, smart bracelets or any other types of smart jewelry (including smart necklaces), and the like. The headphones <NUM> may couple wirelessly or via wired connection to the additional wearable devices.

Additionally, the headphones <NUM> may couple to the audio playback system <NUM> via a wired connection (such as a standard <NUM> audio jack, a universal system bus (USB) connection, an optical audio jack, or other forms of wired connection) or wirelessly (such as by way of a Bluetooth™ connection, a wireless network connection, and the like). The headphones <NUM> may recreate, based on the left and right speaker feeds <NUM>, the soundfield represented by the audio data <NUM>. The headphones <NUM> may include a left headphone speaker and a right headphone speaker which are powered (or, in other words, driven) by the corresponding left and right speaker feeds <NUM>.

<FIG> is a block diagram illustrating in more detail the audio playback system shown in <FIG> and <FIG>. As shown in the example of <FIG>, audio playback system <NUM> includes an effects unit <NUM> and a rendering unit <NUM> in addition to the above described audio decoding device <NUM>. The effects unit <NUM> represents a unit configured to obtain the effects matrix <NUM> (shown as "EM <NUM>" in the example of <FIG>) described above. The rendering unit <NUM> represents a unit configured to determine and/or apply one or more of the audio renderers <NUM> (shown as "ARs <NUM>" in the example of <FIG>) described above.

The audio decoding device <NUM> may, as noted above, represent a unit configured to decode bitstream <NUM> in accordance with the MPEG-H 3D Audio Coding Standard. The audio decoding device <NUM> may include a bitstream extraction unit <NUM>, an inverse gain control and reassignment unit <NUM>, a predominant sound synthesis unit <NUM>, an ambient synthesis unit <NUM>, and a composition unit <NUM>. More information concerning each of the foregoing units <NUM>-<NUM> can be found in the MPEG-H 3D Audio Coding Standard.

While described in detail in the MPEG-H 3D Audio Coding Standard, a brief description of each of the units <NUM>-<NUM> is provided below. The bitstream extraction unit <NUM> may represent a unit configured to extract decompositions of the audio data, along with other syntax elements or data required to compose a representation of the soundfield defined by the audio data <NUM>. The bitstream extraction unit <NUM> may identify one or more transport channels <NUM> in the bitstream <NUM>, each of which may specify either an ambient audio signal or a predominant audio signal. The bitstream extraction unit <NUM> may extract the transport channels <NUM> and output the transport channels <NUM> to the inverse gain control and reassignment unit <NUM>.

Although not shown in the example of <FIG> for ease of illustration purposes, the audio decoding device <NUM> may include a psychoacoustic audio decoder that performs psychoacoustic audio decoding (e.g., advanced audio coding - AAC) with respect to the transport channels <NUM>. Moreover, the audio decoding device <NUM> may include further units that perform various other operations not shown in the example of <FIG>, such as fading between transport channels <NUM>, and the like.

The bitstream extraction unit <NUM> may further extract side information <NUM> defining syntax elements and other data for performing gain control and assignment. The bitstream extraction unit <NUM> may output the side information <NUM> to inverse gain control and reassignment unit <NUM>.

The bitstream extraction unit <NUM> may also extract side information <NUM> defining syntax elements and other data for performing predominant sound synthesis (including, a vector defining spatial characteristics - such as a width, direction, and/or shape - of a corresponding predominant audio signal defined in the transport channels <NUM>). Additionally, the bitstream extraction unit <NUM> may extract side information <NUM> defining syntax elements and other data for performing ambient synthesis. The bitstream extraction unit <NUM> outputs the side information <NUM> to the predominant sound synthesis unit <NUM>, and the side information <NUM> to the ambient synthesis unit <NUM>.

The inverse gain control and reassignment unit <NUM> may represent a unit configured to perform, based on the side information <NUM>, inverse gain control and reassignment with respect to the transport channels <NUM>. The inverse gain control and reassignment unit <NUM> may determine, based on the side information <NUM>, gain control information and apply the gain control information to each of the transport channels <NUM> to invert gain control applied at the audio encoding device implemented by the soundfield representation generation <NUM> in an effort to reduce dynamic range of the transport channels <NUM>. The inverse gain control and reassignment unit <NUM> may next, based on the side information <NUM>, determine whether each of the transport channels <NUM> specifies a predominant audio signal <NUM> or an ambient audio signal <NUM>. The inverse gain control and reassignment unit <NUM> may output the predominant audio signals <NUM> to the predominant sound synthesis unit <NUM> and the ambient audio signals <NUM> to the ambient synthesis unit <NUM>.

The predominant sound synthesis unit <NUM> may represent a unit configured to synthesize, based on the side information <NUM>, predominant audio components of the soundfield represented by the audio data <NUM>. The predominant sound synthesis unit <NUM> may multiply each of the predominant audio signals <NUM> by a corresponding spatial vector (which may also be referred to as a "vector-based signal") specified in the side information <NUM>. The predominant sound synthesis unit <NUM> output, to composition unit <NUM>, the result of the multiplication as predominant sound representation <NUM>.

The ambient synthesis unit <NUM> may represent a unit configured to synthesize, based on the side information <NUM>, ambient components of the soundfield represented by the audio data <NUM>. The ambient synthesis unit <NUM> outputs, to composition unit <NUM>, the result of the synthesis as ambient sound representation <NUM>.

The composition unit <NUM> may represent a unit configured to compose, based on predominant sound representation <NUM> and the ambient sound representation <NUM>, the audio data <NUM>. The composition unit <NUM> may, in some examples, add the predominant sound representation <NUM> to the ambient sound representation <NUM> to obtain the audio data <NUM>. The composition unit <NUM> may output the audio data <NUM> to the effects unit <NUM>.

The effects unit <NUM> may represent a unit configured to perform various aspects of the effects techniques described in this disclosure to generate the EM <NUM> based on the translational distance <NUM>, or as described in more detail below, the translational distance <NUM> and a depth map <NUM>.

The depth map <NUM> shown in <FIG> was generated from a photograph. The whiter the area is in the depth map <NUM>, the closer it was to the camera when the photograph was taken. The blacker the area is in the depth map <NUM>, the further away it was from the camera when the photograph was taken. The effects unit <NUM> may apply EM <NUM> to the audio data <NUM> to obtain adapted audio data <NUM>. The adapted audio data <NUM> may be adapted to provide the three degrees of freedom plus effects that accounts, in the soundfield, for the translational head movement indicated by the translational distance <NUM>. The effects unit <NUM> may output the adapted audio data <NUM> to rendering unit <NUM>.

The rendering unit <NUM> may represent a unit configured to apply one or more of ARs <NUM> to adapted audio data <NUM>, and thereby obtain the speaker feeds <NUM>. The rendering unit <NUM> may output the speaker feeds <NUM> to the headphones <NUM> shown in the example of <FIG>.

Although described as separate units <NUM> and <NUM>, the effects unit <NUM> may be incorporated within rendering unit <NUM>, where the EM <NUM> is multiplied by the selected one of ARs <NUM> in the manner described below in more detail. The multiplication of the EM <NUM> by the selected one of ARs <NUM> may result in an updated AR (which may be denoted as "updated AR <NUM>"). The rendering unit <NUM> may then apply the updated AR <NUM> to the audio data <NUM> to both adapt the audio data <NUM> to provide the 3DOF+ effect that accounts for the translational distance <NUM> and render the speaker feeds <NUM>.

<FIG> is a diagram illustration in more detail how the effects unit shown in the example of <FIG> obtains the effects matrix in accordance with various aspects of the techniques described in this disclosure. As shown in the example of <FIG>, the user <NUM> initially resides in the middle of recreated soundfield <NUM>, as shown at the left of <FIG> denoted "INITIAL USER LOCATION. " The recreated soundfield <NUM>, while shown as a circle, is modeled as a sphere surrounding the user <NUM> at a reference distance <NUM>. In some examples, the user <NUM> may input the reference distance <NUM> in configuring VR device <NUM> for playback of the audio data. In other examples, the reference distance <NUM> is static, or defined as a syntax element of the bitstream <NUM>. When defined using the syntax element, the reference distance <NUM> may be static (such as sent a single time and therefore static for the duration of the experience) or dynamic (such as sent multiple times during the experience, e.g., per audio frame or per some periodic or non-periodic number of audio frames).

The effects unit <NUM> may receive the reference distance <NUM> and determine anchor points <NUM> positioned at the reference distance <NUM> from the head of the user <NUM> prior the translational head movement <NUM>. The anchor points <NUM> are shown in the example of <FIG> as "X" marks. The effects unit <NUM> may determine the anchor points <NUM> as a plurality of uniformly distributed anchor points on a surface of the spherical soundfield <NUM> having a radius equal to the reference distance <NUM>. In other examples, the anchor points may be determined by a camera (not shown) and may be provided to the effects unit <NUM>.

The anchor points <NUM> may represent points of reference by which to determine translational head movement <NUM>. The anchor points <NUM> may, in other words, represent reference points distributed about the spherical soundfield <NUM> by which the translational head movement <NUM> may be determined so as to adapt the soundfield. The anchor points <NUM> should not be confused with anchor points or key points as understood in visual image search algorithms. Again, the anchor points <NUM> may denote reference points at the reference distance from the head of user <NUM> used for determining the translational head movement <NUM> relative to each of the anchor points <NUM>. The extent of the translational head movement <NUM> relative to each of the anchor points <NUM> may impact rendering with respect to the portion of the soundfield in which the respective one of the anchor points <NUM> resides. As such, the anchor points <NUM> may also represent soundfield sampling points by which to determine translational head movement <NUM> and adapt, based on the relative translational head movement <NUM>, rendering of the soundfield.

In any event, the user <NUM> may then perform the translational head movement <NUM>, moving, as shown in the example of <FIG> under the heading "USER LOCATION AFTER TRANSLATIONAL MOVEMENT," the head the translational distance <NUM> to the right. The effects unit <NUM> may determine, after the translational head movement <NUM>, an updated distance <NUM> relative to each of the plurality of anchor points <NUM>. Although only a single updated distance <NUM> is shown in the example of <FIG>, the effects unit <NUM> may determine an updated distance <NUM> relative to each of the anchor points <NUM>. The effects unit <NUM> may next determine, based on each of the updated distances <NUM>, the EM <NUM>.

The effects unit <NUM> may compute a distant-dependent loudness adjustment (in the form of the EM <NUM>) for each translated anchor point. The computation for each reference point may be denoted as g|, where the original reference distance <NUM> is denoted as distref and the updated distance <NUM> may be denoted as distnew,|. For each of the anchor points <NUM>, the effects unit <NUM> may compute g| using the equation g| = <MAT>. The distPow parameter may control the effect strength, which may be input by the user <NUM> to control the magnitude of the effect strength. While described as being a variable subject to control by the user <NUM>, the distPow parameter may also be specified by the content creator either dynamically or statically.

Mathematically, the soundfield <NUM> surrounding the user <NUM> may be represented as M equidistant anchor points <NUM> (which may also be referred to as "spatial points <NUM>") on a sphere with a center located at the head of the user <NUM>. The variable 'M' is typically selected such that M is greater than or equal to (N+<NUM>)<NUM>, where N denotes the greatest order associated with the audio data <NUM>.

The M equidistant spatial points <NUM> result in M spatial directions extending from the head of the user <NUM> to each of the M equidistant spatial points <NUM>. The M spatial directions may be represented by <IMG>. The effects unit <NUM> may obtain the EM <NUM> that is applied to the rendering matrix based on the M spatial directions, <IMG>. In one example, the effects unit <NUM> obtains EM <NUM> computed from HOA coefficients associated with each of the M spatial directions. The effects unit <NUM> may then perform loudness compensation for each of spatial direction l = <NUM>. M, which is applied to the EM <NUM> to generate the compensated EM <NUM>. While described as being M equidistant spatial point <NUM>, the points <NUM> may also be non-equidistant or, in other words, distributed about the sphere in a non-uniform manner.

In terms of the variables used by the MPEG-H 3D Audio Coding Standard, Annex F. <NUM> of the "DIS" version, when discussing, as one example, loudness compensation, the effects unit <NUM> may compute EM <NUM> from HOA coefficients associated with the M spatial directions as follows: <MAT> with <MAT>
The "†" symbol may denote the pseudo-inverse matrix operation.

The effect unit <NUM> may then perform the loudness compensation for each spatial direction l = <NUM>. M, which is applied for the matrix F according to the following: <MAT> where <MAT>. The effect unit <NUM> may then multiple the selected one of AR <NUM>, denoted below by the variable "R" by the EM <NUM>, denoted above and below by the variable "F" to generate the updated AR <NUM> discussed above and denoted as follows by the variable "D.

The foregoing may, when distance-dependent loudness adjustment is disabled, mathematically represent the distance-independent loudness adjustment by removal of the multiplication by g|, resulting in the following: <MAT> In all other respects, the mathematic representation is unchanged when distance-independent loudness adjustment is enabled (or, in other words, when distance-dependent loudness adjustment is disabled).

In this way, the effects unit <NUM> may provide the EM <NUM> to rendering unit <NUM>, which multiplies the audio renderer <NUM> that converts the audio data <NUM> from the spherical harmonic domain to the spatial domain speaker signals <NUM> (which in this case may be a binaural render that renders the audio data to binaural audio headphone speaker signals) by the compensated EM <NUM> to create an adapted spatial rendering matrix (which is referred to herein as the "updated AR <NUM>") capable of accounting for both the three degrees of freedom and the translation head movement <NUM>.

In some instances, the effects unit <NUM> may determine multiple EMs <NUM>. For example, the effects unit <NUM> may determine a first EM <NUM> for a first frequency range, a second EM <NUM> for a second frequency range, etc. The frequency ranges of the first EM <NUM> and the second EM <NUM> may overlap, or may not overlap (or, in other words, may be distinct from one another). As such, the techniques described in this disclosure should not be limited to the single EM <NUM>, but should include application of multiple EMs <NUM>, including, but not limited to, the example multiple frequency dependent EMs <NUM>.

As discussed above, the effects unit <NUM> may also determine EM <NUM> based on translational distance <NUM> and the depth map <NUM>. The bitstream <NUM> may include video data corresponding to the audio data <NUM>, where such video data is synchronized with the audio data <NUM> (using, e.g., frame synchronization information). Although not shown in the example of <FIG>, the client consumer device <NUM> may include a video playback system that decodes the corresponding bitstream providing the video data, which may include depth maps, such as the depth map <NUM>. As mentioned above, the depth map <NUM> provides a gray scale representation of the <NUM> degree virtual reality scene, where black represents a very far distance, and white represents a near distance with the various shades of gray indicated intermediate distances between black and white.

The video decoding device of the video playback system may utilize the depth map <NUM> to formulate a view for either the left eye or the right eye from the respective right eye view or left eye view specified in the video bitstream. The video decoding device may alter the amount of lateral distance between the right eye view and the left eye view based on the depth map, scaling the lateral distance smaller based on the darker the shade of grey. As such, near objects denoted in white or light shades of gray in the depth map <NUM> may have a larger lateral distance between the left and right eye views, while far objects denoted in black or darker shades of gray in the depth map <NUM> may have a smaller lateral distance between the left and right eye view (thereby more closing resembling a far off point).

The effects unit <NUM> may utilize the depth information provided by the depth map <NUM> to adapt the location of the anchor points <NUM> relative to the head of the user <NUM>. That is, the effects unit <NUM> may map the anchor points <NUM> to the depth map <NUM>, and utilize the depth information of the depth map <NUM> at the mapped locations within the depth map <NUM> to identify a more accurate reference distance <NUM> for each of the anchor points <NUM>. <FIG> is a diagram illustrating the depth map shown in <FIG> having been updated to reflect the mapping of the anchor points to the depth map in accordance with various aspects of the techniques described in this disclosure.

In this respect, instead of assuming a single reference distance <NUM>, the effects unit <NUM> may utilize the depth map <NUM> to estimate individual references distances <NUM> for each of the anchor points <NUM>. As such, the effects unit <NUM> may determine the updated distance <NUM> relative to each of the individually determined reference distances <NUM> of the anchor points <NUM>.

While described as performed with respect to a gray-scale depth map <NUM>, the techniques may be performed with respect to other types of information providing depth information, such as a color image, color or gray-scale stereo images, infrared camera images, etc. The techniques may, in other words, be performed with respect to any type of information providing depth information of a scene associated with the corresponding audio data <NUM>.

<FIG> is a flow diagram depicting techniques in accordance with the present disclosure. The audio playback system <NUM> may receive the bitstream <NUM> and may store audio data contained therein in memory (<NUM>). The audio playback system <NUM> may also store metadata contained in the bitstream <NUM> in memory (<NUM>). Additionally, the audio playback system <NUM> may store adaption metadata in memory (<NUM>). For example, the adaptation metadata may include one or more of a user location and a user orientation. In some examples, certain adaptation metadata, such as the user location and user orientation are contained in the bitstream <NUM>. In other examples, the user location and user orientation are not contained in the bitstream <NUM>, but are received by user input.

The audio playback system <NUM> may then adapt the audio data (<NUM>) based on the adaptation metadata. For example, the audio playback system <NUM> may adapt the audio data to provide fewer degrees of freedom than the source device <NUM> created. In some examples, the audio playback system <NUM> may apply an effects matrix, such as effects matrix <NUM>, to the audio data when adapting the audio data. In some examples, the audio playback system <NUM> may determine the effects matrix based upon the user location. In some examples, the audio playback system <NUM> may multiply the effects matrix <NUM> by a rendering matrix to obtain an updated rendering matrix. In some examples, the audio playback system <NUM> may obtain a rotation indication indicative of a rotational head movement of the user <NUM> and may adapt the audio data based upon the rotational indication and the adaptation metadata.

The audio playback system <NUM> may generate speaker feeds from the adapted audio data (<NUM>). The speaker feeds may be configured for use with headphones, loudspeakers or any other type of speaker. In some examples, the audio playback system <NUM> may apply the updated rendering matrix to the audio data to generate the speaker feeds. In some examples the audio playback system <NUM> may apply a binaural render to adapted higher order ambisonic audio data to generate the speaker feeds.

In some examples, the audio playback system <NUM> may output the speaker feeds (<NUM>) to speakers. In some examples the audio playback system may reproduce a soundfield (<NUM>) on one or more speakers, such as headphones or one or more loudspeakers.

<FIG> is a diagram showing how the systems of <FIG> and <FIG> may process audio data. In the example of <FIG>, the source device <NUM> encodes 6DOF content and transmits bitstream <NUM>, which includes the 6DOF content, to audio playback system <NUM>. In the example of <FIG>, the audio playback system <NUM> supports 6DOF content, and therefore, renders the 6DOF content and generates speaker feeds based on the 6DOF content. The user may consume the 6DOF content utilizing all 6DOF movements, meaning the user can freely move in a 3D space. Bitstream <NUM> includes audio and metadata to decode all possible user locations.

<FIG>, <FIG>, <FIG>, and <FIG> have described an example implementation of audio playback system <NUM> that supports 3DOF+ and 6DOF rendering. <FIG>, however, describe an example implementation of audio playback system <NUM> that either does not support 3DOF+ and 6DOF rendering or that has such functionality disabled.

<FIG> is a diagram illustrating how the systems of <FIG> and <FIG> may process audio data in accordance with the techniques of this disclosure. In the example of <FIG>, the source device <NUM> encodes 6DOF content and transmits bitstream <NUM>, which includes the 6DOF content, to audio playback system <NUM>. In the example of <FIG>, the audio playback system <NUM> only supports 3DOF content. The audio playback system <NUM>, therefore, adapts the 6DOF content to 3DOF content and generates speaker feeds based on the 3DOF content. The user may consume the 3DOF content utilizing 3DOF movements (e.g., changes to pitch, yaw, and roll), meaning the user cannot move translationally but can move rotationally.

In the example of <FIG>, bitstream <NUM> includes audio and metadata to decode all possible user locations. The bitstream <NUM> also includes adaptation metadata that includes user location information. The audio playback system <NUM>, which is being assumed not to support 6DOF content in the example of <FIG>, uses the user location information to adapt the 6DOF content to provide 3DOF and generate speaker feeds based on the adapted audio data. When processed by a device that supports 6DOF, the device can ignore the adaptation metadata that includes user location information, meaning such information may not be needed by a device that supports 6DOF.

<FIG> is a diagram illustrating how the systems of <FIG> and <FIG> may process audio data in accordance with the techniques of this disclosure. In the example of <FIG>, the source device <NUM> encodes 6DOF content and transmits bitstream <NUM>, which includes the 6DOF content, to audio playback system <NUM>. In the example of <FIG>, the audio playback system <NUM> only supports 3DOF content. The audio playback system <NUM>, therefore, adapts the 6DOF content to 3DOF content and generates speaker feeds based on the adapted 3DOF content. The user may consume the 3DOF content utilizing 3DOF movements (e.g., changes to pitch, yaw, and roll), meaning the user cannot move translationally but can move rotationally.

In the example of <FIG>, bitstream <NUM> includes audio and metadata to decode all possible user locations. The bitstream <NUM> also includes adaptation metadata that includes user location information for multiple locations. The audio playback system <NUM>, which is being assumed not to support 6DOF content in the example of <FIG>, uses user location information from one of the multiple locations to adapt the 6DOF content to provide 3DOF content. A user of audio playback system <NUM> may select one of the multiple locations, and audio playback system <NUM> may adapt the 6DOF content to 3DOF content based on the user location information of the selected location. When processed by a device that supports 6DOF, the device can ignore the adaptation metadata that includes the user location information for the multiple locations, meaning such information may not be needed by a device that supports 6DOF.

In the example of <FIG>, a user of audio playback system <NUM> may select a location from a plurality of locations, and audio playback system <NUM> may transmit the user selection to source device <NUM>. Based on the user selection, source device <NUM> may generate bitstream <NUM> such that bitstream <NUM> does not include audio and metadata for decoding all possible user locations. In the example of <FIG>, bitstream <NUM> may, for example, include audio and metadata to decode the user location selected by the user. The bitstream <NUM> also includes adaptation metadata that includes user location information for the selected user location. The audio playback system <NUM>, which is being assumed not to support 6DOF content in the example of <FIG>, uses the user location information to adapt the 6DOF content to provide 3DOF and generates speaker feeds based on the adapted audio data.

<FIG> is a diagram illustrating how the systems of <FIG> and <FIG> may process audio data in accordance with the techniques of this disclosure. In the example of <FIG>, the source device <NUM> encodes 6DOF content and transmits bitstream <NUM>, which includes the 6DOF content, to audio playback system <NUM>. In the example of <FIG>, the audio playback system <NUM> only supports ODOF content. The audio playback system <NUM>, therefore, adapts the 6DOF content to ODOF content and generates speaker feeds based on the ODOF content. The user may consume the ODOF content without using any of the 6DOF movements, meaning the user cannot move either rotationally or translationally.

In the example of <FIG>, bitstream <NUM> includes audio and metadata to decode all possible user locations. The bitstream <NUM> also includes adaptation metadata that includes user location information and user orientation information. The audio playback system <NUM>, which is being assumed not to support 6DOF content in the example of <FIG>, uses the user location information to adapt the 6DOF content to provide ODOF and generate speaker feeds based on the adapted audio data. When processed by a device that supports 6DOF, the device can ignore the adaptation metadata that includes the user location information and the user orientation information, meaning such information may not be needed by a device that supports 6DOF. When processed by a device that supports 3DOF, the device can adapt the 6DOF content to 3DOF based on the user location but ignore the user orientation information.

<FIG> is a diagram illustrating an example of a wearable device <NUM> that may operate in accordance with various aspect of the techniques described in this disclosure. In various examples, the wearable device <NUM> may represent a VR headset (such as the VR headset <NUM> described above), an AR headset, an MR headset, or an extended reality (XR) headset. Augmented Reality "AR" may refer to computer rendered image or data that is overlaid over the real world where the user is actually located. Mixed Reality "MR" may refer to computer rendered image or data that is world locked to a particular location in the real world, or may refer to a variant on VR in which part computer rendered 3D elements and part photographed real elements are combined into an immersive experience that simulates the user's physical presence in the environment. Extended Reality "XR" may refer to a catchall term for VR, AR, and MR. More information regarding terminology for XR can be found in a document by Jason Peterson, entitled "Virtual Reality, Augmented Reality, and Mixed Reality Definitions," and dated July <NUM>, <NUM>.

The wearable device <NUM> may represent other types of devices, such as a watch (including so-called "smart watches"), glasses (including so-called "smart glasses"), headphones (including so-called "wireless headphones" and "smart headphones"), smart clothing, smart jewelry, and the like. Whether representative of a VR device, a watch, glasses, and/or headphones, the wearable device <NUM> may communicate with the computing device supporting the wearable device <NUM> via a wired connection or a wireless connection.

In some instances, the computing device supporting the wearable device <NUM> may be integrated within the wearable device <NUM> and as such, the wearable device <NUM> may be considered as the same device as the computing device supporting the wearable device <NUM>. In other instances, the wearable device <NUM> may communicate with a separate computing device that may support the wearable device <NUM>. In this respect, the term "supporting" should not be understood to require a separate dedicated device but that one or more processors configured to perform various aspects of the techniques described in this disclosure may be integrated within the wearable device <NUM> or integrated within a computing device separate from the wearable device <NUM>.

For example, when the wearable device <NUM> represents the VR device <NUM>, a separate dedicated computing device (such as a personal computer including the one or more processors) may render the audio and visual content, while the wearable device <NUM> may determine the translational head movement upon which the dedicated computing device may render, based on the translational head movement, the audio content (as the speaker feeds) in accordance with various aspects of the techniques described in this disclosure. As another example, when the wearable device <NUM> represents smart glasses, the wearable device <NUM> may include the one or more processors that both determine the translational head movement (by interfacing within one or more sensors of the wearable device <NUM>) and render, based on the determined translational head movement, the speaker feeds.

As shown, the wearable device <NUM> includes a rear camera, one or more directional speakers, one or more tracking and/or recording cameras, and one or more light-emitting diode (LED) lights. In some examples, the LED light(s) may be referred to as "ultra bright" LED light(s). In addition, the wearable device <NUM> includes one or more eye-tracking cameras, high sensitivity audio microphones, and optics/projection hardware. The optics/projection hardware of the wearable device <NUM> may include durable semi-transparent display technology and hardware.

The wearable device <NUM> also includes connectivity hardware, which may represent one or more network interfaces that support multimode connectivity, such as <NUM> communications, <NUM> communications, etc. The wearable device <NUM> also includes ambient light sensors, and bone conduction transducers. In some instances, the wearable device <NUM> may also include one or more passive and/or active cameras with fisheye lenses and/or telephoto lenses. Various devices of this disclosure, such as the content consumer device <NUM> of <FIG> may use the steering angle of the wearable device <NUM> to select an audio representation of a soundfield (e.g., one of MOA representations) to output via the directional speaker(s) - headphones <NUM> - of the wearable device <NUM>, in accordance with various techniques of this disclosure. It will be appreciated that the wearable device <NUM> may exhibit a variety of different form factors.

Furthermore, the tracking and recording cameras and other sensors may facilitate the determination of translational distance <NUM>. Although not shown in the example of <FIG>, wearable device <NUM> may include the above discussed MEMS or other types of sensors for detecting translational distance <NUM>.

Although described with respect to particular examples of wearable devices, such as the VR device <NUM> discussed above with respect to the examples of <FIG> and other devices set forth in the examples of <FIG> and <FIG>, a person of ordinary skill in the art would appreciate that descriptions related to <FIG> may apply to other examples of wearable devices. For example, other wearable devices, such as smart glasses, may include sensors by which to obtain translational head movements. As another example, other wearable devices, such as a smart watch, may include sensors by which to obtain translational movements. As such, the techniques described in this disclosure should not be limited to a particular type of wearable device, but any wearable device may be configured to perform the techniques described in this disclosure.

<FIG> and <FIG> are diagrams illustrating example systems that may perform various aspects of the techniques described in this disclosure. <FIG> illustrates an example in which the source device <NUM> further includes a camera <NUM>. The camera <NUM> may be configured to capture video data and provide the captured raw video data to the content capture device <NUM>. The content capture device <NUM> may provide the video data to another component of the source device <NUM>, for further processing into viewport-divided portions.

In the example of <FIG>, the content consumer device <NUM> also includes the wearable device <NUM>. It will be understood that, in various implementations, the wearable device <NUM> may be included in, or externally coupled to, the content consumer device <NUM>. As discussed above with respect to <FIG>, the wearable device <NUM> includes display hardware and speaker hardware for outputting video data (e.g., as associated with various viewports) and for rendering audio data.

<FIG> illustrates an example similar that illustrated by <FIG>, except that the audio renderers <NUM> shown in <FIG> are replaced with a binaural renderer <NUM> capable of performing binaural rendering using one or more HRTFs or the other functions capable of rendering to left and right speaker feeds <NUM>. The audio playback system <NUM> may output the left and right speaker feeds <NUM> to headphones <NUM>.

The headphones <NUM> may couple to the audio playback system <NUM> via a wired connection (such as a standard <NUM> audio jack, a universal system bus (USB) connection, an optical audio jack, or other forms of wired connection) or wirelessly (such as by way of a Bluetooth™ connection, a wireless network connection, and the like). The headphones <NUM> may recreate, based on the left and right speaker feeds <NUM>, the soundfield represented by the audio data. The headphones <NUM> may include a left headphone speaker and a right headphone speaker which are powered (or, in other words, driven) by the corresponding left and right speaker feeds <NUM>.

In some examples, the VR device (or the streaming device) may communicate, using a network interface coupled to a memory of the VR/streaming device, exchange messages to an external device, where the exchange messages are associated with the multiple available representations of the soundfield. In some examples, the VR device may receive, using an antenna coupled to the network interface, wireless signals including data packets, audio packets, video pacts, or transport protocol data associated with the multiple available representations of the soundfield. In some examples, one or more microphone arrays may capture the soundfield.

In some examples, the multiple available representations of the soundfield stored to the memory device may include a plurality of object-based representations of the soundfield, higher order ambisonic representations of the soundfield, mixed order ambisonic representations of the soundfield, a combination of object-based representations of the soundfield with higher order ambisonic representations of the soundfield, a combination of object-based representations of the soundfield with mixed order ambisonic representations of the soundfield, or a combination of mixed order representations of the soundfield with higher order ambisonic representations of the soundfield.

In some examples, one or more of the soundfield representations of the multiple available representations of the soundfield may include at least one high-resolution region and at least one lower-resolution region, and wherein the selected presentation based on the steering angle provides a greater spatial precision with respect to the at least one high-resolution region and a lesser spatial precision with respect to the lower-resolution region.

Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuitry.

Claim 1:
A wearable device (<NUM>) comprising:
a memory configured to store:
audio data representative of a soundfield captured at a plurality of capture locations,
metadata that enables the audio data to be rendered to support N degrees of freedom of a user wearing the device, the N degrees of freedom comprising yaw, pitch, roll, and translational movement of the user, where the translational movement is defined in either a two-dimensional spatial coordinate space or a three-dimensional spatial coordinate space, and
adaptation metadata that enables the audio data to be rendered to support M degrees of freedom of the user, wherein N is a first integer number and M is a second integer number that is different than the first integer number, wherein M is less than N;
a display device; and
one or more processors coupled to the memory, and configured to:
determine a user location by causing a display device to display a plurality of locations or a trajectory, and receive, from the user, an input indicative of one of the plurality of locations or indicative of a location on the trajectory;
store the user location to the memory as the adaptation metadata;
adapt, based on the adaptation metadata and the input received from the user, the audio data to provide the M degrees of freedom; and
generate speaker feeds based on the adapted audio data.