Interpolating audio streams

In general, various aspects of the techniques are described for interpolating audio streams. A device comprising a memory and a processor may be configured to perform the techniques. The memory may store the one or more audio streams. The processor may obtain one or more microphone locations, each of the one or more microphone locations identifying a location of a respective one or more microphones that captured each of the corresponding one or more audio streams. The processor may also obtain a listener location identifying a location of a listener, and perform interpolation, based on the one or more microphone locations and the listener location, with respect to the audio streams to obtain an interpolated audio stream. The processor may next obtain, based on the interpolated audio stream, one or more speaker feeds, and output the one or more speaker feeds.

TECHNICAL FIELD

This disclosure relates to processing of audio data.

BACKGROUND

Computer-mediated reality systems are being developed to allow computing devices to augment or add to, remove or subtract from, or generally modify existing reality experienced by a user. Computer-mediated reality systems (which may also be referred to as “extended reality systems,” or “XR systems”) may include, as examples, virtual reality (VR) systems, augmented reality (AR) systems, and mixed reality (MR) systems. The perceived success of computer-mediated reality systems are generally related to the ability of such computer-mediated reality systems to provide a realistically immersive experience in terms of both the video and audio experience where the video and audio experience align in ways expected by the user. Although the human visual system is more sensitive than the human auditory systems (e.g., in terms of perceived localization of various objects within the scene), ensuring an adequate auditory experience is an increasingly import factor in ensuring a realistically immersive experience, particularly as the video experience improves to permit better localization of video objects that enable the user to better identify sources of audio content.

SUMMARY

This disclosure generally relates to techniques for interpolating an audio stream from one or more existing audio streams. The techniques may improve the listener experience, while also reducing soundfield reproduction localization errors, as the interpolated audio stream may better reflect a location of a listener relative to the existing audio streams, thereby improving the operation of a playback device (that performs the techniques to reproduce the soundfield) itself.

In one example, the techniques are directed to a device configured to process one or more audio streams, the device comprising: a memory configured to store the one or more audio streams; and a processor coupled to the memory, and configured to: obtain one or more microphone locations, each of the one or more microphone locations identifying a location of a respective one or more microphones that captured each of the corresponding one or more audio streams; obtain a listener location identifying a location of a listener; perform interpolation, based on the one or more microphone locations and the listener location, with respect to the audio streams to obtain an interpolated audio stream; obtain, based on the interpolated audio stream, one or more speaker feeds; and output the one or more speaker feeds.

In another example, the techniques are directed to a method for processing one or more audio streams, the method comprising: obtaining one or more microphone locations, each of the one or more microphone locations identifying a location of a respective one or more microphones that captured each of the corresponding one or more audio streams; obtaining a listener location identifying a location of a listener; performing interpolation, based on the one or more microphone locations and the listener location, with respect to the audio streams to obtain an interpolated audio stream; obtaining, based on the interpolated audio stream, one or more speaker feeds; and outputting the one or more speaker feeds.

In another example, the techniques are directed to a device configured to process one or more audio streams, the device comprising: means for obtaining one or more microphone locations, each of the one or more microphone locations identifying a location of a respective one or more microphones that captured each of the corresponding one or more audio streams; means for obtaining a listener location identifying a location of a listener; means for performing interpolation, based on the one or more microphone locations and the listener location, with respect to the audio streams to obtain an interpolated audio stream; means for obtaining, based on the interpolated audio stream, one or more speaker feeds; and means for outputting the one or more speaker feeds.

In another example, the techniques are directed to a non-transitory computer-readable storage medium having stored thereon instructions that, when executed, cause one or more processors to: obtain one or more microphone locations, each of the one or more microphone locations identifying a location of a respective one or more microphones that captured each of the corresponding one or more audio streams; obtain a listener location identifying a location of a listener; perform interpolation, based on the one or more microphone locations and the listener location, with respect to the audio streams to obtain an interpolated audio stream; obtain, based on the interpolated audio stream, one or more speaker feeds; and output the one or more speaker feeds.

The details of one or more examples of this disclosure are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of various aspects of the techniques will be apparent from the description and drawings, and from the claims.

DETAILED DESCRIPTION

There are a number of different ways to represent a soundfield. Example formats include channel-based audio formats, object-based audio formats, and scene-based audio formats. Channel-based audio formats refer to the 5.1 surround sound format, 7.1 surround sound formats, 22.2 surround sound formats, or any other channel-based format that localizes audio channels to particular locations around the listener in order to recreate a soundfield.

Object-based audio formats may refer to formats in which audio objects, often encoded using pulse-code modulation (PCM) and referred to as PCM audio objects, are specified in order to represent the soundfield. Such audio objects may include metadata identifying a location of the audio object relative to a listener or other point of reference in the soundfield, such that the audio object may be rendered to one or more speaker channels for playback in an effort to recreate the soundfield. The techniques described in this disclosure may apply to any of the foregoing formats, including scene-based audio formats, channel-based audio formats, object-based audio formats, or any combination thereof.

Scene-based audio formats may include a hierarchical set of elements that define the soundfield in three dimensions. One example of a hierarchical set of elements is a set of spherical harmonic coefficients (SHC). The following expression demonstrates a description or representation of a soundfield using SHC:

The expression shows that the pressure piat any point {rr, θr, φr} of the soundfield, at time t, can be represented uniquely by the SHC, Anm(k). Here,

k=ωc,
c is the speed of sound (˜343 m/s), {rr, θr, φr,} is a point of reference (or observation point), jn(·) is the spherical Bessel function of order n, and YNm(θr,φr) are the spherical harmonic basis functions (which may also be referred to as a spherical basis function) of order n and suborder m. It can be recognized that the term in square brackets is a frequency-domain representation of the signal (i.e., S(ω, rr, θr, φr)) which can be approximated by various time-frequency transformations, such as the discrete Fourier transform (DFT), the discrete cosine transform (DCT), or a wavelet transform. Other examples of hierarchical sets include sets of wavelet transform coefficients and other sets of coefficients of multiresolution basis functions.

The SHC Anm(k) can either be physically acquired (e.g., recorded) by various microphone array configurations or, alternatively, they can be derived from channel-based or object-based descriptions of the soundfield. The SHC (which also may be referred to as ambisonic coefficients) represent scene-based audio, where the SHC may be input to an audio encoder to obtain encoded SHC that may promote more efficient transmission or storage. For example, a fourth-order representation involving (1+4)2(25, and hence fourth order) coefficients may be used.

As noted above, the SHC may be derived from a microphone recording using a microphone array. Various examples of how SHC may be physically acquired from microphone arrays are described in Poletti, M., “Three-Dimensional Surround Sound Systems Based on Spherical Harmonics,” J. Audio Eng. Soc., Vol. 53, No. 11, 2005 November, pp. 1004-1025.

The following equation may illustrate how the SHCs may be derived from an object-based description. The coefficients Amm(k) for the soundfield corresponding to an individual audio object may be expressed as:
Anm(k)=g(ω)(−4πik)hn(2)(krs)Ynm*(θs,φs),
where i is √{square root over (−1)}hn(2)(·) is the spherical Hankel function (of the second kind) of order n, and {rs, θs, φs} is the location of the object. Knowing the object source energy g (ω) as a function of frequency (e.g., using time-frequency analysis techniques, such as performing a fast Fourier transform on the pulse code modulated—PCM—stream) may enable conversion of each PCM object and the corresponding location into the SHC Anm(k). Further, it can be shown (since the above is a linear and orthogonal decomposition) that the Anm(k) coefficients for each object are additive. In this manner, a number of PCM objects can be represented by the Anm(k) coefficients (e.g., as a sum of the coefficient vectors for the individual objects). The coefficients may contain information about the soundfield (the pressure as a function of 3D coordinates), and the above represents the transformation from individual objects to a representation of the overall soundfield, in the vicinity of the observation point {rr, θr, φr}.

Computer-mediated reality systems (which may also be referred to as “extended reality systems,” or “XR systems”) are being developed to take advantage of many of the potential benefits provided by ambisonic coefficients. For example, ambisonic coefficients may represent a soundfield in three dimensions in a manner that potentially enables accurate three-dimensional (3D) localization of sound sources within the soundfield. As such, XR devices may render the ambisonic coefficients to speaker feeds that, when played via one or more speakers, accurately reproduce the soundfield.

The use of ambisonic coefficients for XR may enable development of a number of use cases that rely on the more immersive soundfields provided by the ambisonic coefficients, particularly for computer gaming applications and live video streaming applications. In these highly dynamic use cases that rely on low latency reproduction of the soundfield, the XR devices may prefer ambisonic coefficients over other representations that are more difficult to manipulate or involve complex rendering. More information regarding these use cases is provided below with respect toFIGS. 1A and 1B.

While described in this disclosure with respect to the VR device, various aspects of the techniques may be performed in the context of other devices, such as a mobile device. In this instance, the mobile device (such as a so-called smartphone) may present the displayed world via a screen, which may be mounted to the head of the user102or viewed as would be done when normally using the mobile device. As such, any information on the screen can be part of the mobile device. The mobile device may be able to provide tracking information41and thereby allow for both a VR experience (when head mounted) and a normal experience to view the displayed world, where the normal experience may still allow the user to view the displayed world proving a VR-lite-type experience (e.g., holding up the device and rotating or translating the device to view different portions of the displayed world).

FIGS. 1A and 1Bare diagrams illustrating systems that may perform various aspects of the techniques described in this disclosure. As shown in the example ofFIG. 1A, system10includes a source device12and a content consumer device14. While described in the context of the source device12and the content consumer device14, the techniques may be implemented in any context in which any hierarchical representation of a soundfield is encoded to form a bitstream representative of the audio data. Moreover, the source device12may represent any form of computing device capable of generating hierarchical representation of a soundfield, and is generally described herein in the context of being a VR content creator device. Likewise, the content consumer device14may represent any form of computing device capable of implementing the audio stream interpolation techniques described in this disclosure as well as audio playback, and is generally described herein in the context of being a VR client device.

The source device12may be operated by an entertainment company or other entity that may generate multi-channel audio content for consumption by operators of content consumer devices, such as the content consumer device14. In many VR scenarios, the source device12generates audio content in conjunction with video content. The source device12includes a content capture device300and a content soundfield representation generator302.

The content capture device300may be configured to interface or otherwise communicate with one or more microphones5A-5N (“microphones5”). The microphones5may represent an Eigenmike® or other type of 3D audio microphone capable of capturing and representing the soundfield as corresponding scene-based audio data11A-11N (which may also be referred to as ambisonic coefficients11A-11N or “ambisonic coefficients11”). In the context of scene-based audio data11(which is another way to refer to the ambisonic coefficients11″), each of the microphones5may represent a cluster of microphones arranged within a single housing according to set geometries that facilitate generation of the ambisonic coefficients11. As such, the term microphone may refer to a cluster of microphones (which are actually geometrically arranged transducers) or a single microphone (which may be referred to as a spot microphone).

The ambisonic coefficients11may represent one example of an audio stream. As such, the ambisonic coefficients11may also be referred to as audio streams11. Although described primarily with respect to the ambisonic coefficients11, the techniques may be performed with respect to other types of audio streams, including pulse code modulated (PCM) audio streams, channel-based audio streams, object-based audio streams, etc.

The content capture device300may, in some examples, include an integrated microphone that is integrated into the housing of the content capture device300. The content capture device300may interface wirelessly or via a wired connection with the microphones5. Rather than capture, or in conjunction with capturing, audio data via the microphones5, the content capture device300may process the ambisonic coefficients11after the ambisonic coefficients11are input via some type of removable storage, wirelessly and/or via wired input processes. As such, various combinations of the content capture device300and the microphones5are possible.

The content capture device300may also be configured to interface or otherwise communicate with the soundfield representation generator302. The soundfield representation generator302may include any type of hardware device capable of interfacing with the content capture device300. The soundfield representation generator302may the use ambisonic coefficients11provided by the content capture device300to generate various representations of the same soundfield represented by the ambisonic coefficients11.

For instance, to generate the different representations of the soundfield using ambisonic coefficients (which again is one example of the audio data19), soundfield representation generator24may use a coding scheme for ambisonic representations of a soundfield, referred to as Mixed Order Ambisonics (MOA) as discussed in more detail in U.S. application Ser. No. 15/672,058, entitled “MIXED-ORDER AMBISONICS (MOA) AUDIO DATA FO COMPUTER-MEDIATED REALITY SYSTEMS,” filed Aug. 8, 2017, and published as U.S. patent publication no. 20190007781 on Jan. 3, 2019.

To generate a particular MOA representation of the soundfield, the soundfield representation generator24may generate a partial subset of the full set of ambisonic coefficients. For instance, each MOA representation generated by the soundfield representation generator24may provide precision with respect to some areas of the soundfield, but less precision in other areas. In one example, an MOA representation of the soundfield may include eight (8) uncompressed ambisonic coefficients, while the third order ambisonic representation of the same soundfield may include sixteen (16) uncompressed ambisonic coefficients. As such, each MOA representation of the soundfield that is generated as a partial subset of the ambisonic coefficients may be less storage-intensive and less bandwidth intensive (if and when transmitted as part of the bitstream27over the illustrated transmission channel) than the corresponding third order ambisonic representation of the same soundfield generated from the ambisonic coefficients.

Although described with respect to MOA representations, the techniques of this disclosure may also be performed with respect to first-order ambisonic (FOA) representations in which all of the ambisonic coefficients associated with a first order spherical basis function and a zero order spherical basis function are used to represent the soundfield. In other words, rather than represent the soundfield using a partial, non-zero subset of the ambisonic coefficients, the soundfield representation generator302may represent the soundfield using all of the ambisonic coefficients for a given order N, resulting in a total of ambisonic coefficients equaling (N+1)2.

In this respect, the ambisonic audio data (which is another way to refer to the ambisonic coefficients in either MOA representations or full order representation, such as the first-order representation noted above) may include ambisonic coefficients associated with spherical basis functions having an order of one or less (which may be referred to as “1storder ambisonic audio data”), ambisonic coefficients associated with spherical basis functions having a mixed order and suborder (which may be referred to as the “MOA representation” discussed above), or ambisonic coefficients associated with spherical basis functions having an order greater than one (which is referred to above as the “full order representation”).

The content capture device300may, in some examples, be configured to wirelessly communicate with the soundfield representation generator302. In some examples, the content capture device300may communicate, via one or both of a wireless connection or a wired connection, with the soundfield representation generator302. Via the connection between the content capture device300and the soundfield representation generator302, the content capture device300may provide content in various forms of content, which, for purposes of discussion, are described herein as being portions of the HOA coefficients11.

In some examples, the content capture device300may leverage various aspects of the soundfield representation generator302(in terms of hardware or software capabilities of the soundfield representation generator302). For example, the soundfield representation generator302may include dedicated hardware configured to (or specialized software that when executed causes one or more processors to) perform psychoacoustic audio encoding (such as a unified speech and audio coder denoted as “USAC” set forth by the Moving Picture Experts Group (MPEG), the MPEG-H 3D audio coding standard, the MPEG-I Immersive Audio standard, or proprietary standards, such as AptX™ (including various versions of AptX such as enhanced AptX—E-AptX, AptX live, AptX stereo, and AptX high definition—AptX-HD), advanced audio coding (AAC), Audio Codec 3 (AC-3), Apple Lossless Audio Codec (ALAC), MPEG-4 Audio Lossless Streaming (ALS), enhanced AC-3, Free Lossless Audio Codec (FLAC), Monkey's Audio, MPEG-1 Audio Layer II (MP2), MPEG-1 Audio Layer III (MP3), Opus, and Windows Media Audio (WMA).

The content capture device300may not include the psychoacoustic audio encoder dedicated hardware or specialized software and instead provide audio aspects of the content301in a non-psychoacoustic-audio-coded form. The soundfield representation generator302may assist in the capture of content301by, at least in part, performing psychoacoustic audio encoding with respect to the audio aspects of the content301.

The soundfield representation generator302may also assist in content capture and transmission by generating one or more bitstreams21based, at least in part, on the audio content (e.g., MOA representations and/or third order HOA representations) generated from the HOA coefficients11. The bitstream21may represent a compressed version of the HOA coefficients11(and/or the partial subsets thereof used to form MOA representations of the soundfield) and any other different types of the content301(such as a compressed version of spherical video data, image data, or text data).

The soundfield representation generator302may generate the bitstream21for transmission, as one example, across a transmission channel, which may be a wired or wireless channel, a data storage device, or the like. The bitstream21may represent an encoded version of the HOA coefficients11(and/or the partial subsets thereof used to form MOA representations of the soundfield) and may include a primary bitstream and another side bitstream, which may be referred to as side channel information. In some instances, the bitstream21representing the compressed version of the HOA coefficients may conform to bitstreams produced in accordance with the MPEG-H 3D audio coding standard.

The content consumer device14may be operated by an individual, and may represent a VR client device. Although described with respect to a VR client device, content consumer device14may represent other types of devices, such as an augmented reality (AR) client device, a mixed reality (MR) client device (or any other type of head-mounted display device), a standard computer, a headset, headphones, or any other device capable of tracking head movements and/or general translational movements of the individual operating the client consumer device14. As shown in the example ofFIG. 1A, the content consumer device14includes an audio playback system16A, which may refer to any form of audio playback system capable of rendering ambisonic coefficients (whether in form of first order, second order, and/or third order ambisonic representations and/or MOA representations) for playback as multi-channel audio content.

The content consumer device14may retrieve the bitstream21directly from the source device12. In some examples, the content consumer device12may interface with a network, including a fifth generation (5G) cellular network, to retrieve the bitstream21or otherwise cause the source device12to transmit the bitstream21to the content consumer device14.

While shown inFIG. 1Aas being directly transmitted to the content consumer device14, the source device12may output the bitstream21to an intermediate device positioned between the source device12and the content consumer device14. The intermediate device may store the bitstream21for later delivery to the content consumer device14, which may request the bitstream. The intermediate device may comprise a file server, a web server, a desktop computer, a laptop computer, a tablet computer, a mobile phone, a smart phone, or any other device capable of storing the bitstream21for later retrieval by an audio decoder. The intermediate device may reside in a content delivery network capable of streaming the bitstream21(and possibly in conjunction with transmitting a corresponding video data bitstream) to subscribers, such as the content consumer device14, requesting the bitstream21.

Alternatively, the source device12may store the bitstream21to a storage medium, such as a compact disc, a digital video disc, a high definition video disc or other storage media, most of which are capable of being read by a computer and therefore may be referred to as computer-readable storage media or non-transitory computer-readable storage media. In this context, the transmission channel may refer to the channels by which content stored to the mediums are transmitted (and may include retail stores and other store-based delivery mechanism). In any event, the techniques of this disclosure should not therefore be limited in this respect to the example ofFIG. 1A.

As noted above, the content consumer device14includes the audio playback system16. The audio playback system16may represent any system capable of playing back multi-channel audio data. The audio playback system16A may include a number of different audio renderers22. The renderers22may each provide for a different form of audio rendering, where the different forms of rendering may include one or more of the various ways of performing vector-base amplitude panning (VBAP), and/or one or more of the various ways of performing soundfield synthesis. As used herein, “A and/or B” means “A or B”, or both “A and B”.

The audio playback system16A may further include an audio decoding device24. The audio decoding device24may represent a device configured to decode bitstream21to output reconstructed HOA coefficients11A′-11N′ (which may form the full first, second, and/or third order ambisonic representation or a subset thereof that forms an MOA representation of the same soundfield or decompositions thereof, such as the predominant audio signal, ambient ambisonic coefficients, and the vector based signal described in the MPEG-H 3D Audio Coding Standard and/or the MPEG-I Immersive Audio standard).

As such, the ambisonic coefficients11A′-11N′ (“ambisonic coefficients11′”) may be similar to a full set or a partial subset of the ambisonic coefficients11, but may differ due to lossy operations (e.g., quantization) and/or transmission via the transmission channel. The audio playback system 16 may, after decoding the bitstream21to obtain the ambisonic coefficients11′, obtain ambisonic audio data15from the different streams of ambisonic coefficients11′, and render the ambisonic audio data15to output speaker feeds25. The speaker feeds25may drive one or more speakers (which are not shown in the example ofFIG. 1Afor ease of illustration purposes). Ambisonic representations of a soundfield may be normalized in a number of ways, including N3D, SN3D, FuMa, N2D, or SN2D.

To select the appropriate renderer or, in some instances, generate an appropriate renderer, the audio playback system16A may obtain loudspeaker information13indicative of a number of loudspeakers and/or a spatial geometry of the loudspeakers. In some instances, the audio playback system16A may obtain the loudspeaker information13using a reference microphone and outputting a signal to activate (or, in other words, drive) the loudspeakers in such a manner as to dynamically determine, via the reference microphone, the loudspeaker information13. In other instances, or in conjunction with the dynamic determination of the loudspeaker information13, the audio playback system16A may prompt a user to interface with the audio playback system16A and input the loudspeaker information13.

The audio playback system16A may select one of the audio renderers22based on the loudspeaker information13. In some instances, the audio playback system16A may, when none of the audio renderers22are within some threshold similarity measure (in terms of the loudspeaker geometry) to the loudspeaker geometry specified in the loudspeaker information13, generate the one of audio renderers22based on the loudspeaker information13. The audio playback system16A may, in some instances, generate one of the audio renderers22based on the loudspeaker information13without first attempting to select an existing one of the audio renderers22.

When outputting the speaker feeds25to headphones, the audio playback system16A may utilize one of the renderers22that provides for binaural rendering using head-related transfer functions (HRTF) or other functions capable of rendering to left and right speaker feeds25for headphone speaker playback. The terms “speakers” or “transducer” may generally refer to any speaker, including loudspeakers, headphone speakers, etc. One or more speakers may then playback the rendered speaker feeds25.

Although described as rendering the speaker feeds25from the ambisonic audio data15, reference to rendering of the speaker feeds25may refer to other types of rendering, such as rendering incorporated directly into the decoding of the ambisonic audio data15from the bitstream21. An example of the alternative rendering can be found in Annex G of the MPEG-H 3D audio coding standard, where rendering occurs during the predominant signal formulation and the background signal formation prior to composition of the soundfield. As such, reference to rendering of the ambisonic audio data15should be understood to refer to both rendering of the actual ambisonic audio data15or decompositions or representations thereof of the ambisonic audio data15(such as the above noted predominant audio signal, the ambient ambisonic coefficients, and/or the vector-based signal—which may also be referred to as a V-vector).

As described above, the content consumer device14may represent a VR device in which a human wearable display is mounted in front of the eyes of the user operating the VR device.FIGS. 5A and 5Bare diagrams illustrating examples of VR devices400A and400B. In the example ofFIG. 5A, the VR device400A is coupled to, or otherwise includes, headphones404, which may reproduce a soundfield represented by the ambisonic audio data15(which is another way to refer to ambisonic coefficients15) through playback of the speaker feeds25. The speaker feeds25may represent an analog or digital signal capable of causing a membrane within the transducers of headphones404to vibrate at various frequencies. Such a process is commonly referred to as driving the headphones404.

Video, audio, and other sensory data may play important roles in the VR experience. To participate in a VR experience, a user402may wear the VR device400A (which may also be referred to as a VR headset400A) or other wearable electronic device. The VR client device (such as the VR headset400A) may track head movement of the user402, and adapt the video data shown via the VR headset400A to account for the head movements, providing an immersive experience in which the user402may experience a virtual world shown in the video data in visual three dimensions.

While VR (and other forms of AR and/or MR, which may generally be referred to as a computer mediated reality device) may allow the user402to reside in the virtual world visually, often the VR headset400A may lack the capability to place the user in the virtual world audibly. In other words, the VR system (which may include a computer responsible for rendering the video data and audio data—that is not shown in the example ofFIG. 5Afor ease of illustration purposes, and the VR headset400A) may be unable to support full three dimension immersion audibly.

FIG. 5Bis a diagram illustrating an example of a wearable device400B that may operate in accordance with various aspect of the techniques described in this disclosure. In various examples, the wearable device400B may represent a VR headset (such as the VR headset400A described above), an AR headset, an MR headset, or any other type of extended reality (XR) headset. Augmented Reality “AR” may refer to computer rendered image or data that is overlaid over the real world where the user is actually located. Mixed Reality “MR” may refer to computer rendered image or data that is world locked to a particular location in the real world, or may refer to a variant on VR in which part computer rendered 3D elements and part photographed real elements are combined into an immersive experience that simulates the user's physical presence in the environment. Extended Reality “XR” may represent a catchall term for VR, AR, and MR. More information regarding terminology for XR can be found in a document by Jason Peterson, entitled “Virtual Reality, Augmented Reality, and Mixed Reality Definitions,” and dated Jul. 7, 2017.

The wearable device400B may represent other types of devices, such as a watch (including so-called “smart watches”), glasses (including so-called “smart glasses”), headphones (including so-called “wireless headphones” and “smart headphones”), smart clothing, smart jewelry, and the like. Whether representative of a VR device, a watch, glasses, and/or headphones, the wearable device400B may communicate with the computing device supporting the wearable device400B via a wired connection or a wireless connection.

In some instances, the computing device supporting the wearable device400B may be integrated within the wearable device400B and as such, the wearable device400B may be considered as the same device as the computing device supporting the wearable device400B. In other instances, the wearable device400B may communicate with a separate computing device that may support the wearable device400B. In this respect, the term “supporting” should not be understood to require a separate dedicated device but that one or more processors configured to perform various aspects of the techniques described in this disclosure may be integrated within the wearable device400B or integrated within a computing device separate from the wearable device400B.

For example, when the wearable device400B represents an example of the VR device400B, a separate dedicated computing device (such as a personal computer including the one or more processors) may render the audio and visual content, while the wearable device400B may determine the translational head movement upon which the dedicated computing device may render, based on the translational head movement, the audio content (as the speaker feeds) in accordance with various aspects of the techniques described in this disclosure. As another example, when the wearable device400B represents smart glasses, the wearable device400B may include the one or more processors that both determine the translational head movement (by interfacing within one or more sensors of the wearable device400B) and render, based on the determined translational head movement, the speaker feeds.

As shown, the wearable device400B includes one or more directional speakers, and one or more tracking and/or recording cameras. In addition, the wearable device400B includes one or more inertial, haptic, and/or health sensors, one or more eye-tracking cameras, one or more high sensitivity audio microphones, and optics/projection hardware. The optics/projection hardware of the wearable device400B may include durable semi-transparent display technology and hardware.

The wearable device400B also includes connectivity hardware, which may represent one or more network interfaces that support multimode connectivity, such as 4G communications, 5G communications, Bluetooth, etc. The wearable device400B also includes one or more ambient light sensors, and bone conduction transducers. In some instances, the wearable device400B may also include one or more passive and/or active cameras with fisheye lenses and/or telephoto lenses. Although not shown inFIG. 5B, the wearable device400B also may include one or more light emitting diode (LED) lights. In some examples, the LED light(s) may be referred to as “ultra bright” LED light(s). The wearable device400B also may include one or more rear cameras in some implementations. It will be appreciated that the wearable device400B may exhibit a variety of different form factors.

Furthermore, the tracking and recording cameras and other sensors may facilitate the determination of translational distance. Although not shown in the example ofFIG. 5B, wearable device400B may include other types of sensors for detecting translational distance.

Although described with respect to particular examples of wearable devices, such as the VR device400B discussed above with respect to the examples ofFIG. 5Band other devices set forth in the examples ofFIGS. 1A and 1B, a person of ordinary skill in the art would appreciate that descriptions related toFIGS. 1A-4Bmay apply to other examples of wearable devices. For example, other wearable devices, such as smart glasses, may include sensors by which to obtain translational head movements. As another example, other wearable devices, such as a smart watch, may include sensors by which to obtain translational movements. As such, the techniques described in this disclosure should not be limited to a particular type of wearable device, but any wearable device may be configured to perform the techniques described in this disclosure.

In any event, the audio aspects of VR have been classified into three separate categories of immersion. The first category provides the lowest level of immersion, and is referred to as three degrees of freedom (3DOF). 3DOF refers to audio rendering that accounts for movement of the head in the three degrees of freedom (yaw, pitch, and roll), thereby allowing the user to freely look around in any direction. 3DOF, however, cannot account for translational head movements in which the head is not centered on the optical and acoustical center of the soundfield.

The second category, referred to 3DOF plus (3DOF+), provides for the three degrees of freedom (yaw, pitch, and roll) in addition to limited spatial translational movements due to the head movements away from the optical center and acoustical center within the soundfield. 3DOF+ may provide support for perceptual effects such as motion parallax, which may strengthen the sense of immersion.

The third category, referred to as six degrees of freedom (6DOF), renders audio data in a manner that accounts for the three degrees of freedom in term of head movements (yaw, pitch, and roll) but also accounts for translation of the user in space (x, y, and z translations). The spatial translations may be induced by sensors tracking the location of the user in the physical world or by way of an input controller.

3DOF rendering is the current state of the art for audio aspects of VR. As such, the audio aspects of VR are less immersive than the video aspects, thereby potentially reducing the overall immersion experienced by the user, and introducing localization errors (e.g., such as when the auditory playback does not match or correlate exactly to the visual scene).

In accordance with the techniques described in this disclosure, various ways are described to perform interpolation with respect to the existing audio streams11and thereby allow for 6DOF immersion. As described below, the techniques may improve the listener experience, while also reducing soundfield reproduction localization errors, as the interpolated audio stream may better reflect a location of a listener relative to the existing audio streams, thereby improving the operation of a playback device (that performs the techniques to reproduce the soundfield) itself

In operation, the audio playback system16A may include an interpolation device30(“INT DEVICE30”), e.g., as shown inFIG. 1A, which may be configured to process the audio streams11′ to obtain an interpolated audio stream15(which is another way to refer to the ambisonic audio data15). Although shown as being a separate device, the interpolation device30may be integrated or otherwise incorporated within one of the audio decoding devices24.

The interpolation device may be implemented by one or more processors, including fixed function processing circuitry and/or programmable processing circuitry, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuitry.

The interpolation device30may first obtain one or more microphone locations, each of the one or more microphone locations identifying a location of a respective one or more microphones that captured the one or more audio streams11′. More information regarding operation of the interpolation device30is described with respect to the examples ofFIGS. 2-3B.

FIG. 2is a block diagram illustrating example operation of the interpolation device30ofFIGS. 1A and 1Bin performing various aspects of the audio stream interpolation techniques described in this disclosure. In the example ofFIG. 2, the interpolation device30receives the ambisonic audio streams11′ (shown as “ambisonic streams11′”), which were captured by microphones5(which may, as noted above, represent clusters or arrays of microphones). As noted above, the signals output by the microphones5may undergo a conversion from the microphone format to the HOA format, which is shown by the box labeled “MicAmbisonics,” resulting in the ambisonic audio streams11′.

The interpolation device30may also receive audio metadata511A-511N (“audio metadata511”), which may include a microphone location identifying a location of a corresponding microphone5A-5N that captured the corresponding one of the audio streams11′. The microphones5may provide the microphone location, an operator of the microphones5may enter the microphone locations, a device coupled to the microphone (e.g., the content capture device300) may specify the microphone location, or some combination of the foregoing. The content capture device300may specify the audio metadata511as part of the content301. In any event, the interpolation device30may parse the audio metadata511from the bitstream21representative of the content301.

The interpolation device30may also obtain a listener location17that identifies a location of a listener, such as that shown in the example ofFIG. 5A. The audio metadata may specify a location and an orientation of the microphone as shown in the example ofFIG. 2, or only a microphone location. Further, the listener location17may include a listener position (or, in other words, location) and an orientation, or only a listener location. Referring briefly back toFIG. 1A, the audio playback system16A may interface with a tracking device306to obtain the listener location17. The tracking device306may represent any device capable of tracking the listener, and may include one or more of a global positioning system (GPS) device, a camera, a sonar device, an ultrasonic device, an infrared emitting and receiving device, or any other type of device capable of obtaining the listener location17.

The interpolation device30may next perform interpolation, based on the one or more microphone locations and the listener location17, with respect to the audio streams11′ to obtain interpolated audio stream15. The audio streams11′ may be stored in a memory of the interpolation device30. To perform the interpolation, the interpolation device30may read the audio streams11′ form memory and determine, based on the one or more microphones locations and the listener location17(which may also be stored in the memory), a weight for each of the audio streams (which are shown as Weight(1) . . . Weight(n)).

To determine the weights, the interpolation device30may calculate each weight as a ratio of inverse distance to the listener location17for the corresponding one of the audio streams11′ by the total inverse distance from all of the other audio streams11′, except for the edge cases when the listener is at the same location as one of the microphones5as represented in the virtual world. That is to say, it may be possible for a listener to navigate a virtual world, or a real world location represented on a display of a device, which has the same location as where one of the microphones5captured the audio streams11′. When the listener is at the same location as one of the microphones5, the interpolation unit30may calculate the weight for the one of the audio streams11′ captured by the one of the microphones5at which the listener is at the same location as one of the microphones5, and the weights for the remaining audio streams11′ are set to zero.

Otherwise, the interpolation device30may calculate each weight as follows: Weight(n)=(1/(distance of mic n to the listener position))/(1/(distance of mic 1 to the listener position)+ . . . +1/(distance of mic n to the listener position)), In the above, the listener position refers to the listener position17, Weight(n) refers to the weight for the audio stream11N′, and the distance of mic <number> to the listener position refers to the absolute value of the difference between the corresponding microphone location and the listener position17.

The interpolation device30may next multiply the weight by the corresponding one of the audio streams11′ to obtain one or more weighted audio streams, which the interpolation device30may add together to obtain the interpolated audio stream15. The foregoing may be denoted mathematically by the following equation: Weight(1)*audio stream 1+ . . . +Weight(n)*audio stream n=Interpolated audio stream, where Weight(<number>) denotes the weight for the corresponding audio stream <number>, and the interpolated ambisonic audio data refers to the interpolated audio stream15. The interpolated audio stream may be stored in the memory of the interpolation device30and may also be available to be played out by loudspeakers (e.g., a VR or AR device or a headset worn by the listener). The interpolation equation represents the weighted average ambisonic audio shown in the example ofFIG. 2. It should be noted that it may be possible in some configuration to interpolate non-ambisonic audio streams; however, there may be a loss of audio quality or resolution if the interpolation is not performed on ambisonic audio data.

In some examples, the interpolation device30may determine the foregoing weights on a frame-by-frame basis. In other examples, the interpolation device30may determine the foregoing weights on a more frequent basis (e.g., some sub-frame basis) or on a more infrequent basis (e.g., after some set number of frames). In these and other examples, the interpolation device30may only calculate the weights responsive to detection of some change in the listener location and/or orientation or responsive to some other characteristics of the underlying ambisonic audio streams (which may enable and disable various aspects of the interpolation techniques described in this disclosure).

In some examples, the above techniques may only be enabled with respect to the audio streams11′ having certain characteristics. For example, the interpolation device30may only interpolate the audio streams11′ when audio sources represented by the audio streams11′ are located at locations different than the microphones5. More information regarding this aspect of the techniques is provided below with respect toFIGS. 4A and 4B.

FIG. 4Ais a diagram illustrating, in more detail, how the interpolation device ofFIGS. 1A-2may perform various aspects of the techniques described in this disclosure. As shown inFIG. 4A, the listener52may progress within the area54defined by the microphones (shown as “mic arrays”)5A-5E. In some examples, the microphones5(including when the microphones5represent clusters or, in other words, arrays of microphones) may be positioned at a distance from one another that is greater than five feet. In any event, the interpolation device30(referring toFIG. 2) may perform the interpolation when sound sources50A-50D (“sound sources50” or “audio sources50” as shown inFIG. 4A) are outside of the area54defined by the microphones5A-5E given mathematical constraints imposed by the equations discussed above.

Returning to the example ofFIG. 4A, the listener52may enter or otherwise issue one or more navigational commands (potentially by walking or through use of a controller or other interface device, including smart phones, etc.) to navigate within the area54(along the line56). A tracking device (such as the tracking device306shown in the exampleFIG. 2) may receive these navigational commands and generate the listener location17.

As the listener52starts navigating from the starting location, the interpolation device30may generate the interpolated audio stream15to heavily weight the audio stream11C′ captured by the microphone5C, and assign relatively less weight to the audio stream11B′ captured by the microphone5B and the audio stream11D′ captured by the microphone5D, and still relatively less weight (and possibly no weight) to the audio streams11A′ and11E′ captured by the respective microphones5A and5E.

As the listener52navigates along the line56next to the location of the microphone5B, the interpolation device30may assign more weight to the audio stream11B′, relatively less weight to the audio stream11C′ and yet less weight (and possibly no weight) to the audio streams11A′,11D′, and11E′. As the listener52navigates (where the notch indicates the direction in which the listener52is moving) closer to the location of the microphone5E toward the end of the line56, the interpolation device30may assign more weight to the audio stream11E′, relatively less weight to the audio stream11A′, and yet relatively less weight (and possibly no weight) to the audio streams11B′,11C′, and11D′.

In this respect, the interpolation device30may perform interpolation based on changes to the listener location17based on navigational commands issued by the listener32to assign varying weights over time to the audio streams11A′-11E′. The changing listener location17may result in different emphasis within the interpolated audio stream15, thereby promoting better auditory localization within the area54.

Although not described in the examples set forth above, the techniques may also adapt to changes in the location of the microphones. In other words, the microphones may be manipulated during recording, changing locations and orientations. Because the above noted equations are only concerned with differences between the microphone locations and the listener location17, the interpolation device30may continue to perform the interpolation even though the microphones have been manipulated to change location and/or orientation.

FIG. 4Bis a block diagram illustrating, in more detail, how the interpolation device ofFIGS. 1A-2may perform various aspects of the techniques described in this disclosure. The example shown inFIG. 4Bis similar to the example shown inFIG. 4A, except that the microphones5are replaced with wearable devices500A-500E (which may represent an example of wearable devices400A and/or400B). The wearable devices500A-500E may each include a microphone that captures the audio streams described in more detail above.

FIG. 1Bis a block diagram illustrating another example system100configured to perform various aspects of the techniques described in this disclosure. The system100is similar to the system10shown inFIG. 1A, except that the audio renderers22shown inFIG. 1Aare replaced with a binaural renderer102capable of performing binaural rendering using one or more HRTFs or the other functions capable of rendering to left and right speaker feeds103.

The audio playback system16B may output the left and right speaker feeds103to headphones104, which may represent another example of a wearable device and which may be coupled to additional wearable devices to facilitate reproduction of the soundfield, such as a watch, the VR headset noted above, smart glasses, smart clothing, smart rings, smart bracelets or any other types of smart jewelry (including smart necklaces), and the like. The headphones104may couple wirelessly or via wired connection to the additional wearable devices.

Additionally, the headphones104may couple to the audio playback system16via a wired connection (such as a standard 3.5 mm audio jack, a universal system bus (USB) connection, an optical audio jack, or other forms of wired connection) or wirelessly (such as by way of a Bluetooth™ connection, a wireless network connection, and the like). The headphones104may recreate, based on the left and right speaker feeds103, the soundfield represented by the ambisonic coefficients11. The headphones104may include a left headphone speaker and a right headphone speaker which are powered (or, in other words, driven) by the corresponding left and right speaker feeds103.

Although described with respect to a VR device as shown in the example ofFIGS. 7A and 7B, the techniques may be performed by other types of wearable devices, including watches (such as so-called “smart watches”), glasses (such as so-called “smart glasses”), headphones (including wireless headphones coupled via a wireless connection, or smart headphones coupled via wired or wireless connection), and any other type of wearable device. As such, the techniques may be performed by any type of wearable device by which a user may interact with the wearable device while worn by the user.

FIG. 3Ais a block diagram illustrating further example operation of the interpolation device ofFIGS. 1A and 1Bin performing various aspects of the audio stream interpolation techniques described in this disclosure. The interpolation device30A shown in the example ofFIG. 3Ais similar to that shown in the example ofFIG. 2, except that the interpolation device30shown inFIG. 2receives audio streams11′ that were not captured from a microphone (and that which were pre-captured and/or mixed). The interpolation device30shown in the example ofFIG. 2represents an example use during live capture (for live events, like sporting events, concerts, lectures, etc.), while the interpolation device30A shown in the example ofFIG. 3Arepresents an example use during pre-recorded or generated events (such as video games, movies, etc.). The interpolation device30A may include a memory for storing the audio streams as shown inFIG. 3A.

FIG. 3Bis a block diagram illustrating yet further example operation of the interpolation device ofFIGS. 1A and 1Bin performing various aspects of the audio stream interpolation techniques described in this disclosure. The example shown inFIG. 3Bis similar to the example shown inFIG. 3Aexcept that wearable devices500A-500N may capture audio streams11A-11N (which are compressed and decoded as audio streams11A′-11N′). The interpolation device3BA may include a memory for storing the audio streams as shown inFIG. 3B.

FIGS. 6A and 6Bare diagrams illustrating example systems that may perform various aspects of the techniques described in this disclosure.FIG. 6Aillustrates an example in which the source device12further includes a camera200. The camera200may be configured to capture video data, and provide the captured raw video data to the content capture device300. The content capture device300may provide the video data to another component of the source device12, for further processing into viewport-divided portions.

In the example ofFIG. 6A, the content consumer device14also includes the wearable device800. It will be understood that, in various implementations, the wearable device800may be included in, or externally coupled to, the content consumer device14. As discussed above with respect toFIGS. 5A and 5B, the wearable device800includes display hardware and speaker hardware for outputting video data (e.g., as associated with various viewports) and for rendering audio data.

FIG. 6Billustrates an example similar that illustrated byFIG. 6A, except that the audio renderers22shown inFIG. 6Aare replaced with a binaural renderer102capable of performing binaural rendering using one or more HRTFs or the other functions capable of rendering to left and right speaker feeds103. The audio playback system16may output the left and right speaker feeds103to headphones104.

The headphones104may couple to the audio playback system16via a wired connection (such as a standard 3.5 mm audio jack, a universal system bus (USB) connection, an optical audio jack, or other forms of wired connection) or wirelessly (such as by way of a Bluetooth™ connection, a wireless network connection, and the like). The headphones104may recreate, based on the left and right speaker feeds103, the soundfield represented by the ambisonic coefficients11. The headphones104may include a left headphone speaker and a right headphone speaker which are powered (or, in other words, driven) by the corresponding left and right speaker feeds103.

FIG. 7is a flowchart illustrating example operation of the audio playback system ofFIGS. 1A-6Bin performing various aspects of the audio interpolation techniques described in this disclosure. The interpolation device30of the audio playback system16may first obtain one or more microphone locations (950), each of the one or more microphone locations identifying a location of a respective one or more microphones that captured each of the corresponding one or more audio streams (in the virtual coordinate system). The interpolation device30may next obtain a listener location identifying a location of a listener (952).

The interpolation device30may, as described above in more detail, perform interpolation, based on the one or more microphone locations and the listener location, with respect to the audio streams to obtain an interpolated audio stream (954). The audio playback system16may next invoke the audio renderers22to obtain, based on the interpolated audio streams (e.g., ambisonic audio data15), one or more speaker feeds25(956). The audio playback system16may output the one or more speaker feeds25(958) to drive or otherwise power transducers (e.g., speakers).

FIG. 8is a block diagram of the audio playback device shown in the examples ofFIGS. 1A and 1Bin performing various aspects of the techniques described in this disclosure. The audio playback device16may represent an example of the audio playback device 16A and/or the audio playback device16B. The audio playback system16may include the audio decoding device24in combination with a 6DOF audio renderer22A, which may represent one example of the audio renderers22shown in the example ofFIGS. 1A.

The audio decoding device24may include a low delay decoder900A, an audio decoder900B, and a local audio buffer902. The low delay decoder900A may process XR audio bitstream21A to obtain audio stream901A, where the low delay decoder900A may perform relatively low complexity decoding (compared to the audio decoder900B) to facilitate low delay reconstruction of the audio stream901A. The audio decoder900B may perform relatively higher complexity decoding (compared to the audio decoder900A) with respect to the audio bitstream21B to obtain audio stream901B. The audio decoder900B may perform audio decoding that conforms to the MPEG-H 3D Audio coding standard. The local audio buffer902may represent a unit configured to buffer local audio content, which the local audio buffer902may output as audio stream903.

The bitstream21(comprised of one or more of the XR audio bitstream21A and/or the audio bitstream21B) may also include XR metadata905A (which may include the microphone location information noted above) and 6DOF metadata905B (which may specify various parameters related to 6DOF audio rendering). The 6DOF audio renderer22A may obtain the audio streams901A,901B, and/or903along with the XR metadata905A and the 6DOF metadata905B and render the speaker feeds25and/or103based on the listener positions and the microphone positions. In the example ofFIG. 8, the 6DOF audio renderer22A includes the interpolation device30, which may perform various aspects of the audio stream interpolation techniques described in more detail above to facilitate 6DOF audio rendering.

UEs115may be dispersed throughout the wireless communications system100, and each UE115may be stationary or mobile. A UE115may also be referred to as a mobile device, a wireless device, a remote device, a handheld device, or a subscriber device, or some other suitable terminology, where the “device” may also be referred to as a unit, a station, a terminal, or a client. A UE115may also be a personal electronic device such as a cellular phone, a personal digital assistant (PDA), a tablet computer, a laptop computer, or a personal computer. In examples of this disclosure, a UE115may be any of the audio sources described in this disclosure, including a VR headset, an XR headset, an AR headset, a vehicle, a smartphone, a microphone, an array of microphones, or any other device including a microphone or is able to transmit a captured and/or synthesized audio stream. In some examples, an synthesized audio stream may be an audio stream that was stored in memory or was previously created or synthesized. In some examples, a UE115may also refer to a wireless local loop (WLL) station, an Internet of Things (IoT) device, an Internet of Everything (IoE) device, or an MTC device, or the like, which may be implemented in various articles such as appliances, vehicles, meters, or the like.

Some UEs115, such as MTC or IoT devices, may be low cost or low complexity devices, and may provide for automated communication between machines (e.g., via Machine-to-Machine (M2M) communication). M2M communication or MTC may refer to data communication technologies that allow devices to communicate with one another or a base station105without human intervention. In some examples, M2M communication or MTC may include communications from devices that exchange and/or use audio metadata indicating privacy restrictions and/or password-based privacy data to toggle, mask, and/or null various audio streams and/or audio sources as will be described in more detail below.

In this respect, various aspects of the techniques are described that enable one or more of the following examples:

Example 1. A device configured to process one or more audio streams, the device comprising: a memory configured to store the one or more audio streams; and a processor coupled to the memory, and configured to: obtain one or more microphone locations, each of the one or more microphone locations identifying a location of a respective one or more microphones that captured each of the corresponding one or more audio streams; obtain a listener location identifying a location of a listener; perform interpolation, based on the one or more microphone locations and the listener location, with respect to the audio streams to obtain an interpolated audio stream; obtain, based on the interpolated audio stream, one or more speaker feeds; and output the one or more speaker feeds.

Example 2. The device of example 1, wherein the one or more processors are configured to: determine, based on the one or more microphone locations and the listener location, a weight for each of the audio streams; and obtain, based on the weight, the interpolated audio stream.

Example 3. The device of example 1, wherein the one or more processors are configured to: determine, based on the one or more microphone locations and the listener location, a weight for each of the audio streams; and multiply the weight by the corresponding one of the one or more audio streams to obtain one or more weighted audio stream; and obtain, based on the one or more weighted audio streams, the interpolated audio stream.

Example 4. The device of example 1, wherein the one or more processors are configured to: determine, based on the one or more microphone locations and the listener location, a weight for each of the audio streams; and multiply the weight by the corresponding one of the one or more audio streams to obtain one or more weighted audio stream; and add the one or more weighted audio streams together to obtain the interpolated audio stream.

Example 5. The device of any combination of examples 2-4, wherein the one or more processors are configured to: determine a difference between each of the one or more microphone locations and the listener location; and determine, based on the difference between each of the one or more microphone locations and the listener location, the weight for each of the audio streams.

Example 6. The device of any combination of examples 2-5, wherein the one or more processors are configured to determine the weights for each audio frame of the one or more audio streams.

Example 7. The device of any combination of examples 1-6, wherein audio sources represented by the audio streams reside outside of the one or more microphones.

Example 8. The device of any combination of examples 1-7, wherein the one or more processors are configured to obtain, from a computer mediated reality device, the listener location.

Example 9. The device of example 8, wherein the computer mediated reality device comprises a head mounted display device.

Example 10. The device of any combination of examples 1-9, wherein the one or more processors are configured to obtain, from a bitstream that includes the audio streams, audio metadata that identifies the one or more microphone locations.

Example 11. The device of any combination of examples 1-10, wherein at least one of the one or more microphone locations changes to reflect movement of the corresponding one of the one or more microphones.

Example 12. The device of any combination of examples 1-11, wherein the one or more audio streams include a ambisonic audio stream (including higher order, mixed order, first order, second order), and wherein the interpolated audio stream includes an interpolated ambisonic audio stream (including higher order, mixed order, first order, second order).

Example 13. The device of any combination of claims1-11, wherein the one or more audio streams include an ambisonic audio stream, and wherein the interpolated audio stream includes an interpolated ambisonic audio stream.

Example 14. The device of any combination of examples 1-13, wherein the listener location changes based on navigational commands issued by the listener.

Example 15. The device of any combination of examples 1-14, wherein the one or more processors are configured to receive audio metadata specifying the microphone locations, each of the microphone locations identifying a location of a cluster of microphones that captured the corresponding one or more audio streams.

Example 16. The device of any combination of examples 15, wherein the cluster of microphones are each positioned at a distance from one another that is greater than five feet.

Example 17. The device of any combination of examples 1-14, wherein the microphones are each positioned at a distance greater than five feet from one another.

Example 18. A method for processing one or more audio streams, the method comprising: obtaining one or more microphone locations, each of the one or more microphone locations identifying a location of a respective one or more microphones that captured each of the corresponding one or more audio streams; obtaining a listener location identifying a location of a listener; performing interpolation, based on the one or more microphone locations and the listener location, with respect to the audio streams to obtain an interpolated audio stream; obtaining, based on the interpolated audio stream, one or more speaker feeds; and outputting the one or more speaker feeds.

Example 19. The method of example 18, wherein performing the interpolation comprises: determining, based on the one or more microphone locations and the listener location, a weight for each of the audio streams; and obtaining, based on the weight, the interpolated audio stream.

Example 20. The method of example 18, wherein performing the interpolation comprises: determining, based on the one or more microphone locations and the listener location, a weight for each of the audio streams; multiplying the weight by the corresponding one of the one or more audio streams to obtain one or more weighted audio stream; and obtaining, based on the one or more weighted audio streams, the interpolated audio stream.

Example 21. The method of example 18, wherein performing the interpolation comprises: determining, based on the one or more microphone locations and the listener location, a weight for each of the audio streams; and multiplying the weight by the corresponding one of the one or more audio streams to obtain one or more weighted audio stream; and adding the one or more weighted audio streams together to obtain the interpolated audio stream.

Example 22. The method of any combination of example 19-21, wherein determining the weights comprises: determining a difference between each of the one or more microphone locations and the listener location; and determining, based on the difference between each of the one or more microphone locations and the listener location, the weight for each of the audio streams.

Example 23. The method of any combination of example 19-22, wherein determining the weights comprises determining the weights for each audio frame of the one or more audio streams.

Example 24. The method of any combination of examples 18-23, wherein audio sources represented by the audio streams reside outside of the one or more microphones.

Example 25. The method of any combination of examples 18-24, wherein obtaining the listener location comprises obtaining, from a computer mediated reality device, the listener location.

Example 26. The method of example 25, wherein the computer mediated reality device comprises a head mounted display device.

Example 27. The method of any combination of examples 18-26, wherein obtaining the one or more microphone locations comprises obtaining, from a bitstream that includes the audio streams, audio metadata that identifies the one or more microphone locations.

Example 28. The method of any combination of examples 18-27, wherein at least one of the one or more microphone locations changes to reflect movement of the corresponding one of the one or more microphones.

Example 29. The method of any combination of examples 18-28, wherein the one or more audio streams include a ambisonic audio stream (including higher order, mixed order, first order, second order), and wherein the interpolated audio stream includes an interpolated ambisonic audio stream (including higher order, mixed order, first order, second order).

Example 30. The method of any combination of examples 18-28, wherein the one or more audio streams include an ambisonic audio stream, and wherein the interpolated audio stream includes an interpolated ambisonic audio stream.

Example 31. The method of any combination of examples 18-30, wherein the listener location changes based on navigational commands issued by the listener.

Example 32. The method of any combination of examples 18-31, wherein obtaining the microphone locations comprises receiving audio metadata specifying the microphone locations, each of the microphone locations identifying a location of a cluster of microphones that captured the corresponding one or more audio streams.

Example 33. The method of example 32, wherein the cluster of microphones are each positioned at a distance from one another that is greater than five feet.

Example 34. The method of any combination of examples 18-31, wherein the microphones are each positioned at a distance greater than five feet from one another.

Example 35. A device configured to process one or more audio streams, the device comprising: means for obtaining one or more microphone locations, each of the one or more microphone locations identifying a location of a respective one or more microphones that captured each of the corresponding one or more audio streams; means for obtaining a listener location identifying a location of a listener; means for performing interpolation, based on the one or more microphone locations and the listener location, with respect to the audio streams to obtain an interpolated audio stream; means for obtaining, based on the interpolated audio stream, one or more speaker feeds; and means for outputting the one or more speaker feeds.

Example 36. The device of example 35, wherein the means for performing the interpolation comprises: means for determining, based on the one or more microphone locations and the listener location, a weight for each of the audio streams; and means for obtaining, based on the weight, the interpolated audio stream.

Example 37. The device of example 35, wherein the means for performing the interpolation comprises: means for determining, based on the one or more microphone locations and the listener location, a weight for each of the audio streams; means for multiplying the weight by the corresponding one of the one or more audio streams to obtain one or more weighted audio stream; and means for obtaining, based on the one or more weighted audio streams, the interpolated audio stream.

Example 38. The device of example 35, wherein the means for performing the interpolation comprises: means for determining, based on the one or more microphone locations and the listener location, a weight for each of the audio streams; means for multiplying the weight by the corresponding one of the one or more audio streams to obtain one or more weighted audio stream; and means for adding the one or more weighted audio streams together to obtain the interpolated audio stream.

Example 39. The device of any combination of examples 36-38, wherein the means for determining the weights comprises: means for determining a difference between each of the one or more microphone locations and the listener location; and means for determining, based on the difference between each of the one or more microphone locations and the listener location, the weight for each of the audio streams.

Example 40. The device of any combination of examples 36-39, wherein the means for determining the weights comprises means for determining the weights for each audio frame of the one or more audio streams.

Example 41. The device of any combination of examples 35-40, wherein audio sources represented by the audio streams reside outside of the one or more microphones.

Example 42. The device of any combination of examples 35-41, wherein the means for obtaining the listener location comprises means for obtaining, from a computer mediated reality device, the listener location.

Example 43. The device of example 42, wherein the computer mediated reality device comprises a head mounted display device.

Example 44. The device of any combination of examples 35-43, wherein the means for obtaining the one or more microphone locations comprises means for obtaining, from a bitstream that includes the audio streams, audio metadata that identifies the one or more microphone locations.

Example 45. The device of any combination of examples 35-44, wherein at least one of the one or more microphone locations changes to reflect movement of the corresponding one of the one or more microphones.

Example 46. The device of any combination of examples 35-45, wherein the one or more audio streams include a ambisonic audio stream (including higher order, mixed order, first order, second order), and wherein the interpolated audio stream includes an interpolated ambisonic audio stream (including higher order, mixed order, first order, second order).

Example 47. The device of any combination of examples 35-44, wherein the one or more audio streams include an ambisonic audio stream, and wherein the interpolated audio stream includes an interpolated ambisonic audio stream.

Example 48. The device of any combination of examples 35-47, wherein the listener location changes based on navigational commands issued by the listener.

Example 49. The device of any combination of examples 35-48, wherein the means for obtaining the microphone locations comprises means for receiving audio metadata specifying the microphone locations, each of the microphone locations identifying a location of a cluster of microphones that captured the corresponding one or more audio streams.

Example 50. The device of any combination of examples 49, wherein the cluster of microphones are each positioned at a distance from one another that is greater than five feet.

Example 51. The device of any combination of examples 35-48, wherein the microphones are each positioned at a distance greater than five feet from one another.

Example 52. A non-transitory computer-readable storage medium having stored thereon instructions that, when executed, cause one or more processors to: obtain one or more microphone locations, each of the one or more microphone locations identifying a location of a respective one or more microphones that captured each of the corresponding one or more audio streams; obtain a listener location identifying a location of a listener; perform interpolation, based on the one or more microphone locations and the listener location, with respect to the audio streams to obtain an interpolated audio stream; obtain, based on the interpolated audio stream, one or more speaker feeds; and output the one or more speaker feeds.

In some examples, the VR device (or the streaming device) may communicate, using a network interface coupled to a memory of the VR/streaming device, exchange messages to an external device, where the exchange messages are associated with the multiple available representations of the soundfield. In some examples, the VR device may receive, using an antenna coupled to the network interface, wireless signals including data packets, audio packets, video packets, or transport protocol data associated with the multiple available representations of the soundfield. In some examples, one or more microphone arrays may capture the soundfield.

In some examples, the multiple available representations of the soundfield stored to the memory device may include a plurality of object-based representations of the soundfield, higher order ambisonic representations of the soundfield, mixed order ambisonic representations of the soundfield, a combination of object-based representations of the soundfield with higher order ambisonic representations of the soundfield, a combination of object-based representations of the soundfield with mixed order ambisonic representations of the soundfield, or a combination of mixed order representations of the soundfield with higher order ambisonic representations of the soundfield.

In some examples, one or more of the soundfield representations of the multiple available representations of the soundfield may include at least one high-resolution region and at least one lower-resolution region, and wherein the selected presentation based on the steering angle provides a greater spatial precision with respect to the at least one high-resolution region and a lesser spatial precision with respect to the lower-resolution region.