Patent Description:
A higher order ambisonic (HOA) signal (often represented by a plurality of spherical harmonic coefficients (SHC) or other hierarchical elements) is a three-dimensional (3D) representation of a soundfield. The HOA or SHC representation may represent this soundfield in a manner that is independent of the local speaker geometry used to playback a multi-channel audio signal rendered from this SHC signal. The SHC signal may also facilitate backwards compatibility as the SHC signal may be rendered to well-known and highly adopted multi-channel formats, such as a <NUM> audio channel format or a <NUM> audio channel format. The SHC representation may therefore enable a better representation of a soundfield that also accommodates backward compatibility.

<CIT> discloses a decoding device including at least one circuit configured to acquire one or more encoded audio signals including a plurality of channels and/or a plurality of objects and priority information for each of the plurality of channels and/or the plurality of objects, and to decode the one or more encoded audio signals according to the priority information.

<CIT> discloses a device for audio signal processing and describes techniques for grouping audio objects into clusters to reduce bit rate requirements when encoding audio in 2D and/or 3D.

In general, techniques are described for a vector-based higher order ambisonic format with priority information to potentially prioritize subsequent processing of higher order ambisonic audio data. Higher order ambisonic audio data may comprise at least one spherical harmonic coefficient corresponding to a spherical harmonic basis function having an order greater than one and, in some examples, a plurality of spherical harmonic coefficients corresponding to multiple spherical harmonic basis functions having an order greater than one.

The details of one or more aspects of the techniques are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of these techniques will be apparent from the description and drawings, and from the claims.

There are various 'surround-sound' channel-based formats in the market. They range, for example, from the <NUM> home theatre system (which has been the most successful in terms of making inroads into living rooms beyond stereo) to the <NUM> system developed by NHK (Nippon Hoso Kyokai or Japan Broadcasting Corporation). Content creators (e.g., Hollywood studios, which may also be referred to as content providers) would like to produce the soundtrack for a movie once, and not spend effort to remix it for each speaker configuration. The Moving Pictures Expert Group (MPEG) has released a standard allowing for soundfields to be represented using a hierarchical set of elements (e.g., Higher-Order Ambisonic - HOA - coefficients) that can be rendered to speaker feeds for most speaker configurations, including <NUM> and <NUM> configuration whether in location defined by various standards or in non-uniform locations.

MPEG released the standard as MPEG-H 3D Audio standard, formally entitled "Information technology - High efficiency coding and media delivery in heterogeneous environments - Part <NUM>: 3D audio," set forth by ISO/IEC JTC <NUM>/SC <NUM>, with document identifier ISO/IEC DIS <NUM>-<NUM>, and dated July <NUM>, <NUM>. MPEG also released a second edition of the 3D Audio standard, entitled "Information technology - High efficiency coding and media delivery in heterogeneous environments - Part <NUM>: 3D audio, set forth by ISO/IEC JTC <NUM>/SC <NUM>, with document identifier ISO/IEC <NUM>-<NUM>:201x(E), and dated October <NUM>, <NUM>. Reference to the "3D Audio standard" in this disclosure may refer to one or both of the above standards.

As noted above, one example of a hierarchical set of elements is a set of spherical harmonic coefficients (SHC). The following expression demonstrates a description or representation of a soundfield using SHC: <MAT>.

The expression shows that the pressure pi at any point {rr, θr, φr} of the soundfield, at time t, can be represented uniquely by the SHC, <MAT>. Here, <MAT>, c is the speed of sound (~<NUM>/s), {rr, θr, φr} is a point of reference (or observation point), jn(·) is the spherical Bessel function of order n, and <MAT> are the spherical harmonic basis functions (which may also be referred to as a spherical basis function) of order n and suborder m. It can be recognized that the term in square brackets is a frequency-domain representation of the signal (i.e., S(ω, rr, θr, φr)) which can be approximated by various time-frequency transformations, such as the discrete Fourier transform (DFT), the discrete cosine transform (DCT), or a wavelet transform. Other examples of hierarchical sets include sets of wavelet transform coefficients and other sets of coefficients of multiresolution basis functions.

<FIG> is a diagram illustrating spherical harmonic basis functions from the zero order (n = <NUM>) to the fourth order (n = <NUM>). As can be seen, for each order, there is an expansion of suborders m which are shown but not explicitly noted in the example of <FIG> for ease of illustration purposes.

The SHC <MAT> can either be physically acquired (e.g., recorded) by various microphone array configurations or, alternatively, they can be derived from channel-based or object-based descriptions of the soundfield. The SHC (which also may be referred to as higher order ambisonic - HOA - coefficients) represent scene-based audio, where the SHC may be input to an audio encoder to obtain encoded SHC that may promote more efficient transmission or storage. For example, a fourth-order representation involving (<NUM>+<NUM>)<NUM> (<NUM>, and hence fourth order) coefficients may be used.

As noted above, the SHC may be derived from a microphone recording using a microphone array. Various examples of how SHC may be derived from microphone arrays are described in <NPL>.

To illustrate how the SHCs may be derived from an object-based description, consider the following equation. The coefficients <MAT> for the soundfield corresponding to an individual audio object may be expressed as: <MAT> where i is <MAT> is the spherical Hankel function (of the second kind) of order n, and {rs, θs, φs} is the location of the object. Knowing the object source energy g(ω) as a function of frequency (e.g., using time-frequency analysis techniques, such as performing a fast Fourier transform on the PCM stream) allows us to convert each PCM object and the corresponding location into the SHC <MAT>. Further, it can be shown (since the above is a linear and orthogonal decomposition) that the <MAT> coefficients for each object are additive. In this manner, a number of PCM objects can be represented by the <MAT> coefficients (e.g., as a sum of the coefficient vectors for the individual objects). Essentially, the coefficients contain information about the soundfield (the pressure as a function of 3D coordinates), and the above represents the transformation from individual objects to a representation of the overall soundfield, in the vicinity of the observation point {rr, θr, φr}. The remaining figures are described below in the context of SHC-based audio coding.

<FIG> is a diagram illustrating a system <NUM> that may perform various aspects of the techniques described in this disclosure. As shown in the example of <FIG>, the system <NUM> includes a broadcasting network <NUM> and a content consumer <NUM>. While described in the context of the broadcasting network <NUM> and the content consumer <NUM>, the techniques may be implemented in any context in which SHCs (which may also be referred to as HOA coefficients) or any other hierarchical representation of a soundfield are encoded to form a bitstream representative of the audio data. Moreover, the broadcasting network <NUM> may represent a system comprising one or more of any form of computing devices capable of implementing the techniques described in this disclosure, including a handset (or cellular phone, including a so-called "smart phone"), a tablet computer, a laptop computer, a desktop computer, or dedicated hardware to provide a few examples. Likewise, the content consumer <NUM> may represent any form of computing device capable of implementing the techniques described in this disclosure, including a handset (or cellular phone, including a so-called "smart phone"), a tablet computer, a television, a set-top box, a laptop computer, a gaming system or console, or a desktop computer to provide a few examples.

The broadcasting network <NUM> may represent any entity that may generate multi-channel audio content and possibly video content for consumption by content consumers, such as the content consumer <NUM>. The broadcasting network <NUM> may represent one example of a content provider. The broadcasting network <NUM> may capture live audio data at events, such as sporting events, while also inserting various other types of additional audio data, such as commentary audio data, commercial audio data, intro or exit audio data and the like, into the live audio content.

The content consumer <NUM> represents an individual that owns or has access to an audio playback system, which may refer to any form of audio playback system capable of rendering higher order ambisonic audio data (which includes higher order audio coefficients that, again, may also be referred to as spherical harmonic coefficients) for play back as multi-channel audio content. The higher-order ambisonic audio data may be defined in the spherical harmonic domain and rendered or otherwise transformed form the spherical harmonic domain to a spatial domain, resulting in the multi-channel audio content. In the example of <FIG>, the content consumer <NUM> includes an audio playback system <NUM>.

The broadcasting network <NUM> includes microphones <NUM> that record or otherwise obtain live recordings in various formats (including directly as HOA coefficients) and audio objects. When the microphone array <NUM> (which may also be referred to as "microphones <NUM>") obtains live audio directly as HOA coefficients, the microphones <NUM> may include an HOA transcoder, such as an HOA transcoder <NUM> shown in the example of <FIG>. In other words, although shown as separate from the microphones <NUM>, a separate instance of the HOA transcoder <NUM> may be included within each of the microphones <NUM> so as to naturally transcode the captured feeds into the HOA coefficients <NUM>. However, when not included within the microphones <NUM>, the HOA transcoder <NUM> may transcode the live feeds output from the microphones <NUM> into the HOA coefficients <NUM>. In this respect, the HOA transcoder <NUM> may represent a unit configured to transcode microphone feeds and/or audio objects into the HOA coefficients <NUM>. The broadcasting network <NUM> therefore includes the HOA transcoder <NUM> as integrated with the microphones <NUM>, as an HOA transcoder separate from the microphones <NUM> or some combination thereof.

The broadcasting network <NUM> may also include a spatial audio encoding device <NUM>, a broadcasting network center <NUM> (which may also be referred to as a "network operations center" - NOC - <NUM>) and a psychoacoustic audio encoding device <NUM>. The spatial audio encoding device <NUM> may represent a device capable of performing the mezzanine compression techniques described in this disclosure with respect to the HOA coefficients <NUM> to obtain intermediately formatted audio data <NUM> (which may also be referred to as "mezzanine formatted audio data <NUM>"). Intermediately formatted audio data <NUM> may represent audio data that conforms with an intermediate audio format (such as a mezzanine audio format). As such, the mezzanine compression techniques may also be referred to as intermediate compression techniques.

The spatial audio encoding device <NUM> may be configured to perform this intermediate compression (which may also be referred to as "mezzanine compression") with respect to the HOA coefficients <NUM> by performing, at least in part, a decomposition (such as a linear decomposition, including a singular value decomposition, eigenvalue decomposition, KLT, etc.) with respect to the HOA coefficients <NUM>. Furthermore, the spatial audio encoding device <NUM> may perform the spatial encoding aspects (excluding the psychoacoustic encoding aspects) to generate a bitstream conforming to the above referenced MPEG-H 3D audio coding standard. In some examples, the spatial audio encoding device <NUM> may perform the vector-based aspects of the MPEG-H 3D audio coding standard.

Although described in this disclosure with respect to a bitstream, such as a bitstream having multiple, or in other words, a plurality of transport channels, the techniques may be performed with respect to any type of data object. A data object may refer to any type of formatted data, including the aforementioned bitstream as well as files having multiple tracks, or other types of data objects.

The spatial audio encoding device <NUM> may be configured to encode the HOA coefficients <NUM> using a decomposition involving application of a linear invertible transform (LIT). One example of the linear invertible transform is referred to as a "singular value decomposition" (or "SVD"), which may represent one form of a linear decomposition. In this example, the spatial audio encoding device <NUM> may apply SVD to the HOA coefficients <NUM> to determine a decomposed version of the HOA coefficients <NUM>.

The decomposed version of the HOA coefficients <NUM> may include one or more sound components (which may refer to, as one example, an audio object defined in a spatial domain) and/or one or more corresponding spatial components. The sound components having corresponding spatial components may also be referred to as predominant audio signals, or predominant sound components. The sound components may also refer to ambisonic audio coefficients selected from the HOA coefficients <NUM>. While the predominant sound components may be defined in the spatial domain, the spatial component may be defined in the spherical harmonic domain. The spatial component may represent a weighted summation of two or more directional vectors defining shapes, width, and directions of the associated predominant audio signals (which may be referred to in the MPEG-H 3D audio coding standard as a "V-vector").

The spatial audio encoding device <NUM> may then analyze the decomposed version of the HOA coefficients <NUM> to identify various parameters, which may facilitate reordering of the decomposed version of the HOA coefficients <NUM>. The spatial audio encoding device <NUM> may reorder the decomposed version of the HOA coefficients <NUM> based on the identified parameters, where such reordering, as described in further detail below, may improve coding efficiency given that the transformation may reorder the HOA coefficients across frames of the HOA coefficients (where a frame commonly includes M samples of the HOA coefficients <NUM> and M is, in some examples, set to <NUM>).

After reordering the decomposed version of the HOA coefficients <NUM>, the spatial audio encoding device <NUM> may select those of the decomposed version of the HOA coefficients <NUM> representative of foreground (or, in other words, distinct, predominant or salient) components of the soundfield. The spatial audio encoding device <NUM> may specify the decomposed version of the HOA coefficients <NUM> representative of the foreground components as an audio object (which may also be referred to as a "predominant sound signal," or a "predominant sound component") and associated spatial information (which may also be referred to as a spatial component).

The spatial audio encoding device <NUM> may next perform a soundfield analysis with respect to the HOA coefficients <NUM> in order to, at least in part, identify the HOA coefficients <NUM> representative of one or more background (or, in other words, ambient) components of the soundfield. The spatial audio encoding device <NUM> may perform energy compensation with respect to the background components given that, in some examples, the background components may only include a subset of any given sample of the HOA coefficients <NUM> (e.g., such as those corresponding to zero and first order spherical basis functions and not those corresponding to second or higher order spherical basis functions). When order-reduction is performed, in other words, the spatial audio encoding device <NUM> may augment (e.g., add/subtract energy to/from) the remaining background HOA coefficients of the HOA coefficients <NUM> to compensate for the change in overall energy that results from performing the order reduction.

The spatial audio encoding device <NUM> may perform a form of interpolation with respect to the foreground directional information (which again may be another way to refer to the spatial components) and then perform an order reduction with respect to the interpolated foreground directional information to generate order reduced foreground directional information. The spatial audio encoding device <NUM> may further perform, in some examples, a quantization with respect to the order reduced foreground directional information, outputting coded foreground directional information. In some instances, this quantization may comprise a scalar/entropy quantization.

The spatial audio encoding device <NUM> may then output the mezzanine formatted audio data <NUM> as the background components, the foreground audio objects, and the quantized directional information. Each of the background components and the foreground audio objects may be specified in the bitstream as separate pulse code modulated (PCM) transport channels in some examples. Each of the quantized directional information corresponding to each of the foreground audio objects may be specified in the bitstream as sideband information (which may not, in some examples, undergo subsequent psychoacoustic audio encoding/compression to preserve the spatial information). The mezzanine formatted audio data <NUM> may represent one example of a data object (in the form, in this instance, of a bitstream), and as such may be referred to as a mezzanine formatted data object <NUM> or mezzanine formatted bitstream <NUM>.

The spatial audio encoding device <NUM> may then transmit or otherwise output the mezzanine formatted audio data <NUM> to the broadcasting network center <NUM>. Although not shown in the example of <FIG>, further processing of the mezzanine formatted audio data <NUM> may be performed to accommodate transmission from the spatial audio encoding device <NUM> to the broadcasting network center <NUM> (such as encryption, satellite compression schemes, fiber compression schemes, etc.).

Mezzanine formatted audio data <NUM> may represent audio data that conforms to a so-called mezzanine format, which is typically a lightly compressed (relative to end-user compression provided through application of psychoacoustic audio encoding to audio data, such as MPEG surround, MPEG-AAC, MPEG-USAC or other known forms of psychoacoustic encoding) version of the audio data. Given that broadcasters prefer dedicated equipment that provides low latency mixing, editing, and other audio and/or video functions, broadcasters are reluctant to upgrade the equipment given the cost of such dedicated equipment.

To accommodate the increasing bitrates of video and/or audio and provide interoperability with older or, in other words, legacy equipment that may not be adapted to work on high definition video content or 3D audio content, broadcasters have employed this intermediate compression scheme, which is generally referred to as "mezzanine compression," to reduce file sizes and thereby facilitate transfer times (such as over a network or between devices) and improved processing (especially for older legacy equipment). In other words, this mezzanine compression may provide a more lightweight version of the content which may be used to facilitate editing times, reduce latency and potentially improve the overall broadcasting process.

The broadcasting network center <NUM> may therefore represent a system responsible for editing and otherwise processing audio and/or video content using an intermediate compression scheme to improve the work flow in terms of latency. The broadcasting network center <NUM> may, in some examples, include a collection of mobile devices. In the context of processing audio data, the broadcasting network center <NUM> may, in some examples, insert intermediately formatted additional audio data into the live audio content represented by the mezzanine formatted audio data <NUM>. This additional audio data may comprise commercial audio data representative of commercial audio content (including audio content for television commercials), television studio show audio data representative of television studio audio content, intro audio data representative of intro audio content, exit audio data representative of exit audio content, emergency audio data representative of emergency audio content (e.g., weather warnings, national emergencies, local emergencies, etc.) or any other type of audio data that may be inserted into mezzanine formatted audio data <NUM>.

In some examples, the broadcasting network center <NUM> includes legacy audio equipment capable of processing up to <NUM> audio channels. In the context of 3D audio data that relies on HOA coefficients, such as the HOA coefficients <NUM>, the HOA coefficients <NUM> may have more than <NUM> audio channels (e.g., a <NUM>th order representation of the 3D soundfield would require (<NUM>+<NUM>)<NUM> or <NUM> HOA coefficients per sample, which is equivalent to <NUM> audio channels). This limitation in legacy broadcasting equipment may slow adoption of 3D HOA-based audio formats, such as that set forth in the ISO/IEC DIS <NUM>-<NUM>:201x(E) document, entitled "Information technology - High efficiency coding and media delivery in heterogeneous environments - Part <NUM>: 3D audio," by ISO/IEC JTC <NUM>/SC <NUM>/WG <NUM>, dated <NUM>-<NUM>-<NUM> (which may be referred to herein as the "3D Audio Coding Standard" or the "MPEG-H 3D Audio Coding Standard").

As such, the mezzanine compression allows for obtaining the mezzanine formatted audio data <NUM> from the HOA coefficients <NUM> in a manner that overcomes the channel-based limitations of legacy audio equipment. That is, the spatial audio encoding device <NUM> may be configured to obtain the mezzanine audio data <NUM> having <NUM> or fewer audio channels (and possibly as few as <NUM> audio channels given that legacy audio equipment may, in some examples, allow for processing <NUM> audio content, where the '. <NUM>' represents the sixth audio channel).

The broadcasting network center <NUM> may output updated mezzanine formatted audio data <NUM>. The updated mezzanine formatted audio data <NUM> may include the mezzanine formatted audio data <NUM> and any additional audio data inserted into the mezzanine formatted audio data <NUM> by the broadcasting network center <NUM>. Prior to distribution, the broadcasting network <NUM> may further compress the updated mezzanine formatted audio data <NUM>. As shown in the example of <FIG>, the psychoacoustic audio encoding device <NUM> may perform psychoacoustic audio encoding (e.g., any one of the examples described above) with respect to the updated mezzanine formatted audio data <NUM> to generate a bitstream <NUM>. The broadcasting network <NUM> may then transmit the bitstream <NUM> via a transmission channel to the content consumer <NUM>.

In some examples, the psychoacoustic audio encoding device <NUM> may represent multiple instances of a psychoacoustic audio coder, each of which is used to encode a different audio object or HOA channel of each of updated mezzanine formatted audio data <NUM>. In some instances, this psychoacoustic audio encoding device <NUM> may represent one or more instances of an advanced audio coding (AAC) encoding unit. Often, the psychoacoustic audio coder unit <NUM> may invoke an instance of an AAC encoding unit for each channel of the updated mezzanine formatted audio data <NUM>.

More information regarding how the background spherical harmonic coefficients may be encoded using an AAC encoding unit can be found in a convention paper by <NPL>. In some instances, the psychoacoustic audio encoding device <NUM> may audio encode various channels (e.g., background channels) of the updated mezzanine formatted audio data <NUM> using a lower target bitrate than that used to encode other channels (e.g., foreground channels) of the updated mezzanine formatted audio data <NUM>.

While shown in <FIG> as being directly transmitted to the content consumer <NUM>, the broadcasting network <NUM> may output the bitstream <NUM> to an intermediate device positioned between the broadcasting network <NUM> and the content consumer <NUM>. The intermediate device may store the bitstream <NUM> for later delivery to the content consumer <NUM>, which may request this bitstream. The intermediate device may comprise a file server, a web server, a desktop computer, a laptop computer, a tablet computer, a mobile phone, a smart phone, or any other device capable of storing the bitstream <NUM> for later retrieval by an audio decoder. The intermediate device may reside in a content delivery network capable of streaming the bitstream <NUM> (and possibly in conjunction with transmitting a corresponding video data bitstream) to subscribers, such as the content consumer <NUM>, requesting the bitstream <NUM>. Alternately, the intermediate device may reside within broadcasting network <NUM>.

Alternatively, the broadcasting network <NUM> may store the bitstream <NUM> to a storage medium as a file, such as a compact disc, a digital video disc, a high definition video disc or other storage media, most of which are capable of being read by a computer and therefore may be referred to as computer-readable storage media or non-transitory computer-readable storage media. In this context, the transmission channel may refer to those channels by which content stored to these mediums are transmitted (and may include retail stores and other store-based delivery mechanism). In any event, the techniques of this disclosure should not therefore be limited in this respect to the example of <FIG>. As a file, the transport channels to which various aspects of the decomposed version of the HOA coefficients <NUM> are stored may be referred to as tracks.

As further shown in the example of <FIG>, the content consumer <NUM> includes the audio playback system <NUM>. The audio playback system <NUM> may represent any audio playback system capable of playing back multi-channel audio data. The audio playback system <NUM> may include a number of different audio renderers <NUM>. The audio renderers <NUM> may each provide for a different form of rendering, where the different forms of rendering may include one or more of the various ways of performing vector-base amplitude panning (VBAP), and/or one or more of the various ways of performing soundfield synthesis.

The audio playback system <NUM> may further include an audio decoding device <NUM>. The audio decoding device <NUM> may represent a device configured to decode HOA coefficients <NUM>' from the bitstream <NUM>, where the HOA coefficients <NUM>' may be similar to the HOA coefficients <NUM> but differ due to lossy operations (e.g., quantization) and/or transmission via the transmission channel.

That is, the audio decoding device <NUM> may dequantize the foreground directional information specified in the bitstream <NUM>, while also performing psychoacoustic decoding with respect to the foreground audio objects specified in the bitstream <NUM> and the encoded HOA coefficients representative of background components. The audio decoding device <NUM> may further perform interpolation with respect to the decoded foreground directional information and then determine the HOA coefficients representative of the foreground components based on the decoded foreground audio objects and the interpolated foreground directional information. The audio decoding device <NUM> may then determine the HOA coefficients <NUM>' based on the determined HOA coefficients representative of the foreground components and the decoded HOA coefficients representative of the background components.

The audio playback system <NUM> may, after decoding the bitstream <NUM> to obtain the HOA coefficients <NUM>', render the HOA coefficients <NUM>' to output loudspeaker feeds <NUM>. The audio playback system <NUM> may output loudspeaker feeds <NUM> to one or more of loudspeakers <NUM>. The loudspeaker feeds <NUM> may drive one or more loudspeakers <NUM>.

To select the appropriate renderer or, in some instances, generate an appropriate renderer, the audio playback system <NUM> may obtain loudspeaker information <NUM> indicative of a number of the loudspeakers <NUM> and/or a spatial geometry of the loudspeakers <NUM>. In some instances, the audio playback system <NUM> may obtain the loudspeaker information <NUM> using a reference microphone and drive the loudspeakers <NUM> in such a manner as to dynamically determine the loudspeaker information <NUM>. In other instances or in conjunction with the dynamic determination of the loudspeaker information <NUM>, the audio playback system <NUM> may prompt a user to interface with the audio playback system <NUM> and input the loudspeaker information <NUM>.

The audio playback system <NUM> may select one of the audio renderers <NUM> based on the loudspeaker information <NUM>. In some instances, the audio playback system <NUM> may, when none of the audio renderers <NUM> are within some threshold similarity measure (in terms of the loudspeaker geometry) to that specified in the loudspeaker information <NUM>, generate the one of audio renderers <NUM> based on the loudspeaker information <NUM>. The audio playback system <NUM> may, in some instances, generate the one of audio renderers <NUM> based on the loudspeaker information <NUM> without first attempting to select an existing one of the audio renderers <NUM>.

While described with respect to loudspeaker feeds <NUM>, the audio playback system <NUM> may render headphone feeds from either the loudspeaker feeds <NUM> or directly from the HOA coefficients <NUM>', outputting the headphone feeds to headphone speakers. The headphone feeds may represent binaural audio speaker feeds, which the audio playback system <NUM> renders using a binaural audio renderer.

As noted above, the spatial audio encoding device <NUM> may analyze the soundfield to select a number of HOA coefficients (such as those corresponding to spherical basis functions having an order of one or less) to represent am ambient component of the soundfield. The spatial audio encoding device <NUM> may also, based on this or another analysis, select a number of predominant audio signals and corresponding spatial components to represent various aspects of a foreground component of the soundfield, discarding any remaining predominant audio signals and corresponding spatial components.

The spatial audio encoding device <NUM> may specify these various components of the soundfield in separate transport channels (or, in the example of files, tracks) of the bitstream (or, in the example of tracks, files). The psychoacoustic audio encoding device <NUM> may then further reduce the number of transport channels (or tracks) when forming bitstream <NUM> (which may also be illustrative of files, and as such may be referred to as "files <NUM>" or, more generally, "data object <NUM>," which may refer to both bitstreams and/or files). The psychoacoustic audio encoding device <NUM> may reduce the number of transport channels to generate bitstream <NUM> that achieves a specified target bitrate. The target bitrate may be mandated by broadcasting network <NUM>, determined through analysis of transmission channel <NUM>, requested by audio playback system <NUM>, or obtained through any other mechanism employed to determine a target bitrate.

The psychoacoustic audio encoding device <NUM> may implement any number of different processes by which to select the non-zero subset of the transport channels of the mezzanine formatted audio data <NUM> (which is included in updated mezzanine formatted audio data <NUM>). Reference to a "subset" in this disclosure is intended to refer to a "non-zero subset" having less data than the total number of elements in the larger set unless explicitly noted otherwise, and not the strict mathematical definition of a subset that would include zero or more elements of the larger set up to total elements of the larger set. However, the psychoacoustic audio encoding device <NUM> may not have sufficient time (e.g., when live broadcasting) or computational capacity to perform detailed analysis that enable accurate identification of which transport channels of the larger set of transport channels set forth in the mezzanine formatted audio data <NUM> are to be specified in the bitstream <NUM> while still preserving adequate audio quality (and limiting injection of audio artifacts that decrease perceived audio quality).

Furthermore, as noted above, the spatial audio encoding device <NUM> may specify the background components (or, in other words, the ambient HOA coefficients) to transport channels of bitstream <NUM>, while specifying foreground components (or, in other words, the predominant sound components) and the corresponding spatial components to transport channels of bitstream <NUM> and sideband information, respectively. Having to specify the background components in a manner differently than foreground components (in that the foreground components also include the corresponding spatial components) may result in bandwidth inefficiencies, due to having to signal separate transport channel formats to identify which of the transport channels specify a background component and which of the transport channels specify a foreground component.

The signaling of transport format results in memory, storage, and/or bandwidth inefficiencies as the transport format is signaled on a per transport channel basis for every frame, resulting in increased bitstream size (as bitstreams may include thousands, hundreds of thousands, millions, and possible tens of millions of frames), leading to potentially larger memory and/or storage space consumption, slower retrieval of the bitstream from memory and/or storage space, increased internal memory bus bandwidth consumption, increased network bandwidth consumption, etc. These memory, storage, and/or bandwidth inefficiencies may impact operation of the underlying computing devices themselves.

In accordance with the techniques described in this disclosure, the spatial audio encoding device <NUM> may determine, based on one or more of the sound component and the corresponding spatial component, priority information indicative of a priority of the sound component relative to other sound components of the soundfield represented by the HOA coefficients <NUM>. As noted above, the term "sound component" may refer to both a predominant sound component (e.g., an audio object defined in a spatial domain), and an ambient HOA coefficient (which is defined in the spherical harmonic domain). The corresponding spatial component may refer to the above noted V-vector, which defines shape, width, and directions of the predominant sound component, and is also defined in a spherical harmonic domain.

The spatial audio encoding device <NUM> may determine the priority information in a number of different ways. For example, the spatial audio encoding device <NUM> may determine an energy of the sound component or of an HOA representation of the sound component. To determine the energy of the HOA representation of the sound component, the spatial audio encoding device <NUM> may multiply the sound component by the corresponding spatial component (or, in some instances, a transpose of the corresponding spatial component) to obtain the HOA representation of the sound component, and then determine the energy of the HOA representation of the sound component.

The spatial audio encoding device <NUM> may next determine, based on the determined energy, the priority information. In some examples, the spatial audio encoding device <NUM> may determine the energy for each sound component decomposed from the HOA coefficients <NUM> (or the HOA representation of each sound component). The spatial audio encoding device <NUM> may determine a highest priority for the sound component having the highest energy (where the highest priority may be denoted by a lowest priority value or a highest priority value relative to the other priority values), a second highest priority for the sound component having the second highest energy, etc..

Although described with respect to energy, the spatial audio encoding device <NUM> may determine a loudness measure of the sound component or the HOA representation of the sound component. The spatial audio encoding device <NUM> may determine, based on the loudness measure, the priority information. Moreover, in some examples, the spatial audio encoding device <NUM> may determine both an energy and a loudness measure of the sound component, and next determine, based on one or more of the energy and the loudness measure, the priority information.

In this and other examples, the spatial audio encoding device <NUM> may, to determine the energy or the loudness measure, render the HOA representation of the sound component to one or more speaker feeds. The spatial audio encoding device <NUM> may render the HOA representation of the sound component to, as one example, the one or more speakers feeds suited for speakers arranged in a regular geometry (such as the speaker geometry defined for <NUM>, <NUM>, <NUM>, <NUM>, and other uniform surround sound formats, including those introducing speakers on multiple heights, such as <NUM>. <NUM>, <NUM>. <NUM>, etc. where the third numeral (e.g., the <NUM> in <NUM>. <NUM> or <NUM> in <NUM>. <NUM>) indicates the number of speakers on the higher horizontal plane). The spatial audio encoding device <NUM> may then determine, based on the one or more speaker feeds, the energy and/or the loudness measure.

In this and other examples, the spatial audio encoding device <NUM> may determine, based on the spatial component, a spatial weighting indicative of a relevance of the sound component to the soundfield. To illustrate, the spatial audio encoding device <NUM> may determine a spatial weighting indicating that the corresponding current sound component is located in the soundfield at approximately head-height, directly in front of the listener, which indicates that the current sound component is likely to be of relatively more importance in comparison to other sound components located in the soundfield to the right, left, above, or below the current sound component.

The spatial audio encoding device <NUM> may determine, based on the spatial component and as another illustration, that the current sound component is higher in the soundfield, which may be indicative of the current sound component being of relatively more importance than those below head-height, as the human auditory system is more sensitive to sound arriving from above the head than sounds arriving from below the head. Likewise, the spatial audio encoding device <NUM> may determine a spatial weighting indicating that the sound component is in front of the listener's head and potentially of more importance than other sound components located behind the listener's head as the human auditory system is more sensitive to sound arriving from in front of the listener's head relative to sounds arriving at the listener's head from behind. The spatial audio encoding device <NUM> may determine, as yet another example, based on one or more of the energy, the loudness measure, and the spatial weighting, the priority information.

In these and other examples, the spatial audio encoding device <NUM> may determine a continuity indication indicative of whether a current portion (e.g., a current frame in the case of a transport channel in the bitstream <NUM> or a current track in the case of a file) defines the same sound component as a previous portion (e.g., a previous frame of the same transport channel in the bitstream <NUM> or a previous track in the case of a file). Based on the continuity indication, the spatial audio encoding device <NUM> may determine the priority information. The spatial audio encoding device <NUM> may assign sound components having positive continuity indications across portions a higher priority than sound components having negative continuity indications as continuity in audio scenes is generally more important (in terms of a positive listening experience in terms of quality and noticeable artifacts) relative to failures to inject new sound components at the correct time.

In these and other examples, the spatial audio encoding device <NUM> may perform signal classification with respect to the sound component, the higher order ambisonic representation of the sound component and/or the one or more rendered speaker feeds to determine a class to which the sound component corresponds. As one example, the spatial audio encoding device <NUM> may perform signal classification to identify whether the sound component belongs to a speech class or a non-speech class, where the speech class indicates that the sound component is primarily speech content, while the non-speech class indicates that the sound component is primarily non-speech content.

The spatial audio encoding device <NUM> may then determine, based on the class, the priority information. The spatial audio encoding device <NUM> may assign sound components associated with the speech class with a higher priority compared to sound components associated with the non-speech class, as speech content is generally more important to a given audio scene than non-speech content.

As yet another example, the spatial audio encoding device <NUM> may obtain, from the content provider providing the HOA audio data (which may refer to the HOA coefficients <NUM> among other metadata or audio data), a preferred priority of the sound component relative to other sound components of the soundfield. That is, the content provider may indicate which locations in the 3D soundfield have a higher priority (or, in other words, a preferred priority) than other locations in the soundfield. The spatial audio encoding device <NUM> may determine, based on the preferred priority, the priority information.

Although described above as determining the priority information based on various combinations of different types of data, the spatial audio encoding device <NUM> may determine the priority information based on one or more of the energy, the loudness measure, the spatial weighting, the continuity indication, the preferred priority, and the class, as a few examples. A number of detailed examples of different combination are described below with respect to FIGS.

The spatial audio encoding device <NUM> may specify, in the bitstream <NUM> representative of a compressed version of the HOA coefficients <NUM>, the sound component and the priority information. In some examples, the spatial audio encoding device <NUM> may specify a plurality of sound components and priority information indicative of a priority of each of the plurality of sound components relative to remaining ones of the sound components.

The psychoacoustic audio encoding device <NUM> may obtain, from the bitstream <NUM> (embedded in the bitstream <NUM>), the plurality of sound components and the priority information indicative of the priority of each of the plurality of sound components relative to remaining ones of the sound components. The psychoacoustic audio encoding device <NUM> may select, based on the priority information, a non-zero subset of the plurality of sound components.

As noted above, the psychoacoustic audio encoding device <NUM> may have different channel or track constraints than the spatial audio encoding device <NUM> had when formulating the bitstream <NUM>, where the psychoacoustic audio encoding device <NUM> may have a reduced number of channels or tracks by which to specify the sound components relative to the spatial audio encoding device <NUM>. Using the priority information, the psychoacoustic audio encoding device <NUM> may more efficiently identify the more important sound components that should undergo psychoacoustic encoding, and thereby result in a better quality representation of the HOA coefficients <NUM>.

The efficiencies gained by using the priority information come as a result of reducing the computational operations performed by the psychoacoustic audio encoding device <NUM> (and reducing memory consumption resulting from performing increased computation operations), while also improving the speed with which the psychoacoustic audio encoding device <NUM> may encode the bitstream <NUM>. Furthermore, the foregoing aspects of the techniques may reduce energy consumption and prolong potential operating times (e.g., for devices reliant on batteries or other forms of mobile power supply), which impact operation of the psychoacoustic audio encoding device <NUM> itself.

Additionally, the above aspects of the techniques may solve a problem rooted in technology itself given the nature of the computer broadcasting given that the psychoacoustic audio encoding device <NUM> may not have sufficient time (e.g., when live broadcasting) or computational capacity to perform detailed analysis that enables accurate identification of which transport channels of the larger set of transport channels set forth in the mezzanine formatted audio data <NUM> are to be specified in the bitstream <NUM> while still preserving adequate audio quality (and limiting injection of audio artifacts that decrease perceived audio quality). The above noted techniques solve this problem by allowing the spatial audio encoding device <NUM> (which already performs many if not all of the determinations related to energy, loudness, continuity, class, etc. of sound components for purposes of compression) to leverage the functionality used for compression to identify the priority information that may allow the psychoacoustic audio encoding device <NUM> to rapidly select the transport channels that should be specified in the bitstream <NUM>.

In addition to specifying the sound components, the psychoacoustic audio encoding device <NUM> may also obtain a spatial component corresponding to each of the plurality of sound components, and specify, in the bitstream <NUM>, a non-zero subset of the spatial components corresponding to the non-zero subset of the plurality of sound components. After specifying the various sound components and corresponding spatial components, the psychoacoustic audio encoding device <NUM> may perform psychoacoustic audio encoding to obtain the bitstream <NUM>.

In addition or as an alternative to the above described aspects of the techniques, the spatial audio encoding device <NUM> may specify both types of sound components (e.g., the ambient HOA coefficients and the predominant sound components) using a unified format that results in associating a repurposed spatial component to each of the ambient HOA coefficients. The repurposed spatial component may be indicative of one or more of an order and a sub-order of a spherical basis function to which the ambient higher order ambisonic coefficient corresponds.

The format is unified in the sense that both types of the sound components are specified with a corresponding spatial component having a same number of elements. In the case of the repurposed spatial component, the spatial audio encoder device <NUM> may utilize a spatial component having a same number of elements as the spatial components corresponding to the predominant sound components, but repurpose the spatial component to specify a value of one for a single one of the elements that indicates the order and/or the sub-order of the spherical basis function to which the ambient HOA coefficient corresponds.

That is, the repurposed spatial component comprises a vector having a number of elements equal to a maximum order (N) plus one squared (N+<NUM>)<NUM>, where the maximum order is defined as a maximum order of the spherical basis functions to which the HOA coefficients <NUM> corresponds. The vector identifies the order and the sub-order by having a value of one for one of the elements and a value of zero for the remaining elements of the vector. The spatial audio encoding device <NUM> may specify, in the data object and according to the same format, the ambient higher order ambisonic coefficient and the corresponding repurposed spatial component without specifying, in the data object, the order and the sub-order of the ambient higher order ambisonic coefficient.

To identify the correct order and/or suborder, the spatial audio encoder device <NUM> may obtain a harmonic coefficient ordering format indicator indicative of either a symmetric harmonic coefficient ordering format or a linear harmonic coefficient ordering format for the HOA coefficients. More information regarding the harmonic coefficient ordering format indicator, the symmetric harmonic coefficient, and the linear harmonic coefficient ordering format can be found in <CIT>. The spatial audio encoder device <NUM> may obtain, based on the harmonic coefficient ordering format indicator, the repurposed vector. The element of the vector set to a value of one indicates the order and/or the suborder of the spherical basis function to which the corresponding ambient HOA coefficient corresponds by identifying which of the spherical basis functions the ambient HOA coefficient corresponds to when the spherical basis function are ordered according to the indicated ordering format (either symmetric or linear).

The spatial audio encoder device <NUM> may then specify, in the bitstream <NUM> and according to a format (e.g., a transport format or a track format), the predominant sound component and the corresponding spatial component. The spatial audio encoder device <NUM> may also specify, in the bitstream <NUM> and according to the same format, the ambient higher order ambisonic coefficient and the corresponding repurposed spatial component.

The foregoing unified format aspects of the techniques may avoid repeated signaling of the transport format for each transport channel, replacing the signaling of the transport format for each transport channel with the repurposed spatial component, which can be potentially predicted from previous frames, thereby resulting in various efficiencies similar to those described above that result in improvements in the device itself (in terms of decreasing storage consumption, processing cycles - or, in other words, performance of computation operations - bandwidth consumption, etc.).

The audio decoding device <NUM> may receive the bitstream <NUM> having the transport channels specified according to the unified format. The audio decoding device <NUM> may obtain, from the bitstream <NUM> (which again is one example of a data object) and according to a format, an ambient higher order ambisonic coefficient descriptive of an ambient component of the soundfield. The audio decoding device <NUM> may also, obtain, from the bitstream <NUM>, a repurposed spatial component corresponding to the ambient higher order ambisonic coefficient. The audio decoding device <NUM> may further obtain, from the bitstream <NUM> and according to the same format, the predominant sound component, while also obtaining, from the bitstream <NUM>, the corresponding spatial component. Prior to obtaining the various components noted above, the audio decoding device <NUM> may perform psychoacoustic audio decoding with respect to the bitstream <NUM> in a manner reciprocal to the psychoacoustic audio encoding performed by psychoacoustic audio encoding device <NUM> to obtain a bandwidth decompressed version of the bitstream <NUM>.

The audio decoding device <NUM> may then operate in the manner described above to reconstruct and then output the reconstructed HOA coefficients <NUM>' or in the manner set forth in Annex G of the second edition of the MPEG-H 3D Audio Coding Standard referenced above to render, based on the ambient higher order ambisonic coefficient, the repurposed spatial component, the predominant sound component, and the corresponding spatial component, one or more speaker feeds <NUM> (which in the latter case effectively incorporates audio renderers <NUM> into audio decoding device <NUM>). Audio playback system <NUM> may next output, to one or more speakers <NUM>, the one or more speaker feeds <NUM>.

The audio decoding device <NUM> may obtain, from the bitstream <NUM>, a harmonic coefficient ordering format indicator, and determine, based on the harmonic coefficient ordering format indicator, the repurposed vector, and in a manner reciprocal to that described above with respect to the spatial audio encoding device <NUM>, the order and the sub-order of the spherical basis function to which the higher order ambisonic coefficient corresponds. The audio decoding device <NUM> may associate, prior to rendering the one or more speaker feeds <NUM>, the ambient higher order ambisonic coefficient with the spherical basis function having the determined order and sub-order.

Although the audio playback system <NUM> is not shown relative to a larger location, a television, an automobile, headphones, or a headset including the headphones may include the audio playback system <NUM> in which the one or more speakers <NUM> are included as integrated speakers <NUM>. When integrated into headphones or a headset including the headphones, the audio playback system <NUM> may render the speaker feeds <NUM> as one or more binaural audio headphone feeds.

<FIG> are block diagrams illustrating examples of the system <NUM> of <FIG> in more detail. As shown in the example of <FIG>, system 800A is an example of system <NUM>, where system 800A includes a remote truck <NUM>, the network operations center (NOC) <NUM>, a local affiliate <NUM>, and the content consumer <NUM>. The remote truck <NUM> includes the spatial audio encoding device <NUM> (shown as "SAE device <NUM>" in the example of <FIG>) and a contribution encoder device <NUM> (shown as "CE device <NUM>" in the example of <FIG>).

The SAE device <NUM> operates in the manner described above with respect to the spatial audio encoding device <NUM> described above with respect to the example of <FIG>. The SAE device <NUM>, as shown in the example of <FIG>, receives <NUM> HOA coefficients <NUM> and generates the intermediately formatted bitstream <NUM> including <NUM> channels - <NUM> channels of predominant audio signals and ambient HOA coefficients, and <NUM> channel of sideband information defining the spatial components corresponding to the predominant audio signals and adaptive gain control (AGC) information among other sideband information.

The CE device <NUM> operates with respect to the intermediately formatted bitstream <NUM> and video data <NUM> to generate mixed-media bitstream <NUM>. The CE device <NUM> may perform lightweight compression with respect to intermediately formatted audio data <NUM> and video data <NUM> (e.g., captured concurrent to the capture of the HOA coefficients <NUM>). The CE device <NUM> may multiplex frames of the compressed intermediately formatted audio bitstream <NUM> and the compressed video data <NUM> to generate the mixed-media bitstream <NUM>. The CE device <NUM> may transmit the mixed-media bitstream <NUM> to NOC <NUM> for further processing as described above.

The local affiliate <NUM> may represent a local broadcasting affiliate, which broadcasts the content represented by the mixed-media bitstream <NUM> locally. The local affiliate <NUM> may include a contribution decoder device <NUM> (shown as "CD device <NUM>" in the example of <FIG>) and a psychoacoustic audio encoding device <NUM> (shown as "PAE device <NUM>" in the example of <FIG>). The CD device <NUM> may operate in a manner that is reciprocal to operation of the CE device <NUM>. As such, the CD device <NUM> may demultiplex the compressed versions of the intermediately formatted audio bitstream <NUM> and the video data <NUM> and decompress both the compressed versions of the intermediately formatted audio bitstream <NUM> and the video data <NUM> to recover the intermediately formatted bitstream <NUM> and the video data <NUM>. The PAE device <NUM> may operate in the manner described above with respect to the psychoacoustic audio encoder device <NUM> shown in <FIG> to output the bitstream <NUM>. The PAE device <NUM> may be referred to, in the context of broadcasting systems, as an "emission encoder <NUM>.

The emission encoder <NUM> may transcode the bitstream <NUM>, updating the hoaIndependencyFlag syntax element depending on whether the emission encoder <NUM> utilized prediction between audio frames or not, while also potentially changing the value of the number of predominant sound components syntax element when selecting the non-zero subset of the transport channels according to the priority information, and the value of the number of ambient HOA coefficients syntax element. The emission encoder <NUM> may change the hoaIndependentFlag syntax element, the number of predominant sound components syntax element and the number of ambient HOA coefficients syntax element to achieve a target bitrate.

Although not shown in the example of <FIG>, the local affiliate <NUM> may include further devices to compress the video data <NUM>. Moreover, although described as being distinct devices (e.g., the SAE device <NUM>, the CE device <NUM>, the CD device <NUM>, the PAE device <NUM>, the APB device <NUM>, and a VPB device <NUM> described below in more detail, etc.), the various devices may be implemented as distinct units or hardware within one or more devices.

The content consumer <NUM> shown in the example of <FIG> includes the audio playback device <NUM> described above with respect to the example of <FIG> (shown as "APB device <NUM>" in the example of <FIG>) and a video playback (VPB) device <NUM>. The APB device <NUM> may operate as described above with respect to <FIG> to generate multi-channel audio data <NUM> that are output to speakers <NUM> (which may refer to loudspeakers or speakers integrated into headphones, earbuds, headsets - which include headphones but also may include transducers to detect spoken or other audio signals, etc.). The VPB device <NUM> may represent a device configured to playback video data <NUM>, and may include video decoders, frame buffers, displays, and other components configured to playback video data <NUM>.

System 800B shown in the example of <FIG> is similar to the system 800A of <FIG> except that the remote truck <NUM> includes an additional device <NUM> configured to perform modulation with respect to the sideband information (SI) 15B of the bitstream <NUM> (where the other <NUM> channels are denoted as "channels 15A" or "transport channels 15A"). The additional device <NUM> is shown in the example of <FIG> as "mod device <NUM>. " The modulation device <NUM> may perform modulation of sideband information <NUM> to potentially reduce clipping of the sideband information and thereby reduce signal loss.

<FIG> are block diagrams illustrating different examples of a system that may be configured to perform various aspects of the techniques described in this disclosure. The system 410A shown in <FIG> is similar to the system <NUM> of <FIG>, except that the microphone array <NUM> of the system <NUM> is replaced with a microphone array <NUM>. The microphone array <NUM> shown in the example of <FIG> includes the HOA transcoder <NUM> and the spatial audio encoding device <NUM>. As such, the microphone array <NUM> generates the spatially compressed HOA audio data <NUM>, which is then compressed using the bitrate allocation in accordance with various aspects of the techniques set forth in this disclosure.

The system 410B shown in <FIG> is similar to the system 410A shown in <FIG> except that an automobile <NUM> includes the microphone array <NUM>. As such, the techniques set forth in this disclosure may be performed in the context of automobiles.

The system 410C shown in <FIG> is similar to the system 410A shown in <FIG> except that a remotely-piloted and/or autonomous controlled flying device <NUM> includes the microphone array <NUM>. The flying device <NUM> may for example represent a quadcopter, a helicopter, or any other type of drone. As such, the techniques set forth in this disclosure may be performed in the context of drones.

The system 410D shown in <FIG> is similar to the system 410A shown in <FIG> except that a robotic device <NUM> includes the microphone array <NUM>. The robotic device <NUM> may for example represent a device that operates using artificial intelligence, or other types of robots. In some examples, the robotic device <NUM> may represent a flying device, such as a drone. In other examples, the robotic device <NUM> may represent other types of devices, including those that do not necessarily fly. As such, the techniques set forth in this disclosure may be performed in the context of robots.

<FIG> is a block diagram illustrating another example of a system that may be configured to perform various aspects of the techniques described in this disclosure. The system shown in <FIG> is similar to the system <NUM> of <FIG> except that the broadcasting network <NUM> includes an additional HOA mixer <NUM>. As such, the system shown in <FIG> is denoted as system <NUM>' and the broadcast network of <FIG> is denoted as broadcast network <NUM>'. The HOA transcoder <NUM> may output the live feed HOA coefficients as HOA coefficients 11A to the HOA mixer <NUM>. The HOA mixer represents a device or unit configured to mix HOA audio data. HOA mixer <NUM> may receive other HOA audio data 11B (which may be representative of any other type of audio data, including audio data captured with spot microphones or non-3D microphones and converted to the spherical harmonic domain, special effects specified in the HOA domain, etc.) and mix this HOA audio data 11B with HOA audio data 11A to obtain HOA coefficients <NUM>.

<FIG> is a diagram illustrating an example of the psychoacoustic audio encoding device <NUM> shown in the examples of <FIG>. As shown in the example of <FIG>, the psychoacoustic audio encoding device <NUM> may include a spatial audio encoding unit <NUM>, a psychoacoustic audio encoding unit <NUM>, and a packetizer unit <NUM>.

The spatial audio encoding unit <NUM> may represent a unit configured to perform further spatial audio encoding with respect to the intermediately formatted audio data <NUM>. The spatial audio encoding unit <NUM> may include an extraction unit <NUM>, a demodulation unit <NUM> and a selection unit <NUM>.

The extraction unit <NUM> may represent a unit configured to extract the transport channels 15A and the modulated sideband information 15B from the intermediately formatted bitstream <NUM>. The extraction unit <NUM> may output the transport channels 15A to the selection unit <NUM>, and the modulated sideband information 15B to the demodulation unit <NUM>.

The demodulation unit <NUM> may represent a unit configured to demodulate the modulated sideband information 15B to recover the original sideband information 15B. The demodulation unit <NUM> may operate in a manner reciprocal to the operation of the modulation device <NUM> described above with respect to system 800B shown in the example of <FIG>. When modulation is not performed with respect to the sideband information 15B, the extraction unit <NUM> may extract the sideband information 15B directly from the intermediately formatted bitstream <NUM> and output the sideband information 15B directly to the selection unit <NUM> (or the demodulation unit <NUM> may pass through the sideband information 15B to the selection unit <NUM> without performing demodulation).

The selection unit <NUM> may represent a unit configured to select, based on configuration information <NUM> - which may represent an example of the above noted preferred priority, target bitrate, the above described independency flag (which may be denoted by an hoaIndependencyFlag syntax element), and/or other types of data externally defined - and the priority information, subsets of the transport channels 15A and the sideband information 15B.

The selection unit <NUM> may output the selected ambient HOA coefficients and predominant audio signals to the PAE unit <NUM> as transport channels 701A. The selection unit <NUM> may output the selected spatial components to the packetizer unit <NUM> as spatial components <NUM>. The techniques enable the selection unit <NUM> to select various combinations of the transport channels 15A and the sideband information 15B suitable to achieve, as one example, the target bitrate and independency set forth by the configuration information <NUM> by virtue of the spatial audio encoding device <NUM> providing the transport channels 15A and the sideband information 15B along with the priority information.

The PAE unit <NUM> may represent a unit configured to perform psychoacoustic audio encoding with respect to the transport channels 701A to generate encoded transport channels 701B. The PAE unit <NUM> may output the encoded transport channels 701B to the packetizer unit <NUM>. The packetizer unit <NUM> may represent a unit configured to generate, based on the encoded transport channels 701B and the sideband information <NUM>, the bitstream <NUM> as a series of packets for delivery to the content consumer <NUM>.

<FIG> is a diagram illustrating various aspects of the spatial audio encoding device of <FIG> in perform various aspects of the techniques described in this disclosure. In the example of <FIG>, microphone <NUM> captures audio signals representative of HOA audio data, which the spatial audio encoder device <NUM> reduces to a number of different sound components 750A-750N ("sound components <NUM>") and corresponding spatial components 752A-752N ("spatial components <NUM>"), where the spatial components may generally refer to both the spatial components corresponding to predominant sound components and the corresponding repurposed sound components.

As shown in a table <NUM>, the unified data object format, which may be referred to as a "V-vector based HOA transport format" (VHTF) or "vector based HOA transport format" in the case bitstreams, may include an audio object (which again is another way to refer to a sound component), and a corresponding spatial component (which may be referred to as a "vector"). The audio object (shown as "audio" in the example of <FIG>) may be denoted by the variable Ai, where i denotes the i-th audio object. The vector (shown as "V-vector" in the example of <FIG>) is denoted by the variable Vi, where i denotes the i-th vector. Ai is an Lx1 column matrix (with L being the number of samples in the frame), and Vi is a Mx1 column matrix (with M being the number of elements in the vector).

The reconstructed HOA coefficients <NUM>' may be denoted as H̃. The reconstructed HOA coefficients <NUM>' may be determined according to the following equation: <MAT>.

According to the above equation, N denotes a total number of sound components in the selected non-zero subset of the plurality of spatial components. The reconstructed HOA coefficients <NUM>' (H̃) may be determined as a summation of each iterative (up to N-<NUM> starting at zero) multiplication the audio object (Ai) by the transpose of the vector (ViT). The spatial audio encoding device <NUM> may specify the bitstream <NUM> as shown at the bottom of <FIG>, where the audio objects <NUM> are specified along with corresponding spatial components <NUM> in each frame (denoted by T = <NUM> for the first frame, T = <NUM> for the second frame, etc.).

<FIG> are diagrams illustrating different representations within the bitstream according to various aspects of the unified data object format techniques described in this disclosure. In the example of <FIG>, the HOA coefficients <NUM> are shown as "input", which the spatial audio encoding device <NUM> shown in the example of <FIG> may transform into a VHTF representation <NUM> as described above. The VHTF representation <NUM> in the example of <FIG> represents the predominant sound (or foreground - FG - sound) representation. The table <NUM> is further shown to illustrate the VHTF representation <NUM> in more detail. In the example of <FIG>, there is also spatial representations <NUM> of the different V-vectors to illustrate how the spatial component defines shape, widths, and directions of the corresponding spatial component.

In the example of <FIG>, the HOA coefficients <NUM> are shown as "input", which the spatial audio encoding device <NUM> shown in the example of <FIG> may transform into a VHTF representation <NUM> as described above. The VHTF representation <NUM> in the example of <FIG> represents the ambient sound (or background - BG - sound) representation. The table <NUM> is further shown to illustrate the VHTF representation <NUM> in more detail, where both the VHTF representation <NUM> and the VHTF representation <NUM> have the same format. In the example of <FIG>, there is also examples <NUM> of the different repurposed V-vectors to illustrate how the repurposed V-vectors may include a single element with a value of one with every other element being set to a value of zero so as to, as described above, identify the order and sub-order of the spherical basis function to which the ambient HOA coefficient corresponds.

In the example of <FIG>, the HOA coefficients <NUM> are shown as "input", which the spatial audio encoding device <NUM> shown in the example of <FIG> may transform into a VHTF representation <NUM> as described above. The VHTF representation <NUM> in the example of <FIG> represents the sound components, but also includes the priority information <NUM> (shown as "PriorityOfTC," which refers to a priority of transport channels). The table <NUM> is updated in <FIG> to further illustrate the VHTF representation <NUM> in more detail, where both the VHTF representation <NUM> and the VHTF representation <NUM> have the same format and VHTF representation <NUM> includes the priority information <NUM>.

In each instance, the spatial audio encoding device <NUM> may specify the unified transport type (or, in other words, the VHTF) by setting the HoaTransportType syntax element in the following table to <NUM>.

As noted in the below table, the HoaTransportType indicates the HOA transport mode, and when set to a value of three (<NUM>) signals that the transport type is VHTF.

Regarding the VHTF (HoaTransportType = <NUM>), <FIG> and <FIG>may illustrate how VHTF is composed of audio signals, {Ai}, and the associated V-vectors, {Vi}, where an input HOA signal, H, can be approximated by <MAT> where an i-th V-vector, Vi, is the spatial representation of the i-th audio signal, Ai. N is the number of transport channels. The dynamic range of each Vi is bound by [-<NUM>, <NUM>]. Examples of V-vector based spatial representation <NUM> are shown in <FIG>.

VHTF can also represent an original input HOA, which means H̃ =H, in the following conditions:.

Thus, VHTF can represent both pre-dominant and ambient sound fields.

As shown in Table <NUM>, the HOAFrame_VvecTransportFormat() holds the information that is required to decode the L samples (HoaFrameLength in Table <NUM>) of an HOA frame.

Syntax of HOAFrame_VvecTransportFormat()
<IMG>.

In the foregoing syntax tables, Vvector[i][j] refers to the spatial component, where i identifies which transport channel, and j identifies which coefficient (by way of the order and sub-order of the spherical basis function to which the ambient HOA coefficient corresponds in the case when Vvector represents the repurposed spatial component).

The audio decoding device <NUM> (shown in the example of <FIG>) may receive the bitstream <NUM> and obtain the HoaTransportType syntax element from the bitstream <NUM>. Based on the HoaTransportType syntax element, the audio decoding device <NUM> may extract the various sound components and corresponding spatial components to render the speaker feeds in the manner described above in more detail.

<FIG> are diagrams illustrating various ways by which the spatial audio encoding device of <FIG> may determine the priority information in accordance with various aspects of the techniques described in this disclosure. In the example of <FIG>, the spatial audio encoding device <NUM> may determine an HOA representation of the sound component (which is denoted as Hi) in the manner described above (<NUM>). The spatial audio encoding device <NUM> may next determine an energy (denoted by the variable Ei) of HOA representation of the sound component (<NUM>). The spatial audio encoding device <NUM> may also determine, based on the spatial component (denoted by the variable Vi), a spatial weighting (denoted by the variable Wi) (<NUM>). The spatial audio encoding device <NUM> may obtain, based on the energy and the spatial weighting, the priority information (<NUM>).

In the example of <FIG>, the spatial audio encoding device <NUM> may determine an HOA representation of the sound component (which is denoted as Hi) in the manner described above (<NUM>). The spatial audio encoding device <NUM> may next render the HOA representation of the sound component to one or more speaker feeds (which may refer to, as one example, the shown "loudspeaker output") (<NUM>). The spatial audio encoding device <NUM> may determine an energy (denoted by the variable Ei) of one or more speaker feeds (<NUM>). The spatial audio encoding device <NUM> may also determine, based on the spatial component (denoted by the variable Vi), a spatial weighting (denoted by the variable Wi) (<NUM>). The spatial audio encoding device <NUM> may obtain, based on the energy and the spatial weighting, the priority information (<NUM>).

In the example of <FIG>, the spatial audio encoding device <NUM> may determine an HOA representation of the sound component (which is denoted as Hi) in the manner described above (<NUM>). The spatial audio encoding device <NUM> may next determine a loudness measure (denoted by the variable Li) of HOA representation of the sound component (<NUM>). The spatial audio encoding device <NUM> may also determine, based on the spatial component (denoted by the variable Vi), a spatial weighting (denoted by the variable Wi) (<NUM>). The spatial audio encoding device <NUM> may obtain, based on the loudness measure and the spatial weighting, the priority information (<NUM>).

In the example of <FIG>, the spatial audio encoding device <NUM> may determine an HOA representation of the sound component (which is denoted as Hi) in the manner described above (<NUM>). The spatial audio encoding device <NUM> may next render the HOA representation of the sound component to one or more speaker feeds (which may refer to, as one examples, the shown "loudspeaker output") (<NUM>). The spatial audio encoding device <NUM> may determine a loudness measure (denoted by the variable Li) of one or more speaker feeds (<NUM>). The spatial audio encoding device <NUM> may also determine, based on the spatial component (denoted by the variable Vi), a spatial weighting (denoted by the variable Wi) (<NUM>). The spatial audio encoding device <NUM> may obtain, based on the loudness measure and the spatial weighting, the priority information (<NUM>).

In the example of <FIG>, the spatial audio encoding device <NUM> may determine an HOA representation of the sound component (which is denoted as Hi) in the manner described above (<NUM>). The spatial audio encoding device <NUM> may next determine a loudness measure (denoted by the variable Li) of the HOA representation of the sound component (<NUM>). The spatial audio encoding device <NUM> may also determine, based on the spatial component (denoted by the variable Vi), a spatial weighting. The spatial audio encoding device <NUM> may also determine the above noted continuity indication, the class resulting from signal classification, and the content provider preferred priority (which is shown as "content provider driven priority"), integrating the above noted continuity indication, the class resulting from signal classification, and the content provider preferred priority into the spatial weighting (denoted by the variable Wi) (<NUM>). The spatial audio encoding device <NUM> may obtain, based on the loudness measure and the spatial weighting, the priority information (<NUM>).

In the example of <FIG>, the spatial audio encoding device <NUM> may determine an HOA representation of the sound component (which is denoted as Hi) in the manner described above (<NUM>). The spatial audio encoding device <NUM> may next render the HOA representation of the sound component to one or more speaker feeds (which may refer to, as one example, the shown "loudspeaker output") (<NUM>). The spatial audio encoding device <NUM> may determine a loudness measure (denoted by the variable Li) of one or more speaker feeds (<NUM>). The spatial audio encoding device <NUM> may also determine, based on the spatial component (denoted by the variable Vi), a spatial weighting. The spatial audio encoding device <NUM> may also determine the above noted continuity indication, the class resulting from signal classification, and the content provider preferred priority (which is shown as "content provider driven priority"), integrating the above noted continuity indication, the class resulting from signal classification, and the content provider preferred priority into the spatial weighting (denoted by the variable Wi) (<NUM>). The spatial audio encoding device <NUM> may obtain, based on the loudness measure and the spatial weighting, the priority information (<NUM>).

<FIG> is a block diagram illustrating a different system configured to perform various aspects of the techniques described in this disclosure. In the example of <FIG>, a system <NUM> includes a microphone array <NUM> and computing devices <NUM> and <NUM>. The microphone array <NUM> may be similar, if not substantially similar, to the microphone array <NUM> described above with respect to the example of <FIG>. The microphone array <NUM> includes the HOA transcoder <NUM> and the mezzanine encoder <NUM> discussed in more detail above.

The computing devices <NUM> and <NUM> may each represent one or more of a cellular phone (which may be interchangeably be referred to as a "mobile phone," or "mobile cellular handset" and where such cellular phone may including so-called "smart phones"), a tablet, a laptop, a personal digital assistant, a wearable computing headset, a watch (including a so-called "smart watch"), a gaming console, a portable gaming console, a desktop computer, a workstation, a server, or any other type of computing device. For purposes of illustration, each of the computing devices <NUM> and <NUM> is referred to a respective mobile phone <NUM> and <NUM>. In any event, the mobile phone <NUM> may include the emission encoder <NUM>, while the mobile phone <NUM> may include the audio decoding device <NUM>.

The microphone array <NUM> may capture audio data in the form of microphone signals <NUM>. The HOA transcoder <NUM> of the microphone array <NUM> may transcode the microphone signals <NUM> into the HOA coefficients <NUM>, which the mezzanine encoder <NUM> (shown as "mezz encoder <NUM>") may encode (or, in other words, compress) to form the bitstream <NUM> in the manner described above. The microphone array <NUM> may be coupled (either wirelessly or via a wired connection) to the mobile phone <NUM> such that the microphone array <NUM> may communicate the bitstream <NUM> via a transmitter and/or receiver (which may also be referred to as a transceiver, and abbreviated as "TX") 910A to the emission encoder <NUM> of the mobile phone <NUM>. The microphone array <NUM> may include the transceiver 910A, which may represent hardware or a combination of hardware and software (such as firmware) configured to transmit data to another transceiver.

The emission encoder <NUM> may operate in the manner described above to generate the bitstream <NUM> conforming to the 3D Audio Coding Standard from the bitstream <NUM>. The emission encoder <NUM> may include a transceiver 910B (which is similar to if not substantially similar to transceiver 910A) configured to receive the bitstream <NUM>. The emission encoder <NUM> may select the target bitrate, hoaIndependencyFlag syntax element, and the number of transport channels when generating the bitstream <NUM> from the received bitstream <NUM> (selecting the number of transport channels as the subset of transport channels according to the priority information). The emission encoder <NUM> may communicate (although not necessarily directly, meaning that such communication may have intervening devices, such as servers, or by way of dedicated non-transitory storage media, etc.) the bitstream <NUM> via the transceiver 910B to the mobile phone <NUM>.

The mobile phone <NUM> may include transceiver 910C (which is similar to if not substantially similar to transceivers 910A and 910B) configured to receive the bitstream <NUM>, whereupon the mobile phone <NUM> may invoke audio decoding device <NUM> to decode the bitstream <NUM> so as to recover the HOA coefficients <NUM>'. Although not shown in <FIG> for ease of illustration purposes, the mobile pohne <NUM> may render the HOA coefficients <NUM>' to speaker feeds, and reproduce the soundfield via a speaker (e.g., a loudspeaker integrated into the mobile phone <NUM>, a loudspeaker wirelessly coupled to the mobile pohone <NUM>, a loudspeaker coupled by wire to the mobile phone <NUM>, or a headphone speaker coupled either wirelessly or via wired connection to the mobile phone <NUM>) based on the speaker feeds. For reproducing the soundfield by way of headphone speakers (which again may be standalone headphones or headphones integrated into a headset), the mobile phone <NUM> may render binaural audio speaker feeds from either the loudspeaker feeds or directly from the HOA coefficients <NUM>'.

<FIG> is a flowchart illustrating example operation of the psychoacoustic audio encoding device of <FIG> in performing various aspects of the techniques described in this disclosure. The psychoacoustic audio encoding device <NUM> may first obtain a first data object <NUM> representative of a compressed version of higher order ambisonic coefficients (<NUM>). The psychoacoustic audio encoding device <NUM> may obtain, from the first data object <NUM>, a plurality of sound components <NUM> (shown in the example of <FIG>) and priority information <NUM> (shown in the example of <FIG>) indicative of a priority of each of the plurality of sound components relative to remaining ones of the sound components (<NUM>).

The psychoacoustic audio encoding device <NUM> may select, based on the priority information <NUM>, a non-zero subset of the plurality of sound components (<NUM>). In some examples, the psychoacoustic audio encoding device <NUM> may select the non-zero subset of the plurality of sound components to achieve a target bitrate. The psychoacoustic audio encoding device <NUM> may next specify, in a second data object <NUM> different that the first data object <NUM>, the selected non-zero subset of the plurality of sound components (<NUM>).

In some examples, the first data object <NUM> comprises a first bitstream <NUM>, where the first bitstream <NUM> comprises a first plurality of transport channels. The second data object <NUM> may comprise a second bitstream <NUM>, where the second bitstream <NUM> comprises a second plurality of transport channels. In this and other examples, the priority information <NUM> comprises priority channel information <NUM>, and the psychoacoustic audio encoding device <NUM> may obtain, from the first plurality of transport channels, the plurality of sound components, and specify, in each of the second plurality of transport channels, a respective one of the selected non-zero subset of the plurality of sound components.

In some examples, the first data object <NUM> comprises a first file <NUM>, where the first file <NUM> comprises a first plurality of tracks. The second data object <NUM> may comprise a second file <NUM>, where the second file <NUM> comprises a second plurality of tracks. In this and other examples, the priority information <NUM> comprises priority track information <NUM>, and the psychoacoustic audio encoding device <NUM> may obtain, from the first plurality of tracks, the plurality of sound components, and specify, in each of the second plurality of tracks, a respective one of the selected non-zero subset of the plurality of sound components.

In some examples, the first data object <NUM> comprises a bitstream <NUM>, and the second data object <NUM> comprises a file <NUM>. In other examples, the first data object <NUM> comprises a file <NUM>, and the second data object <NUM> comprises a bitstream <NUM>. That is, various aspects of the techniques may allow for conversion between different types of data objects.

<FIG> is a flowchart illustrating example operation of the spatial audio encoding device of <FIG> in performing various aspects of the techniques described in this disclosure. As shown in the example of <FIG>, the spatial audio encoding device <NUM> (shown in the example of <FIG>) may, as described above, decompose the HOA coefficients <NUM> into a sound component and a corresponding spatial component (<NUM>). The spatial audio encoding device <NUM> may next determine, based on one or more of the sound component and the corresponding spatial component, priority information indicative of a priority of the sound component relative to other sound components of the soundfield represented by the HOA coefficients <NUM>, as described above in more detail.

The spatial audio encoding device <NUM> may specify, in the data object (e.g., bitstream <NUM>) representative of a compressed version of the HOA coefficients <NUM>, the sound component and the priority information (<NUM>). In some examples, the spatial audio encoding device <NUM> may specify a plurality of sound components and priority information indicative of a priority of each of the plurality of sound components relative to remaining ones of the sound components.

In this manner, higher Order Ambisonics (HOA) signals are able to deliver a significantly enhanced immersive sound compared to conventional stereo or <NUM> channel audio signals. However, there are some use cases where HOA signals cannot be transported because of the large number of HOA input channels.

One use case is mobile devices as shown in Figure <NUM> (a), where the number of input channels can be limited by <NUM> Pulse-Code Modulation (PCM) channels, and thus only a maximum of 1st order HOA (requiring <NUM> PCM channels) can be transported.

Another use case is a typical broadcast workflow. Here, a contribution encoder can transmit <NUM> PCM channels from the remote truck to the network operation centre (NOC) or local affiliate(s). However, the use of single High-Definition Serial Digital Interface (HD-SDI) link has a limitation of being able to transport only <NUM> PCM channels. This restricts the transport to a maximum of 3rd order HOA signals (requiring <NUM> PCM channels). If additional audio elements are to be transported, only a maximum of 2nd order HOA (requiring <NUM> PCM channels) can be transported.

The above described techniques may address these use cases in various ways as discussed in more detail above.

In addition, the foregoing techniques may be performed with respect to any number of different contexts and audio ecosystems and should not be limited to any of the contexts or audio ecosystems described above. A number of example contexts are described below, although the techniques should be limited to the example contexts. One example audio ecosystem may include audio content, movie studios, music studios, gaming audio studios, channel based audio content, coding engines, game audio stems, game audio coding / rendering engines, and delivery systems.

The movie studios, the music studios, and the gaming audio studios may receive audio content. In some examples, the audio content may represent the output of an acquisition. The movie studios may output channel based audio content (e.g., in <NUM>, <NUM>, and <NUM>) such as by using a digital audio workstation (DAW). The music studios may output channel based audio content (e.g., in <NUM>, and <NUM>) such as by using a DAW. In either case, the coding engines may receive and encode the channel based audio content based one or more codecs (e.g., AAC, AC3, Dolby True HD, Dolby Digital Plus, and DTS Master Audio) for output by the delivery systems. The gaming audio studios may output one or more game audio stems, such as by using a DAW. The game audio coding / rendering engines may code and or render the audio stems into channel based audio content for output by the delivery systems. Another example context in which the techniques may be performed comprises an audio ecosystem that may include broadcast recording audio objects, professional audio systems, consumer on-device capture, HOA audio format, on-device rendering, consumer audio, TV, and accessories, and car audio systems.

The broadcast recording audio objects, the professional audio systems, and the consumer on-device capture may all code their output using HOA audio format. In this way, the audio content may be coded using the HOA audio format into a single representation that may be played back using the on-device rendering, the consumer audio, TV, and accessories, and the car audio systems. In other words, the single representation of the audio content may be played back at a generic audio playback system (i.e., as opposed to requiring a particular configuration such as <NUM>, <NUM>, etc.), such as audio playback system <NUM>.

Other examples of context in which the techniques may be performed include an audio ecosystem that may include acquisition elements, and playback elements. The acquisition elements may include wired and/or wireless acquisition devices (e.g., Eigen microphones), on-device surround sound capture, and mobile devices (e.g., smartphones and tablets). In some examples, wired and/or wireless acquisition devices may be coupled to mobile device via wired and/or wireless communication channel(s).

In accordance with one or more techniques of this disclosure, the mobile device (such as a mobile communication handset) may be used to acquire a soundfield. For instance, the mobile device may acquire a soundfield via the wired and/or wireless acquisition devices and/or the on-device surround sound capture (e.g., a plurality of microphones integrated into the mobile device). The mobile device may then code the acquired soundfield into the HOA coefficients for playback by one or more of the playback elements. For instance, a user of the mobile device may record (acquire a soundfield of) a live event (e.g., a meeting, a conference, a play, a concert, etc.), and code the recording into HOA coefficients.

The mobile device may also utilize one or more of the playback elements to playback the HOA coded soundfield. For instance, the mobile device may decode the HOA coded soundfield and output a signal to one or more of the playback elements that causes the one or more of the playback elements to recreate the soundfield. As one example, the mobile device may utilize the wireless and/or wireless communication channels to output the signal to one or more speakers (e.g., speaker arrays, sound bars, etc.). As another example, the mobile device may utilize docking solutions to output the signal to one or more docking stations and/or one or more docked speakers (e.g., sound systems in smart cars and/or homes). As another example, the mobile device may utilize headphone rendering to output the signal to a set of headphones, e.g., to create realistic binaural sound.

In some examples, a particular mobile device may both acquire a 3D soundfield and playback the same 3D soundfield at a later time. In some examples, the mobile device may acquire a 3D soundfield, encode the 3D soundfield into HOA, and transmit the encoded 3D soundfield to one or more other devices (e.g., other mobile devices and/or other non-mobile devices) for playback.

Yet another context in which the techniques may be performed includes an audio ecosystem that may include audio content, game studios, coded audio content, rendering engines, and delivery systems. In some examples, the game studios may include one or more DAWs which may support editing of HOA signals. For instance, the one or more DAWs may include HOA plugins and/or tools which may be configured to operate with (e.g., work with) one or more game audio systems. In some examples, the game studios may output new stem formats that support HOA. In any case, the game studios may output coded audio content to the rendering engines which may render a soundfield for playback by the delivery systems.

The techniques may also be performed with respect to exemplary audio acquisition devices. For example, the techniques may be performed with respect to an Eigen microphone which may include a plurality of microphones that are collectively configured to record a 3D soundfield. In some examples, the plurality of microphones of an Eigen microphone may be located on the surface of a substantially spherical ball with a radius of approximately <NUM>. In some examples, the audio encoding device <NUM> may be integrated into the Eigen microphone so as to output a bitstream <NUM> directly from the microphone.

Another exemplary audio acquisition context may include a production truck which may be configured to receive a signal from one or more microphones, such as one or more Eigen microphones. The production truck may also include an audio encoder, such as audio encoder <NUM> of <FIG>.

The mobile device may also, in some instances, include a plurality of microphones that are collectively configured to record a 3D soundfield. In other words, the plurality of microphones may have X, Y, Z diversity. In some examples, the mobile device may include a microphone which may be rotated to provide X, Y, Z diversity with respect to one or more other microphones of the mobile device. The mobile device may also include an audio encoder, such as audio encoder <NUM> of <FIG>.

A ruggedized video capture device may further be configured to record a 3D soundfield. In some examples, the ruggedized video capture device may be attached to a helmet of a user engaged in an activity. For instance, the ruggedized video capture device may be attached to a helmet of a user whitewater rafting. In this way, the ruggedized video capture device may capture a 3D soundfield that represents the action all around the user (e.g., water crashing behind the user, another rafter speaking in front of the user, etc..

The techniques may also be performed with respect to an accessory enhanced mobile device, which may be configured to record a 3D soundfield. In some examples, the mobile device may be similar to the mobile devices discussed above, with the addition of one or more accessories. For instance, an Eigen microphone may be attached to the above noted mobile device to form an accessory enhanced mobile device. In this way, the accessory enhanced mobile device may capture a higher quality version of the 3D soundfield than just using sound capture components integral to the accessory enhanced mobile device.

Example audio playback devices that may perform various aspects of the techniques described in this disclosure are further discussed below. In accordance with one or more techniques of this disclosure, speakers and/or sound bars may be arranged in any arbitrary configuration while still playing back a 3D soundfield. Moreover, in some examples, headphone playback devices may be coupled to a decoder <NUM> via either a wired or a wireless connection. In accordance with one or more techniques of this disclosure, a single generic representation of a soundfield may be utilized to render the soundfield on any combination of the speakers, the sound bars, and the headphone playback devices.

A number of different example audio playback environments may also be suitable for performing various aspects of the techniques described in this disclosure. For instance, a <NUM> speaker playback environment, a <NUM> (e.g., stereo) speaker playback environment, a <NUM> speaker playback environment with full height front loudspeakers, a <NUM> speaker playback environment, a <NUM> speaker playback environment, an automotive speaker playback environment, and a mobile device with ear bud playback environment may be suitable environments for performing various aspects of the techniques described in this disclosure.

In accordance with one or more techniques of this disclosure, a single generic representation of a soundfield may be utilized to render the soundfield on any of the foregoing playback environments. Additionally, the techniques of this disclosure enable a rendered to render a soundfield from a generic representation for playback on the playback environments other than that described above. For instance, if design considerations prohibit proper placement of speakers according to a <NUM> speaker playback environment (e.g., if it is not possible to place a right surround speaker), the techniques of this disclosure enable a render to compensate with the other <NUM> speakers such that playback may be achieved on a <NUM> speaker playback environment.

Moreover, a user may watch a sports game while wearing headphones. In accordance with one or more techniques of this disclosure, the 3D soundfield of the sports game may be acquired (e.g., one or more Eigen microphones may be placed in and/or around the baseball stadium), HOA coefficients corresponding to the 3D soundfield may be obtained and transmitted to a decoder, the decoder may reconstruct the 3D soundfield based on the HOA coefficients and output the reconstructed 3D soundfield to a renderer, and the renderer may obtain an indication as to the type of playback environment (e.g., headphones), and render the reconstructed 3D soundfield into signals that cause the headphones to output a representation of the 3D soundfield of the sports game.

In each of the various instances described above, it should be understood that the audio encoding device <NUM> may perform a method or otherwise comprise means to perform each step of the method for which the audio encoding device <NUM> is configured to perform In some instances, the means may comprise one or more processors, e.g., formed by fixed-function processing circuitry, programmable processing circuitry or a combination thereof. In some instances, the one or more processors may represent a special purpose processor configured by way of instructions stored to a non-transitory computer-readable storage medium. In other words, various aspects of the techniques in each of the sets of encoding examples may provide for a non-transitory computer-readable storage medium having stored thereon instructions that, when executed, cause the one or more processors to perform the method for which the audio encoding device <NUM> has been configured to perform.

Computer-readable media may include computer-readable storage media, which corresponds to a tangible medium such as data storage media.

Likewise, in each of the various instances described above, it should be understood that the audio decoding device <NUM> may perform a method or otherwise comprise means to perform each step of the method for which the audio decoding device <NUM> is configured to perform. In some instances, the means may comprise one or more processors, e.g., formed by fixed-function processing circuitry, programmable processing circuitry or a combination thereof. In some instances, the one or more processors may represent a special purpose processor configured by way of instructions stored to a non-transitory computer-readable storage medium. In other words, various aspects of the techniques in each of the sets of encoding examples may provide for a non-transitory computer-readable storage medium having stored thereon instructions that, when executed, cause the one or more processors to perform the method for which the audio decoding device <NUM> has been configured to perform.

By way of example, and not limitation, computer-readable storage media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage, or other magnetic storage devices, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer.

Claim 1:
A device (<NUM>) configured to decompress higher order ambisonic audio data representative of a soundfield, the device comprising:
a memory configured to store, at least in part, a data object representative of a compressed version of higher order ambisonic coefficients, the higher order ambisonic coefficients being representative of a soundfield, wherein the data object is of a vector based higher order ambisonic transport format;
and one or more processors configured to:
receive a bitstream (<NUM>) used to decode the data object, wherein the bitstream includes bits relating to a priority of an ith transport channel, where there are a plurality of transport channels;
obtain, from the data object and according to a format, an ambient higher order ambisonic coefficient descriptive of an ambient component of the soundfield; obtain a vector, which represents a repurposed spatial component, based on the priority of the ith transport channel, the repurposed spatial component corresponding to the ambient higher order ambisonic coefficient, the repurposed spatial component indicative of one or more of an order and sub-order of a spherical basis function to which the ambient higher order ambisonic coefficient corresponds;
obtain, from the data object and according to the same format, the predominant sound component; and
obtain, from the data object, a corresponding spatial component defined in a spherical harmonic domain.