Patent Description:
During production of audio content, the sound engineer may render the audio content using a specific renderer in an attempt to tailor the audio content for target configurations of speakers used to reproduce the audio content. In other words, the sound engineer may render the audio content and playback the rendered audio content using speakers arranged in the targeted configuration. The sound engineer may then remix various aspects of the audio content, render the remixed audio content and again playback the rendered, remixed audio content using the speakers arranged in the targeted configuration. The sound engineer may iterate in this manner until a certain artistic intent is provided by the audio content. In this way, the sound engineer may produce audio content that provides a certain artistic intent or that otherwise provides a certain sound field during playback (e.g., to accompany video content played along with the audio content).

<CIT> discloses techniques for determining an audio renderer that suits a particular local speaker geometry. The techniques determine the local speaker geometry and then determine a renderer for rendering spherical harmonic coefficients (SHC) signals based on the determined local speaker geometry. The rendering device may select from among a number of different renderers, e.g., a mono renderer, a stereo renderer, a horizontal only renderer, or a three-dimensional renderer.

US patent application <CIT> discloses techniques for rendering low frequency speaker feeds from higher-order ambisonic coefficients.

International patent application <CIT> discloses an audio processing apparatus comprising a renderer which is capable of rendering audio components in accordance with a plurality of rendering modes.

In general, techniques are described for specifying audio rendering information in a bitstream representative of audio data. In various examples, the techniques of this disclosure provide for ways by which to signal audio renderer-selection information used during audio content production to a playback device. The playback device may, in turn use the signaled audio renderer-selection information to select one or more renderers, and use the selected renderer(s) to render the audio content. Providing the rendering information in this manner enables the playback device to render the audio content in a manner intended by the sound engineer, and thereby potentially ensure appropriate playback of the audio content such that the artistic intent is preserved and understood by a listener.

In other words, the rendering information used during rendering by the sound engineer is provided in accordance with the techniques described in this disclosure so that the audio playback device may utilize the rendering information to render the audio content in a manner intended by the sound engineer, thereby ensuring a more consistent experience during both production and playback of the audio content in comparison to systems that do not provide this audio rendering information. Moreover, the techniques of this disclosure enable the playback to leverage both object-based and ambisonic representations of a soundfield, in preserving the artistic intent of the soundfield. That is, a content creator device or content producer device may implement the techniques of this disclosure to signal renderer-identifying information to the playback device, thereby enabling the playback to device to select the appropriate renderer for a pertinent portion of the soundfield-representative audio data.

The present disclosure provides methods and devices according to the appended independent claims. Specific embodiments are subject of the dependent claims. The invention is defined solely by the appended claims.

The details of one or more aspects of the techniques are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the techniques will be apparent from the description and drawings, and from the claims.

There are a number of different ways to represent a soundfield. Example formats include channel-based audio formats, object-based audio formats, and scene-based audio formats. Channel-based audio formats refer to the <NUM> surround sound format, <NUM> surround sound formats, <NUM> surround sound formats, or any other channel-based format that localizes audio channels to particular locations around the listener in order to recreate a soundfield.

Object-based audio formats may refer to formats in which audio objects, often encoded using pulse-code modulation (PCM) and referred to as PCM audio objects, are specified in order to represent the soundfield. Such audio objects may include metadata identifying a location of the audio object relative to a listener or other point of reference in the soundfield, such that the audio object may be rendered to one or more speaker channels for playback in an effort to recreate the soundfield. The techniques described in this disclosure may apply to any of the foregoing formats, including scene-based audio formats, channel-based audio formats, object-based audio formats, or any combination thereof.

Scene-based audio formats may include a hierarchical set of elements that define the soundfield in three dimensions. One example of a hierarchical set of elements is a set of spherical harmonic coefficients (SHC). The following expression demonstrates a description or representation of a soundfield using SHC: <MAT>.

The expression shows that the pressure pi at any point {rr, θr, φr} of the soundfield, at time t, can be represented uniquely by the SHC, <MAT>. Here, <MAT>, c is the speed of sound (~<NUM>/s), {rr, θr, φr} is a point of reference (or observation point), jn(·) is the spherical Bessel function of order n, and <MAT> are the spherical harmonic basis functions (which may also be referred to as a spherical basis function) of order n and suborder m. It can be recognized that the term in square brackets is a frequency-domain representation of the signal (i.e., S(ω, rr, θr, φr)) which can be approximated by various time-frequency transformations, such as the discrete Fourier transform (DFT), the discrete cosine transform (DCT), or a wavelet transform. Other examples of hierarchical sets include sets of wavelet transform coefficients and other sets of coefficients of multiresolution basis functions.

The SHC <MAT> can either be physically acquired (e.g., recorded) by various microphone array configurations or, alternatively, they can be derived from channel-based or object-based descriptions of the soundfield. The SHC (which also may be referred to as ambisonic coefficients) represent scene-based audio, where the SHC may be input to an audio encoder to obtain encoded SHC that may promote more efficient transmission or storage. For example, a fourth-order representation involving (<NUM>+<NUM>)<NUM> (<NUM>, and hence fourth order) coefficients may be used.

As noted above, the SHC may be derived from a microphone recording using a microphone array. Various examples of how SHC may be physically acquired from microphone arrays are described in <NPL>.

The following equation may illustrate how the SHCs may be derived from an object-based description. The coefficients <MAT> for the soundfield corresponding to an individual audio object may be expressed as: <MAT> where i is <MAT>, <MAT> is the spherical Hankel function (of the second kind) of order n, and {rs, θs, φs} is the location of the object. Knowing the object source energy g(ω) as a function of frequency (e.g., using time-frequency analysis techniques, such as performing a fast Fourier transform on the pulse code modulated - PCM - stream) may enable conversion of each PCM object and the corresponding location into the SHC <MAT>. Further, it can be shown (since the above is a linear and orthogonal decomposition) that the <MAT> coefficients for each object are additive. In this manner, a number of PCM objects can be represented by the <MAT> coefficients (e.g., as a sum of the coefficient vectors for the individual objects). The coefficients may contain information about the soundfield (the pressure as a function of 3D coordinates), and the above represents the transformation from individual objects to a representation of the overall soundfield, in the vicinity of the observation point {rr, θr, φr}.

<FIG> is a diagram illustrating a system <NUM> that may perform various aspects of the techniques described in this disclosure. As shown in the example of <FIG>, the system <NUM> includes a content creator device <NUM> and a content consumer device <NUM>. While described in the context of the content creator device <NUM> and the content consumer device <NUM>, the techniques may be implemented in any context in which SHCs (which may also be referred to as ambisonic coefficients) or any other hierarchical representation of a soundfield are encoded to form a bitstream representative of the audio data. Moreover, the content creator device <NUM> may represent any form of computing device capable of implementing the techniques described in this disclosure, including a handset (or cellular phone), a tablet computer, a smart phone, or a desktop computer to provide a few examples. Likewise, the content consumer device <NUM> may represent any form of computing device capable of implementing the techniques described in this disclosure, including a handset (or cellular phone), a tablet computer, a smart phone, a set-top box, or a desktop computer to provide a few examples.

The content creator device <NUM> may be operated by a movie studio or other entity that may generate multi-channel audio content for consumption by operators of content consumer devices, such as the content consumer device <NUM>. In some examples, the content creator device <NUM> may be operated by an individual user who would like to compress ambisonic coefficients 11B ("AMB COEFFS 11B").

The ambisonic coefficients 11B may take a number of different forms. For instance, the microphone 5B may use a coding scheme for ambisonic representations of a soundfield, referred to as Mixed Order Ambisonics (MOA) as discussed in more detail in <CIT>, and published as<CIT>.

To generate a particular MOA representation of the soundfield, the microphone 5B may generate a partial subset of the full set of ambisonic coefficients. For instance, each MOA representation generated by the microphone 5B may provide precision with respect to some areas of the soundfield, but less precision in other areas. In one example, an MOA representation of the soundfield may include eight (<NUM>) uncompressed ambisonic coefficients, while the third order ambisonic representation of the same soundfield may include sixteen (<NUM>) uncompressed ambisonic coefficients. As such, each MOA representation of the soundfield that is generated as a partial subset of the ambisonic coefficients may be less storage-intensive and less bandwidth intensive (if and when transmitted as part of the bitstream <NUM> over the illustrated transmission channel) than the corresponding third order ambisonic representation of the same soundfield generated from the ambisonic coefficients.

Another example form of ambisonic coefficients includes a first-order ambisonic (FOA) representations in which all of the ambisonic coefficients associated with a first order spherical basis function and a zero order spherical basis function are used to represent the soundfield. In other words, rather than represent the soundfield using a partial, non-zero subset of the ambisonic coefficients, the microphone 5B may represent the soundfield using all of the ambisonic coefficients for a given order N, resulting in a total of ambisonic coefficients equaling (N+<NUM>)<NUM>.

In this respect, the ambisonic audio data (which is another way to refer to the ambisonic coefficients in either MOA representations or full order representations, such as the first-order representation noted above) may include ambisonic coefficients associated with spherical basis functions having an order of one or less (which may be referred to as "<NUM>st order ambisonic audio data"), ambisonic coefficients associated with spherical basis functions having a mixed order and suborder (which may be referred to as the "MOA representation" discussed above), or ambisonic coefficients associated with spherical basis functions having an order greater than one (which is referred to above as the "full order representation").

In any event, the content creator may generate audio content (including the ambisonic coefficients in one or more of the above noted forms) in conjunction with video content. The content consumer device <NUM> may be operated by an individual. The content consumer device <NUM> may include an audio playback system <NUM>, which may refer to any form of audio playback system capable of rendering SHC (such as the ambisonic coefficients 11B) for play back as multi-channel audio content.

The content creator device <NUM> includes an audio editing system <NUM>. The content creator device <NUM> may obtain live recordings <NUM> in various formats (including directly as ambisonic coefficients, as object-based audio, etc.) and audio objects <NUM>, which the content creator device <NUM> may edit using audio editing system <NUM>. The microphone 5A and/or the microphone 5B (the "microphones <NUM>") may capture the live recordings <NUM>. In the example of <FIG>, the microphone 5A represents a microphone or set of microphones that are configured or otherwise operable to capture audio data and generate object-based and/or channel-based signals representing the captured audio data. As such, the live recordings <NUM> may represent, in various use case scenarios, ambisonic coefficients, object-based audio data, or a combination thereof.

The content creator may, during the editing process, render ambisonic coefficients 11B from audio objects <NUM>, listening to the rendered speaker feeds in an attempt to identify various aspects of the soundfield that require further editing. The content creator device <NUM> may then edit the ambisonic coefficients 11B (potentially indirectly through manipulation of different ones of the audio objects <NUM> from which the source ambisonic coefficients may be derived in the manner described above). The content creator device <NUM> may employ the audio editing system <NUM> to generate the ambisonic coefficients 11B. The audio editing system <NUM> represents any system capable of editing audio data and outputting the audio data as one or more source spherical harmonic coefficients.

When the editing process is complete, the content creator device <NUM> may generate a bitstream <NUM> based on the ambisonic coefficients 11B. That is, the content creator device <NUM> includes an audio encoding device <NUM> that represents a device configured to encode or otherwise compress the ambisonic coefficients 11B in accordance with various aspects of the techniques described in this disclosure to generate the bitstream <NUM>. The audio encoding device <NUM> may generate the bitstream <NUM> for transmission, as one example, across a transmission channel, which may be a wired or wireless channel, a data storage device, or the like. In instances in which the live recordings <NUM> are used to produce the ambisonic coefficients 11B, a portion of the bitstream <NUM> may represent an encoded version of the ambisonic coefficients 11B. In instances where the live recordings <NUM> include an object-based audio signal, the bitstream <NUM> may include an encoded version of the object-based audio data 11A. In any event, the audio encoding device <NUM> may generate the bitstream <NUM> to include a primary bitstream and other side information, such as metadata, which may also be referred to herein as side channel information.

In accordance with aspects of this disclosure, the audio encoding device <NUM> may generate the side channel information of the bitstream <NUM> to include renderer-selection information pertaining to the audio renderers <NUM> illustrated in <FIG>. In some examples, the audio encoding device <NUM> may generate the side channel information of the bitstream <NUM> to indicate whether an object-based renderer of the audio renderers <NUM> was used for content creator-side rendering of the audio data of the bitstream <NUM>, or an ambisonic renderer of the audio renderers <NUM> was used for the content creator-side rendering of the audio data of the bitstream <NUM>. In some examples, if the audio renderers <NUM> include more than one ambisonic renderer and/or more than one object-based renderer, the audio encoding device <NUM> may include additional renderer-selection information in the side channel of the bitstream <NUM>. For instance, if the audio renderers <NUM> include multiple renderers that are applicable to the same type (object or ambisonic) of audio data, the audio encoding device <NUM> may include a renderer identifier (or "renderer ID") in the side channel information, in addition to the renderer type.

According to some example implementations of the techniques of this disclosure, the audio encoding device <NUM> may signal information signifying one or more of the audio renderers <NUM> in the bitstream <NUM>. For instance, if the audio encoding device <NUM> determines that a particular one or more of the audio renderers <NUM> were used for content creator-side rendering of the audio data of the bitstream <NUM>, then the audio encoding device <NUM> may signal one or more matrices signifying the identified audio renderer(s) <NUM> in the bitstream <NUM>. In this way, according to these example implementations of this disclosure, the audio encoding device <NUM> may provide the data necessary to apply one or more of the audio renderers <NUM> directly, via the side channel information of the bitstream <NUM>, for a decoding device to render the audio data signaled via the bitstream <NUM>. Throughout this disclosure, implementations in which the audio encoding device <NUM> transmits matrix information representing any of the audio renderers <NUM> are referred to as "renderer transmission" implementations.

While shown in <FIG> as being directly transmitted to the content consumer device <NUM>, the content creator device <NUM> may output the bitstream <NUM> to an intermediate device positioned between the content creator device <NUM> and the content consumer device <NUM>. The intermediate device may store the bitstream <NUM> for later delivery to the content consumer device <NUM>, which may request the bitstream. The intermediate device may comprise a file server, a web server, a desktop computer, a laptop computer, a tablet computer, a mobile phone, a smart phone, or any other device capable of storing the bitstream <NUM> for later retrieval by an audio decoder. The intermediate device may reside in a content delivery network capable of streaming the bitstream <NUM> (and possibly in conjunction with transmitting a corresponding video data bitstream) to subscribers, such as the content consumer device <NUM>, requesting the bitstream <NUM>.

Alternatively, the content creator device <NUM> may store the bitstream <NUM> to a storage medium, such as a compact disc, a digital video disc, a high definition video disc or other storage media, most of which are capable of being read by a computer and therefore may be referred to as computer-readable storage media or non-transitory computer-readable storage media. In this context, the transmission channel may refer to the channels by which content stored to the mediums are transmitted (and may include retail stores and other store-based delivery mechanism). In any event, the techniques of this disclosure should not therefore be limited in this respect to the example of <FIG>.

As further shown in the example of <FIG>, the content consumer device <NUM> includes the audio playback system <NUM>. The audio playback system <NUM> may represent any audio playback system capable of playing back multi-channel audio data. The audio playback system <NUM> may include a number of different renderers <NUM>. The renderers <NUM> may each provide for a different form of rendering, where the different forms of rendering may include one or more of the various ways of performing vector-base amplitude panning (VBAP), and/or one or more of the various ways of performing soundfield synthesis. As used herein, "A and/or B" means "A or B", or both "A and B".

The audio playback system <NUM> may further include an audio decoding device <NUM>. The audio decoding device <NUM> may represent a device configured to decode ambisonic coefficients 11B' from the bitstream <NUM>, where the ambisonic coefficients 11B' may be similar to the ambisonic coefficients 11B but differ due to lossy operations (e.g., quantization) and/or transmission via the transmission channel. The audio playback system <NUM> may, after decoding the bitstream <NUM> to obtain the ambisonic coefficients 11B' and render the ambisonic coefficients 11B' to output loudspeaker feeds <NUM>. The loudspeaker feeds <NUM> may drive one or more speakers <NUM>.

To select the appropriate renderer or, in some instances, generate an appropriate renderer, the audio playback system <NUM> may obtain loudspeaker information <NUM> indicative of a number of loudspeakers and/or a spatial geometry of the loudspeakers. In some instances, the audio playback system <NUM> may obtain the loudspeaker information <NUM> using a reference microphone and driving the loudspeakers in such a manner as to dynamically determine the loudspeaker information <NUM>. In other instances or in conjunction with the dynamic determination of the loudspeaker information <NUM>, the audio playback system <NUM> may prompt a user to interface with the audio playback system <NUM> and input the loudspeaker information <NUM>.

The audio playback system <NUM> may then select one of the audio renderers <NUM> based on the loudspeaker information <NUM>. In some instances, the audio playback system <NUM> may, when none of the audio renderers <NUM> are within some threshold similarity measure (in terms of the loudspeaker geometry) to the loudspeaker geometry specified in the loudspeaker information <NUM>, generate the one of audio renderers <NUM> based on the loudspeaker information <NUM>. The audio playback system <NUM> may, in some instances, generate one of the audio renderers <NUM> based on the loudspeaker information <NUM> without first attempting to select an existing one of the audio renderers <NUM>. One or more speakers <NUM> may then playback the rendered loudspeaker feeds <NUM>.

When the speakers <NUM> represent speakers of headphones, the audio playback system <NUM> may utilize one of the renderers <NUM> that provides for binaural rendering using head-related transfer functions (HRTF) or other functions capable of rendering to left and right speaker feeds <NUM> for headphone speaker playback. The terms "speakers" or "transducer" may generally refer to any speaker, including loudspeakers, headphone speakers, etc. One or more speakers <NUM> may then playback the rendered speaker feeds <NUM>.

In some instances, the audio playback system <NUM> may select any one the of audio renderers <NUM> and may be configured to select the one or more of audio renderers <NUM> depending on the source from which the bitstream <NUM> is received (such as a DVD player, a Blu-ray player, a smartphone, a tablet computer, a gaming system, and a television to provide a few examples). While any one of the audio renderers <NUM> may be selected, often the audio renderer used when creating the content provides for a better (and possibly the best) form of rendering due to the fact that the content was created by the content creator <NUM> using this one of audio renderers, i.e., the audio renderer <NUM> in the example of <FIG>. Selecting the one of the audio renderers <NUM> that is the same or at least close (in terms of rendering form) may provide for a better representation of the sound field and may result in a better surround sound experience for the content consumer <NUM>.

In accordance with the techniques described in this disclosure, the audio encoding device <NUM> may generate the bitstream <NUM> (e.g., the side channel information thereof) to include the audio rendering information <NUM> ("render info <NUM>"). The audio rendering information <NUM> may include a signal value identifying an audio renderer used when generating the multi-channel audio content, i.e., one or more of the audio renderers <NUM> in the example of <FIG>. In some instances, the signal value includes a matrix used to render spherical harmonic coefficients to a plurality of speaker feeds.

As described above, in accordance with aspects of this disclosure, the audio encoding device <NUM> may include the audio rendering information <NUM> in the side channel information of the bitstream <NUM>. In these examples, the audio decoding device <NUM> may parse the side channel information of the bitstream <NUM> to obtain, as part of the audio rendering information <NUM>, an indication of whether an object-based renderer of the audio renderers <NUM> is to be used to render the audio data of the bitstream <NUM>, or an ambisonic renderer of the audio renderers <NUM> is to be used to render the audio data of the bitstream <NUM>. In some examples, if the audio renderers <NUM> include more than one ambisonic renderer and/or more than one object-based renderer, the audio decoding device <NUM> may obtain additional renderer-selection information as part of the audio rendering information <NUM> from the side channel information of the bitstream <NUM>. For instance, if the audio renderers <NUM> include multiple renderers that are applicable to the same type (object or ambisonic) of audio data, the audio decoding device <NUM> may obtain a renderer ID as part of the audio rendering information <NUM> from the side channel information of the bitstream <NUM>, in addition to obtaining the renderer type.

According to renderer transmission implementations of the techniques of this disclosure, the audio decoding device <NUM> may signal information signifying one or more of the audio renderers <NUM> in the bitstream <NUM>. In these examples, the audio decoding device <NUM> may obtain one or more matrices signifying the identified audio renderer(s) <NUM> from the audio rendering information <NUM>, and apply matrix multiplication using the matrix/matrices to render the object-based audio data 11A' and/or the ambisonic coefficients 11B'. In this way, according to these example implementations of this disclosure, the audio encoding device <NUM> may directly receive, via the bitstream <NUM>, the data necessary to apply one or more of the audio renderers <NUM>, to render the object-based audio data 11A' and/or the ambisonic coefficients 11B'.

In other words and as noted above, ambisonic coefficients (including so-called Higher Order Ambisonic - HOA - coefficients) may represent a way by which to describe directional information of a sound-field based on a spatial Fourier transform. Generally, the higher the ambisonics order N, the higher the spatial resolution, the larger the number of spherical harmonics (SH) coefficients (N+<NUM>)^<NUM>, and the larger the required bandwidth for transmitting and storing the data. HOA coefficients generally refer to ambisonic representation having ambisonic coefficients associated with spherical basis functions having an order greater than one.

A potential advantage of this description is the possibility to reproduce this soundfield on most any loudspeaker setup (e.g., <NUM>, <NUM><NUM>, etc.). The conversion from the soundfield description into M loudspeaker signals may be done via a static rendering matrix with (N+<NUM>)<NUM> inputs and M outputs. Consequently, every loudspeaker setup may require a dedicated rendering matrix. Several algorithms may exist for computing the rendering matrix for a desired loudspeaker setup, which may be optimized for certain objective or subjective measures, such as the Gerzon criteria. For irregular loudspeaker setups, algorithms may become complex due to iterative numerical optimization procedures, such as convex optimization.

To compute a rendering matrix for irregular loudspeaker layouts without waiting time, it may be beneficial to have sufficient computation resources available. Irregular loudspeaker setups may be common in domestic living room environments due to architectural constrains and aesthetic preferences. Therefore, for the best soundfield reproduction, a rendering matrix optimized for such scenario may be preferred in that it may enable reproduction of the soundfield more accurately.

Because an audio decoder usually does not require much computational resources, the device may not be able to compute an irregular rendering matrix in a consumer-friendly time. Various aspects of the techniques described in this disclosure may provide for the use a cloud-based computing approach as follows:.

This approach may allow the manufacturer to keep manufacturing costs of an audio decoder low (because a powerful processor may not be needed to compute these irregular rendering matrices), while also facilitating a more optimal audio reproduction in comparison to rendering matrices usually designed for regular speaker configurations or geometries. The algorithm for computing the rendering matrix may also be optimized after an audio decoder has shipped, potentially reducing the costs for hardware revisions or even recalls. The techniques may also, in some instances, gather a lot of information about different loudspeaker setups of consumer products which may be beneficial for future product developments.

Again, in some instances, the system shown in <FIG> may not incorporate signaling of the audio rendering information <NUM> in the bitstream <NUM> as described above, but instead, may use signaling of this audio rendering information <NUM> as metadata separate from the bitstream <NUM>. Alternatively or in conjunction with that described above, the system shown in <FIG> may signal a portion of the audio rendering information <NUM> in the bitstream <NUM> as described above and signal a portion of this audio rendering information <NUM> as metadata separate from the bitstream <NUM>. In some examples, the audio encoding device <NUM> may output this metadata, which may then be uploaded to a server or other device. The audio decoding device <NUM> may then download or otherwise retrieve this metadata, which is then used to augment the audio rendering information extracted from the bitstream <NUM> by the audio decoding device <NUM>. The bitstream <NUM> formed in accordance with the rendering information aspects of the techniques are described below.

<FIG> is a block diagram illustrating, in more detail, one example of the audio encoding device <NUM> shown in the example of <FIG> that may perform various aspects of the techniques described in this disclosure. The audio encoding device <NUM> includes a content analysis unit <NUM>, a vector-based decomposition unit <NUM> and a directional-based decomposition unit <NUM>. Although described briefly below, more information regarding the audio encoding device <NUM> and the various aspects of compressing or otherwise encoding ambisonic coefficients is available in International Patent Application Publication No. <CIT>.

The audio encoding device <NUM> is illustrated in <FIG> as including various units, each of which is further described below with respect to particular functionalities of the audio encoding device <NUM> as a whole. The various units of the audio encoding device <NUM> may be implemented using processor hardware, such as one or more processors. That is, a given processor of the audio encoding device <NUM> may implement the functionalities described below with respect one of the illustrated units, or of multiple units of the illustrated units. The processor(s) of the audio encoding device <NUM> may include processing circuitry (e.g. fixed function circuitry, programmable processing circuitry, or any combination thereof), application specific integrated circuits (ASICs), such as one or more hardware ASICs, digital signal processors (DSPs), general purpose microprocessors, field programmable logic arrays (FPGAs), or other equivalent integrated circuitry or discrete logic circuitry. The processor(s) of the audio encoding device <NUM> may be configured to execute, using the processing hardware thereof, software to perform the functionalities described below with respect to the illustrated units.

The content analysis unit <NUM> represents a unit configured to analyze the content of the object-based audio data 11A and/or ambisonic coefficients 11B (collectively, the "audio data <NUM>") to identify whether the audio data <NUM> represents content generated from a live recording or an audio object or both. The content analysis unit <NUM> may determine whether the audio data <NUM> were generated from a recording of an actual soundfield or from an artificial audio object. In some instances, when the audio data <NUM> (e.g., the framed ambisonic coefficients 11B) were generated from a recording, the content analysis unit <NUM> passes the framed ambisonic coefficients 11B to the vector-based decomposition unit <NUM>.

In some instances, when the audio data <NUM> (e.g., the framed ambisonic coefficients 11B) were generated from a synthetic audio object, the content analysis unit <NUM> passes the ambisonic coefficients 11B to the directional-based synthesis unit <NUM>. The directional-based synthesis unit <NUM> may represent a unit configured to perform a directional-based synthesis of the ambisonic coefficients 11B to generate a directional-based bitstream <NUM>. In examples where the audio data <NUM> includes the object-based audio data 11A, the content analysis unit <NUM> passes the object-based audio data 11A to the bitstream generation unit <NUM>.

As shown in the example of <FIG>, the vector-based decomposition unit <NUM> may include a linear invertible transform (LIT) unit <NUM>, a parameter calculation unit <NUM>, a reorder unit <NUM>, a foreground selection unit <NUM>, an energy compensation unit <NUM>, a psychoacoustic audio coder unit <NUM>, a bitstream generation unit <NUM>, a soundfield analysis unit <NUM>, a coefficient reduction unit <NUM>, a background (BG) selection unit <NUM>, a spatio-temporal interpolation unit <NUM>, and a quantization unit <NUM>.

The linear invertible transform (LIT) unit <NUM> receives the ambisonic coefficients 11B in the form of ambisonic channels, each channel representative of a block or frame of a coefficient associated with a given order, sub-order of the spherical basis functions (which may be denoted as HOA[k], where k may denote the current frame or block of samples). The matrix of ambisonic coefficients 11B may have dimensions D: M x (N+<NUM>)<NUM>.

The LIT unit <NUM> may represent a unit configured to perform a form of analysis referred to as singular value decomposition. While described with respect to SVD, the techniques described in this disclosure may be performed with respect to any similar transformation or decomposition that provides for sets of linearly uncorrelated, energy compacted output. Also, reference to "sets" in this disclosure is generally intended to refer to non-zero sets unless specifically stated to the contrary and is not intended to refer to the classical mathematical definition of sets that includes the so-called "empty set. " An alternative transformation may comprise a principal component analysis, which is often referred to as "PCA. " Depending on the context, PCA may be referred to by a number of different names, such as discrete Karhunen-Loeve transform, the Hotelling transform, proper orthogonal decomposition (POD), and eigenvalue decomposition (EVD) to name a few examples. Properties of such operations that are conducive to the underlying goal of compressing audio data are 'energy compaction' and 'decorrelation' of the multichannel audio data.

In any event, assuming the LIT unit <NUM> performs a singular value decomposition (which, again, may be referred to as "SVD") for purposes of example, the LIT unit <NUM> may transform the ambisonic coefficients 11B into two or more sets of transformed ambisonic coefficients. The "sets" of transformed ambisonic coefficients may include vectors of transformed ambisonic coefficients. In the example of <FIG>, the LIT unit <NUM> may perform the SVD with respect to the ambisonic coefficients 11B to generate a so-called V matrix, an S matrix, and a U matrix. SVD, in linear algebra, may represent a factorization of a y-by-z real or complex matrix X (where X may represent multi-channel audio data, such as the ambisonic coefficients 11B) in the following form: <MAT> U may represent a y-by-y real or complex unitary matrix, where the y columns of U are known as the left-singular vectors of the multi-channel audio data. S may represent a y-by-z rectangular diagonal matrix with non-negative real numbers on the diagonal, where the diagonal values of S are known as the singular values of the multi-channel audio data. V* (which may denote a conjugate transpose of V) may represent a z-by-z real or complex unitary matrix, where the z columns of V* are known as the right-singular vectors of the multi-channel audio data.

In some examples, the V* matrix in the SVD mathematical expression referenced above is denoted as the conjugate transpose of the V matrix to reflect that SVD may be applied to matrices comprising complex numbers. When applied to matrices comprising only real-numbers, the complex conjugate of the V matrix (or, in other words, the V* matrix) may be considered to be the transpose of the V matrix. Below it is assumed, for ease of illustration purposes, that the ambisonic coefficients 11B comprise real-numbers with the result that the V matrix is output through SVD rather than the V* matrix. Moreover, while denoted as the V matrix in this disclosure, reference to the V matrix should be understood to refer to the transpose of the V matrix where appropriate. While assumed to be the V matrix, the techniques may be applied in a similar fashion to ambisonic coefficients 11B having complex coefficients, where the output of the SVD is the V* matrix. Accordingly, the techniques should not be limited in this respect to only provide for application of SVD to generate a V matrix, but may include application of SVD to ambisonic coefficients 11B having complex components to generate a V* matrix.

In this way, the LIT unit <NUM> may perform SVD with respect to the ambisonic coefficients 11B to output US[k] vectors <NUM> (which may represent a combined version of the S vectors and the U vectors) having dimensions D: M x (N+<NUM>)<NUM>, and V[k] vectors <NUM> having dimensions D: (N+<NUM>)<NUM> x (N+<NUM>)<NUM>. Individual vector elements in the US[k] matrix may also be termed XPS(k) while individual vectors of the V[k] matrix may also be termed v(k).

An analysis of the U, S and V matrices may reveal that the matrices carry or represent spatial and temporal characteristics of the underlying soundfield represented above by X. Each of the N vectors in U (of length M samples) may represent normalized separated audio signals as a function of time (for the time period represented by M samples), that are orthogonal to each other and that have been decoupled from any spatial characteristics (which may also be referred to as directional information). The spatial characteristics, representing spatial shape and position (r, theta, phi) may instead be represented by individual ith vectors, v(i)(k), in the V matrix (each of length (N+<NUM>)<NUM>). The individual elements of each of v(i)(k) vectors may represent an ambisonic coefficient describing the shape (including width) and position of the soundfield for an associated audio object.

Both the vectors in the U matrix and the V matrix are normalized such that their root-mean-square energies are equal to unity. The energy of the audio signals in U is thus represented by the diagonal elements in S. Multiplying U and S to form US[k] (with individual vector elements XPS(k)), thus represent the audio signal with energies. The ability of the SVD decomposition to decouple the audio time-signals (in U), their energies (in S) and their spatial characteristics (in V) may support various aspects of the techniques described in this disclosure. Further, the model of synthesizing the underlying HOA[k] coefficients, X, by a vector multiplication of US[k] and V[k] gives rise the term "vector-based decomposition," which is used throughout this document.

Although described as being performed directly with respect to the ambisonic coefficients 11B, the LIT unit <NUM> may apply the linear invertible transform to derivatives of the ambisonic coefficients 11B. For example, the LIT unit <NUM> may apply SVD with respect to a power spectral density matrix derived from the ambisonic coefficients 11B. By performing SVD with respect to the power spectral density (PSD) of the ambisonic coefficients rather than the coefficients themselves, the LIT unit <NUM> may potentially reduce the computational complexity of performing the SVD in terms of one or more of processor cycles and storage space, while achieving the same source audio encoding efficiency as if the SVD were applied directly to the ambisonic coefficients.

The parameter calculation unit <NUM> represents a unit configured to calculate various parameters, such as a correlation parameter (R), directional properties parameters (θ, φ, r), and an energy property (e). Each of the parameters for the current frame may be denoted as R[k], θ[k], φ[k], r[k] and e[k]. The parameter calculation unit <NUM> may perform an energy analysis and/or correlation (or so-called cross-correlation) with respect to the US[k] vectors <NUM> to identify the parameters. The parameter calculation unit <NUM> may also determine the parameters for the previous frame, where the previous frame parameters may be denoted R[k-<NUM>], θ[k-<NUM>], φ[k-<NUM>], r[k-<NUM>] and e[k-<NUM>], based on the previous frame of US[k-<NUM>] vector and V[k-<NUM>] vectors. The parameter calculation unit <NUM> may output the current parameters <NUM> and the previous parameters <NUM> to reorder unit <NUM>.

The parameters calculated by the parameter calculation unit <NUM> may be used by the reorder unit <NUM> to re-order the audio objects to represent their natural evaluation or continuity over time. The reorder unit <NUM> may compare each of the parameters <NUM> from the first US[k] vectors <NUM> turn-wise against each of the parameters <NUM> for the second US[k-<NUM>] vectors <NUM>. The reorder unit <NUM> may reorder (using, as one example, a Hungarian algorithm) the various vectors within the US[k] matrix <NUM> and the V[k] matrix <NUM> based on the current parameters <NUM> and the previous parameters <NUM> to output a reordered US[k] matrix <NUM>' (which may be denoted mathematically as US[k]) and a reordered V[k] matrix <NUM>' (which may be denoted mathematically as V[k]) to a foreground sound (or predominant sound - PS) selection unit <NUM> ("foreground selection unit <NUM>") and an energy compensation unit <NUM>.

The soundfield analysis unit <NUM> may represent a unit configured to perform a soundfield analysis with respect to the ambisonic coefficients 11B so as to potentially achieve a target bitrate <NUM>. The soundfield analysis unit <NUM> may, based on the analysis and/or on a received target bitrate <NUM>, determine the total number of psychoacoustic coder instantiations (which may be a function of the total number of ambient or background channels (BGTOT) and the number of foreground channels or, in other words, predominant channels. The total number of psychoacoustic coder instantiations can be denoted as numHOATransportChannels.

The soundfield analysis unit <NUM> may also determine, again to potentially achieve the target bitrate <NUM>, the total number of foreground channels (nFG) <NUM>, the minimum order of the background (or, in other words, ambient) soundfield (NBG or, alternatively, MinAmbHOAorder), the corresponding number of actual channels representative of the minimum order of background soundfield (nBGa = (MinAmbHOAorder + <NUM>)<NUM>), and indices (i) of additional BG ambisonic channels to send (which may collectively be denoted as background channel information <NUM> in the example of <FIG>). The background channel information <NUM> may also be referred to as ambient channel information <NUM>. Each of the channels that remains from numHOATransportChannels - nBGa, may either be an "additional background/ambient channel", an "active vector-based predominant channel", an "active directional based predominant signal" or "completely inactive". In one aspect, the channel types may be indicated (as a "ChannelType") syntax element by two bits (e.g. <NUM>: directional based signal; <NUM>: vector-based predominant signal; <NUM>: additional ambient signal; <NUM>: inactive signal). The total number of background or ambient signals, nBGa, may be given by (MinAmbHOAorder +<NUM>)<NUM> + the number of times the index <NUM> (in the above example) appears as a channel type in the bitstream for that frame.

The soundfield analysis unit <NUM> may select the number of background (or, in other words, ambient) channels and the number of foreground (or, in other words, predominant) channels based on the target bitrate <NUM>, selecting more background and/or foreground channels when the target bitrate <NUM> is relatively higher (e.g., when the target bitrate <NUM> equals or is greater than <NUM> Kbps). In one aspect, the numHOATransportChannels may be set to <NUM> while the MinAmbHOAorder may be set to <NUM> in the header section of the bitstream. In this scenario, at every frame, four channels may be dedicated to represent the background or ambient portion of the soundfield while the other <NUM> channels can, on a frame-by-frame basis vary on the type of channel - e.g., either used as an additional background/ambient channel or a foreground/predominant channel. The foreground/predominant signals can be one of either vector-based or directional based signals, as described above.

In some instances, the total number of vector-based predominant signals for a frame, may be given by the number of times the ChannelType index is <NUM> in the bitstream of that frame. In the above aspect, for every additional background/ambient channel (e.g., corresponding to a ChannelType of <NUM>), corresponding information of which of the possible ambisonic coefficients (beyond the first four) may be represented in that channel. The information, for fourth order HOA content, may be an index to indicate the HOA coefficients <NUM>-<NUM>. The first four ambient HOA coefficients <NUM>-<NUM> may be sent all the time when minAmbHOAorder is set to <NUM>, hence the audio encoding device may only need to indicate one of the additional ambient HOA coefficients having an index of <NUM>-<NUM>. The information could thus be sent using a <NUM> bits syntax element (for <NUM>th order content), which may be denoted as "CodedAmbCoeffIdx. " In any event, the soundfield analysis unit <NUM> outputs the background channel information <NUM> and the ambisonic coefficients 11B to the background (BG) selection unit <NUM>, the background channel information <NUM> to coefficient reduction unit <NUM> and the bitstream generation unit <NUM>, and the nFG <NUM> to a foreground selection unit <NUM>.

The background selection unit <NUM> may represent a unit configured to determine background or ambient ambisonic coefficients <NUM> based on the background channel information (e.g., the background soundfield (NBG) and the number (nBGa) and the indices (i) of additional BG ambisonic channels to send). For example, when NBG equals one, the background selection unit <NUM> may select the ambisonic coefficients 11B for each sample of the audio frame having an order equal to or less than one. The background selection unit <NUM> may, in this example, then select the ambisonic coefficients 11B having an index identified by one of the indices (i) as additional BG ambisonic coefficients, where the nBGa is provided to the bitstream generation unit <NUM> to be specified in the bitstream <NUM> so as to enable the audio decoding device, such as the audio decoding device <NUM> shown in the example of <FIG> and <FIG>, to parse the background ambisonic coefficients <NUM> from the bitstream <NUM>. The background selection unit <NUM> may then output the ambient ambisonic coefficients <NUM> to the energy compensation unit <NUM>. The ambient ambisonic coefficients <NUM> may have dimensions D: M x [(NBG+<NUM>)<NUM> + nBGa]. The ambient ambisonic coefficients <NUM> may also be referred to as "ambient ambisonic coefficients <NUM>," where each of the ambient ambisonic coefficients <NUM> corresponds to a separate ambient ambisonic channel <NUM> to be encoded by the psychoacoustic audio coder unit <NUM>.

The foreground selection unit <NUM> may represent a unit configured to select the reordered US[k] matrix <NUM>' and the reordered V[k] matrix <NUM>' that represent foreground or distinct components of the soundfield based on nFG <NUM> (which may represent a one or more indices identifying the foreground vectors). The foreground selection unit <NUM> may output nFG signals <NUM> (which may be denoted as a reordered US[k]<NUM>,. , nFG <NUM>, FG<NUM>,. , nfG[k] <NUM>, or <MAT> <NUM>) to the psychoacoustic audio coder unit <NUM>, where the nFG signals <NUM> may have dimensions D: M x nFG and each represent mono-audio objects. The foreground selection unit <NUM> may also output the reordered V[k] matrix <NUM>' (or v(<NUM>. nFG)(k) <NUM>') corresponding to foreground components of the soundfield to the spatio-temporal interpolation unit <NUM>, where a subset of the reordered V[k] matrix <NUM>' corresponding to the foreground components may be denoted as foreground V[k] matrix <NUM>k (which may be mathematically denoted as V<NUM>,. ,nFG[k]) having dimensions D: (N+<NUM>)<NUM> x nFG.

The energy compensation unit <NUM> may represent a unit configured to perform energy compensation with respect to the ambient ambisonic coefficients <NUM> to compensate for energy loss due to removal of various ones of the ambisonic channels by the background selection unit <NUM>. The energy compensation unit <NUM> may perform an energy analysis with respect to one or more of the reordered US[k] matrix <NUM>', the reordered V[k] matrix <NUM>', the nFG signals <NUM>, the foreground V[k] vectors <NUM>k and the ambient ambisonic coefficients <NUM> and then perform energy compensation based on the energy analysis to generate energy compensated ambient ambisonic coefficients <NUM>'. The energy compensation unit <NUM> may output the energy compensated ambient coefficients <NUM>' to the psychoacoustic audio coder unit <NUM>.

The spatio-temporal interpolation unit <NUM> may represent a unit configured to receive the foreground V[k] vectors <NUM>k for the kth frame and the foreground V[k-<NUM>] vectors <NUM>k-<NUM> for the previous frame (hence the k-<NUM> notation) and perform spatio-temporal interpolation to generate interpolated foreground V[k] vectors. The spatio-temporal interpolation unit <NUM> may recombine the nFG signals <NUM> with the foreground V[k] vectors <NUM>k to recover reordered foreground ambisonic coefficients. The spatio-temporal interpolation unit <NUM> may then divide the reordered foreground ambisonic coefficients by the interpolated V[k] vectors to generate interpolated nFG signals <NUM>'.

The spatio-temporal interpolation unit <NUM> may also output the foreground V[k] vectors <NUM>k that were used to generate the interpolated foreground V[k] vectors so that an audio decoding device, such as the audio decoding device <NUM>, may generate the interpolated foreground V[k] vectors and thereby recover the foreground V[k] vectors <NUM>k. The foreground V[k] vectors <NUM>k used to generate the interpolated foreground V[k] vectors are denoted as the remaining foreground V[k] vectors <NUM>. In order to ensure that the same V[k] and V[k-<NUM>] are used at the encoder and decoder (to create the interpolated vectors V[k]) quantized/dequantized versions of the vectors may be used at the encoder and decoder. The spatio-temporal interpolation unit <NUM> may output the interpolated nFG signals <NUM>' to the psychoacoustic audio coder unit <NUM> and the interpolated foreground V[k] vectors <NUM>k to the coefficient reduction unit <NUM>.

The coefficient reduction unit <NUM> may represent a unit configured to perform coefficient reduction with respect to the remaining foreground V[k] vectors <NUM> based on the background channel information <NUM> to output reduced foreground V[k] vectors <NUM> to the quantization unit <NUM>. The reduced foreground V[k] vectors <NUM> may have dimensions D: [(N+<NUM>)<NUM> - (NBG+<NUM>)<NUM>-BGTOT] x nFG. The coefficient reduction unit <NUM> may, in this respect, represent a unit configured to reduce the number of coefficients in the remaining foreground V[k] vectors <NUM>. In other words, coefficient reduction unit <NUM> may represent a unit configured to eliminate the coefficients in the foreground V[k] vectors (that form the remaining foreground V[k] vectors <NUM>) having little to no directional information.

In some examples, the coefficients of the distinct or, in other words, foreground V[k] vectors corresponding to a first and zero order basis functions (which may be denoted as NBG) provide little directional information and therefore can be removed from the foreground V-vectors (through a process that may be referred to as "coefficient reduction"). In this example, greater flexibility may be provided to not only identify the coefficients that correspond NBG but to identify additional ambisonic channels (which may be denoted by the variable TotalOfAddAmbHOAChan) from the set of [(NBG +<NUM>)<NUM>+<NUM>, (N+<NUM>)<NUM>].

The quantization unit <NUM> may represent a unit configured to perform any form of quantization to compress the reduced foreground V[k] vectors <NUM> to generate coded foreground V[k] vectors <NUM>, outputting the coded foreground V[k] vectors <NUM> to the bitstream generation unit <NUM>. In operation, the quantization unit <NUM> may represent a unit configured to compress a spatial component of the soundfield, i.e., one or more of the reduced foreground V[k] vectors <NUM> in this example. The quantization unit <NUM> may perform any one of the following <NUM> quantization modes, as indicated by a quantization mode syntax element denoted "NbitsQ":
<IMG>
<IMG>
The quantization unit <NUM> may also perform predicted versions of any of the foregoing types of quantization modes, where a difference is determined between an element of (or a weight when vector quantization is performed) of the V-vector of a previous frame and the element (or weight when vector quantization is performed) of the V-vector of a current frame is determined. The quantization unit <NUM> may then quantize the difference between the elements or weights of the current frame and previous frame rather than the value of the element of the V-vector of the current frame itself.

The quantization unit <NUM> may perform multiple forms of quantization with respect to each of the reduced foreground V[k] vectors <NUM> to obtain multiple coded versions of the reduced foreground V[k] vectors <NUM>. The quantization unit <NUM> may select the one of the coded versions of the reduced foreground V[k] vectors <NUM> as the coded foreground V[k] vector <NUM>. The quantization unit <NUM> may, in other words, select one of the non-predicted vector-quantized V-vector, predicted vector-quantized V-vector, the non-Huffman-coded scalar-quantized V-vector, and the Huffman-coded scalar-quantized V-vector to use as the output switched-quantized V-vector based on any combination of the criteria discussed in this disclosure.

In some examples, the quantization unit <NUM> may select a quantization mode from a set of quantization modes that includes a vector quantization mode and one or more scalar quantization modes, and quantize an input V-vector based on (or according to) the selected mode. The quantization unit <NUM> may then provide the selected one of the non-predicted vector-quantized V-vector (e.g., in terms of weight values or bits indicative thereof), predicted vector-quantized V-vector (e.g., in terms of error values or bits indicative thereof), the non-Huffman-coded scalar-quantized V-vector and the Huffman-coded scalar-quantized V-vector to the bitstream generation unit <NUM> as the coded foreground V[k] vectors <NUM>. The quantization unit <NUM> may also provide the syntax elements indicative of the quantization mode (e.g., the NbitsQ syntax element) and any other syntax elements used to dequantize or otherwise reconstruct the V-vector.

The psychoacoustic audio coder unit <NUM> included within the audio encoding device <NUM> may represent multiple instances of a psychoacoustic audio coder, each of which is used to encode a different audio object or ambisonic channel of each of the energy compensated ambient ambisonic coefficients <NUM>' and the interpolated nFG signals <NUM>' to generate encoded ambient ambisonic coefficients <NUM> and encoded nFG signals <NUM>. The psychoacoustic audio coder unit <NUM> may output the encoded ambient ambisonic coefficients <NUM> and the encoded nFG signals <NUM> to the bitstream generation unit <NUM>.

The bitstream generation unit <NUM> included within the audio encoding device <NUM> represents a unit that formats data to conform to a known format (which may refer to a format known by a decoding device), thereby generating the vector-based bitstream <NUM>. The bitstream <NUM> may, in other words, represent encoded audio data, having been encoded in the manner described above.

The bitstream generation unit <NUM> may represent a multiplexer in some examples, which may receive the coded foreground V[k] vectors <NUM>, the encoded ambient ambisonic coefficients <NUM>, the encoded nFG signals <NUM> and the background channel information <NUM>. The bitstream generation unit <NUM> may then generate a bitstream <NUM> based on the coded foreground V[k] vectors <NUM>, the encoded ambient ambisonic coefficients <NUM>, the encoded nFG signals <NUM> and the background channel information <NUM>. In this way, the bitstream generation unit <NUM> may thereby specify the vectors <NUM> in the bitstream <NUM> to obtain the bitstream <NUM>. The bitstream <NUM> may include a primary or main bitstream and one or more side channel bitstreams.

Various aspects of the techniques may also enable the bitstream generation unit <NUM> to, as described above, specify the audio rendering information <NUM> in or in parallel with the bitstream <NUM>. While the current version of the upcoming 3D audio compression working draft, provides for signaling specific downmix matrices within the bitstream <NUM>, the working draft does not provide for specifying of renderers used in rendering the object-based audio data 11A or the ambisonic coefficients 11B in the bitstream <NUM>. For AMBISONIC content, the equivalent of such downmix matrix is the rendering matrix which converts the ambisonic representation into the desired loudspeaker feeds. For audio data in the object domain, the equivalent is a rendering matrix that is applied using matrix multiplication to render the object-based audio data into loudspeaker feeds.

Various aspects of the techniques described in this disclosure propose to further harmonize the feature sets of channel content and ambisonic coefficients by allowing the bitstream generation unit <NUM> to signal renderer selection information (e.g., ambisonic versus object-based renderer selection), renderer identification information (e.g., an entry in a codebook accessible to both the audio encoding device <NUM> and the audio decoding device <NUM>), and/or the rendering matrices themselves within the bitstream <NUM> or side channel / metadata thereof (as, for example, the audio rendering information <NUM>).

The audio encoding device <NUM> may include combined or discrete processing hardware configured to perform one or both (as the case may be) of the ambisonic or object-based encoding functionalities described above, as well as the renderer selection and signaling-based techniques of this disclosure. The processing hardware that the audio encoding device <NUM> includes for performing one or more of the ambisonic encoding, object-based encoding, and renderer-based techniques may include as one or more processors. These processor(s) of the audio encoding device <NUM> may include processing circuitry (e.g. fixed function circuitry, programmable processing circuitry, or any combination thereof), application specific integrated circuits (ASICs), such as one or more hardware ASICs, digital signal processors (DSPs), general purpose microprocessors, field programmable logic arrays (FPGAs), or other equivalent integrated circuitry or discrete logic circuitry for one or more ambisonic encoding, object-based audio encoding, and/or renderer selection and/or signaling based techniques. These processor(s) of the audio encoding device <NUM> may be configured to execute, using the processing hardware thereof, software to perform the functionalities described above.

Table <NUM> below is a syntax table providing details of example data that the audio encoding device <NUM> may signal to the audio decoding device <NUM> to provide the renderer information <NUM>. Comment statements, which are bookended by "/*" and "*/" tags in Table <NUM>, provide descriptive information of the corresponding syntax positioned adjacently thereto.

Table <NUM> below is a syntax table providing details of another example of data that the audio encoding device <NUM> may signal to the audio decoding device <NUM> to provide the renderer information <NUM>, in accordance with "soft" rendering aspects of this disclosure. As in the case of Table <NUM> above, comment statements, which are bookended by "/*" and "*/" tags in Table <NUM> provide descriptive information of the corresponding syntax positioned adjacently thereto.

The bitstream generation unit <NUM> of the audio encoding device <NUM> may provide the data represented in the bitstream <NUM> to an interface <NUM>, which in turn may signal the data in the form of the bitstream <NUM> to an external device. The interface <NUM> may include, be, or be part of various types of communication hardware, such as a network interface card (e.g., an Ethernet card), an optical transceiver, a radio frequency transceiver, or any other type of device that can receive (and potentially send) information. Other examples of such network interfaces that may be represented by the interface <NUM> include Bluetooth®, <NUM>, <NUM>, <NUM>, and WiFi® radios. The interface <NUM> may also be implemented according to any version of the Universal Serial Bus (USB) standards. As such, the interface <NUM> enables the audio encoding device <NUM> to communicate wirelessly, or using wired connection, or a combination thereof, with external devices, such as network devices. As such, the audio encoding device <NUM> may implement various techniques of this disclosure to provide renderer-related information to the audio decoding device <NUM> in or along with the bitstream <NUM>. Further details on how the audio decoding device <NUM> may use the render-related information received in or along with the bitstream <NUM> are described below with respect to <FIG>.

<FIG> is a block diagram illustrating the audio decoding device <NUM> of <FIG> in more detail. As shown in the example of <FIG> the audio decoding device <NUM> may include an extraction unit <NUM>, a renderer reconstruction unit <NUM>, a directionality-based reconstruction unit <NUM> and a vector-based reconstruction unit <NUM>. Although described below, more information regarding the audio decoding device <NUM> and the various aspects of decompressing or otherwise decoding ambisonic coefficients is available in International Patent Application with Publication No. <CIT>.

The audio decoding device <NUM> is illustrated in <FIG> as including various units, each of which is further described below with respect to particular functionalities of the audio decoding device <NUM> as a whole. The various units of the audio decoding device <NUM> may be implemented using processor hardware, such as one or more processors. That is, a given processor of the audio decoding device <NUM> may implement the functionalities described below with respect to one of the illustrated units, or of multiple units of the illustrated units. The processor(s) of the audio decoding device <NUM> may include processing circuitry (e.g. fixed function circuitry, programmable processing circuitry, or any combination thereof), application specific integrated circuits (ASICs), such as one or more hardware ASICs, digital signal processors (DSPs), general purpose microprocessors, field programmable logic arrays (FPGAs), or other equivalent integrated circuitry or discrete logic circuitry. The processor(s) of the audio decoding device <NUM> may be configured to execute, using the processing hardware thereof, software to perform the functionalities described below with respect to the illustrated units.

The audio decoding device <NUM> includes an interface <NUM>, which is configured to receive the bitstream <NUM> and relay the data thereof to the extraction unit <NUM>. The interface <NUM> may include, be, or be part of various types of communication hardware, such as a network interface card (e.g., an Ethernet card), an optical transceiver, a radio frequency transceiver, or any other type of device that can receive (and potentially send) information. Other examples of such network interfaces that may be represented by the interface <NUM> include Bluetooth®, <NUM>, <NUM>, <NUM>, and WiFi® radios. The interface <NUM> may also be implemented according to any version of the Universal Serial Bus (USB) standards. As such, the interface <NUM> enables the audio decoding device <NUM> to communicate wirelessly, or using wired connection, or a combination thereof, with external devices, such as network devices.

The extraction unit <NUM> may represent a unit configured to receive the bitstream <NUM> and extract the audio rendering information <NUM> and the various encoded versions (e.g., a directional-based encoded version or a vector-based encoded version) of the object-based audio data 11A and/or ambisonic coefficients 11B. According to various examples of the techniques of this disclosure, the extraction unit <NUM> may obtain, from the audio rendering information <NUM>, one or more of an indication of whether to use an ambisonic or an object-domain renderer of the audio renderers <NUM>, a renderer ID of a particular renderer to be used (in the event that the audio renderers <NUM> include multiple ambisonic renderers or multiple object-based renderers), or the rendering matrix/matrices to be added to the audio renderers <NUM> for use in rendering the audio data <NUM> of the bitstream <NUM>. For instance, in the renderer transmission based implementations of this disclosure, ambisonic and/or object-domain rendering matrices may be transmitted by the audio encoding device <NUM> to enable control over the rendering process at the audio playback system <NUM>.

In the case of ambisonic rendering matrices, transmission may be facilitated by means of the mpegh3daConfigExtension of Type ID_CONFIG_EXT_HOA_MATRIX shown above. The mpegh3daConfigExtension may contain several ambisonic rendering matrices for different loudspeaker reproduction configurations. When ambisonic rendering matrices are transmitted, the audio encoding device <NUM> signals, for each ambisonic rendering matrix signal, the associated target loudspeaker layout that determines together with the HoaOrder the dimensions of the rendering matrix. When object-based rendering matrices are transmitted, the audio encoding device <NUM> signals, for each object-based rendering matrix signal, the associated target loudspeaker layout that determines the dimensions of the rendering matrix.

The transmission of a unique HoaRenderingMatrixId allows referencing to a default ambisonic rendering matrix available at the audio playback system <NUM>, or to a transmitted ambisonic rendering matrix from outside of the audio bitstream <NUM>. In some instances, every ambisonic rendering matrix is assumed to be normalized in N3D and follows the ordering of the ambisonic coefficients as defined in the bitstream <NUM>. In instances in which the audio decoding device <NUM> receives a renderer ID in the bitstream <NUM>, the audio decoding device <NUM> may compare the received renderer ID to entries of a codebook. Upon detecting a match in the codebook, the audio decoding device <NUM> may select the matched audio renderer <NUM> for rendering the audio data <NUM> (whether in the object domain or in the ambisonic domain, as the case may be).

Again, as described above, various aspects of the techniques may also enable the extraction unit <NUM> to parse the audio rendering information <NUM> from data the bitstream <NUM> of or side channel information signaled in parallel with the bitstream <NUM>. While the current version of the upcoming 3D audio compression working draft, provides for signaling specific downmix matrices within the bitstream <NUM>, the working draft does not provide for specifying of renderers used in rendering the object-based audio data 11A or the ambisonic coefficients 11B in the bitstream <NUM>. For ambisonic content, the equivalent of such a downmix matrix is the rendering matrix which converts the ambisonic representation into the desired loudspeaker feeds. For audio data in the object domain, the equivalent is a rendering matrix that is applied using matrix multiplication to render the object-based audio data into loudspeaker feeds.

The audio decoding device <NUM> may include combined or discrete processing hardware configured to perform one or both (as the case may be) of the ambisonic or object-based decoding functionalities described above, as well as the renderer selection-based techniques of this disclosure. The processing hardware that the audio decoding device <NUM> includes for performing one or more of the ambisonic decoding, object-based decoding, and renderer-based techniques may include as one or more processors. These processor(s) of the audio decoding device <NUM> may include processing circuitry (e.g. fixed function circuitry, programmable processing circuitry, or any combination thereof), application specific integrated circuits (ASICs), such as one or more hardware ASICs, digital signal processors (DSPs), general purpose microprocessors, field programmable logic arrays (FPGAs), or other equivalent integrated circuitry or discrete logic circuitry for one or more ambisonic decoding, object-based audio decoding, and/or renderer selection techniques. These processor(s) of the audio decoding device <NUM> may be configured to execute, using the processing hardware thereof, software to perform the functionalities described above.

Various aspects of the techniques described in this disclosure propose to further harmonize the feature sets of channel content and ambisonic by allowing the audio decoding device <NUM> to obtain, in the form of the audio rendering information <NUM> renderer selection information (e.g., ambisonic versus object-based renderer selection), renderer identification information (e.g., an entry in a codebook accessible to both the audio encoding device <NUM> and the audio decoding device <NUM>), and/or the rendering matrices themselves from the bitstream <NUM> itself or from side channel / metadata thereof.

As discussed above with respect to the semantics of Table <NUM>, in one example, the audio decoding device <NUM> may receive one or more of the following syntax elements in the bitstream <NUM>: a RendererFlag_OBJ_HOA flag, a RendererFlag_Transmitted_Reference flag, or RendererFlag_ENTIRE_SEPARATE flag, a RendererFlag_External_Internal, or a rendererID syntax element. The audio decoding device <NUM> may leverage the value of the RendererFlag_OBJ_HOA flag to preserve the artistic intent of the content producer. That is, if the value of the RendererFlag_OBJ_HOA flag is <NUM>, then the audio decoding device <NUM> may select an object-based renderer (OBJ renderer) from the audio renderers <NUM> for rendering the corresponding portion of the audio data <NUM>' obtained from the bitstream <NUM>. Conversely, if the audio decoding device <NUM> determines that that the value of the RendererFlag_OBJ_HOA flag is <NUM>, then the audio decoding device <NUM> may select an ambisonic renderer) from the audio renderers <NUM> for rendering the corresponding portion of the audio data <NUM>' obtained from the bitstream <NUM>.

The audio decoding device <NUM> may use the value of the RendererFlag_ENTIRE_SEPARATE flag to determine the level at which the value of the RendererFlag_OBJ_HOA is applicable. For instance, if the audio decoding device <NUM> determines that the value of the RendererFlag_ENTIRE_SEPARATE flag is <NUM>, then the audio decoding device <NUM> may render all of the audio objects of the bitstream <NUM> based on the value of a single instance of the RendererFlag_OBJ_HOA flag. Conversely, if the audio decoding device <NUM> determines that the value of the RendererFlag_ENTIRE_SEPARATE flag is <NUM>, then the audio decoding device <NUM> may render each audio object of the bitstream <NUM> individually based on the value of a respective corresponding instance of the RendererFlag_OBJ_HOA flag.

Additionally, the audio decoding device <NUM> may use the value of the RendererFlag_External_Internal flag to determine whether an external renderer or an internal renderer of the audio renderers <NUM> is to be used for rendering the corresponding portions of the bitstream <NUM>. If the RendererFlag_External_Internal flag is set to a value of <NUM>, the audio decoding device <NUM> may use an external renderer for rendering the corresponding audio data of the bitstream <NUM>, provided that the external renderer is available. If the RendererFlag_External_Internal flag is set to the value of <NUM> and the audio decoding device <NUM> determines that the external renderer is not available, the audio decoding device may use a reference renderer with ID <NUM> (as a default option) to render the corresponding audio data of the bitstream <NUM>. If the RendererFlag_External_Internal flag is set to a value of <NUM>, then the audio decoding device <NUM> may use an internal renderer of the audio renderers <NUM> to render the corresponding audio data of the bitstream <NUM>.

According to renderer transmission implementations of the techniques of this disclosure, the audio decoding device <NUM> may use the value of the RendererFlag_Transmitted_Reference flag to determine whether to use a renderer (e.g., a rendering matrix) explicitly signaled in the bitstream <NUM> for rendering the corresponding audio data, or to bypass any explicitly-rendered renderer and instead use a reference renderer to render the corresponding audio data of the bitstream <NUM>. If the audio decoding device <NUM> determines that the value of the RendererFlag_Transmitted_Reference flag is <NUM>, then the audio decoding device <NUM> may determine that one of the transmitted renderer(s) is to be used to render the corresponding audio data of the bitstream <NUM>. Conversely, if the audio decoding device <NUM> determines that the value of the RendererFlag_Transmitted_Reference flag is <NUM>, then the audio decoding device <NUM> may determine that one of the reference renderer(s) of the audio renderers <NUM> is to be used for rendering the corresponding audio data of the bitstream <NUM>.

In some examples, if the audio encoding device <NUM> determines that the audio renderers <NUM> accessible to the audio decoding device <NUM> might include multiple renderers of the same type (e.g., multiple ambisonic renderers or multiple object-based renderers), the audio encoding device may signal a rendererID syntax element in the bitstream <NUM>. In turn, the audio decoding device <NUM> may compare the value of the received rendererID syntax element to entries in a codebook. Upon detecting a match between the value of the received rendererID syntax element to a particular entry in the codebook, the audio decoding device <NUM>: It indicates the renderer ID.

This disclosure also includes various "soft" rendering techniques. The syntax for various soft rendering techniques of this disclosure is given in Table <NUM> above. In accordance with the soft rendering techniques of this disclosure, the audio decoding device may parse a SoftRendererParameter_OBJ_HOA bit-field from the bitstream <NUM>. The audio decoding device <NUM> may preserve the artistic intent of content producer based on the value(s) parsed from the bitstream <NUM> for the SoftRendererParameter_OBJ_HOA bit-field. For instance, according to the soft rendering techniques of this disclosure, the audio decoding device <NUM> may output a weighted combination of rendered object-domain audio data and rendered ambisonic-domain audio data.

In accordance with the soft rendering techniques of this disclosure, the audio decoding device <NUM> may use the RendererFlag_ENTIRE_SEPARATE flag, the RendererFlag_OBJ_HOA flag, the RendererFlag_External_Internal flag, the RendererFlag_Transmitted_Reference flag, and the rendererID syntax element in a manner similar to that described above with respect to other implementations of the renderer-selection techniques of this disclosure. In accordance with the soft rendering techniques of this disclosure, the audio decoding device <NUM> may additionally parse an alpha syntax element to obtain a soft rendering parameter value. The value of the alpha syntax element may be set between a lower bound (floor) of <NUM> and an upper bound (ceiling) of <NUM>. To implement the soft rendering techniques of this disclosure, the audio decoding device may perform the following operation to obtain the rendering output: <MAT>.

<FIG> is a diagram illustrating an example of a workflow with respect to object-domain audio data. Additional details on conventional object-based audio data processing can be found in ISO/IEC FDIS <NUM>-<NUM>:<NUM>(E), Information technology - High efficiency coding and media delivery in heterogeneous environments - Part <NUM>: 3D audio.

As shown in the example of <FIG>, an object encoder <NUM>, which may represent another example of the audio encoding device <NUM> shown in the example of <FIG>) may perform object encoding (e.g., according to the MPEG-H 3D Audio encoding standard referenced directly above) with respect to input object audio and object metadata (which is another way to refer to object-domain audio data) to obtain the bitstream <NUM>. The object encoder <NUM> may also output the renderer information <NUM> for an object renderer.

An object decoder <NUM> (which may represent another example of the audio decoding device <NUM>) may then perform audio decoding (e.g., according to the MPEG-H 3D Audio encoding standard referenced above) with respect to the bitstream <NUM> to obtain object-based audio data 11A'. The object decoder <NUM> may output the object-based audio data 11A' to a rendering matrix <NUM>, which may represent an example of the audio renderers <NUM> shown in the example of <FIG>. The audio playback system <NUM> may apply select the rendering matrix <NUM> based on the rendering information <NUM> or from among any object renderer. In any event, the rendering matrix <NUM> may output, based on the object-based audio data 11A', the speaker feeds <NUM>.

<FIG> is a diagram illustrating an example of a workflow in which object-domain audio data is converted to the ambisonic domain and rendered using ambisonic renderer(s). That is, the audio playback system <NUM> invokes an ambisonic conversion unit <NUM> to convert the object-based audio data 11A' from the spatial domain to the spherical harmonic domain and thereby obtain ambisonic coefficients <NUM> (and possibly HOA coefficient <NUM>). The audio playback system <NUM> may then select rendering matrix <NUM>, which is configured to render ambisonic audio data, including the ambisonic coefficients <NUM>, to obtain speaker feeds <NUM>.

To render an object-based input with ambisonic renderer(s) (such as a first order ambisonic renderer or a higher order ambisonic renderer), an audio rendering device may apply the following steps:.

<FIG> is a diagram illustrating a workflow of this disclosure, according to which a renderer type is signaled from the audio encoding device <NUM> to the audio decoding device <NUM>. According to the workflow illustrated in <FIG>, the audio encoding device <NUM> may transmit, to the audio decoding device <NUM>, information regarding which type of renderer shall be used for rendering the audio data of the bitstream <NUM>. According to the workflow illustrated in <FIG>, the audio decoding device <NUM> may use the signaled information (stored as the audio rendering information <NUM>) to select any object renderer or any ambisonic renderer available at the decoder end, e.g., a first order ambisonic renderer or a higher order ambisonic renderer. For instance, the workflow illustrated in <FIG> may use the RendererFlag_OBJ_HOA flag described above with respect to Tables <NUM> and <NUM>.

<FIG> is a is a diagram illustrating a workflow of this disclosure, according to which a renderer type and renderer identification information are signaled from the audio encoding device <NUM> to the audio decoding device <NUM>. According to the workflow illustrated in <FIG>, the audio encoding device <NUM> may transmit, to the audio decoding device <NUM>, information <NUM> regarding the type of renderer as well as which specific renderer shall be used for rendering the audio data of the bitstream <NUM>. According to the workflow illustrated in <FIG>, the audio decoding device <NUM> may use the signaled information (stored as the audio rendering information <NUM>) to select a particular object renderer or a particular ambisonic renderer available at the decoder end.

For instance, the workflow illustrated in <FIG> may use the RendererFlag_OBJ_HOA flag and the rendererID syntax element described above with respect to Tables <NUM> and <NUM>. The workflow illustrated in <FIG> may be particularly useful in scenarios in which the audio renderers <NUM> include multiple ambisonic renderers and/or multiple object-based renderers to select from. For instance, the audio decoding device <NUM> may match the value of the rendererID syntax element to an entry in a codebook to determine which particular audio renderer <NUM> to use for rendering the audio data <NUM>'.

<FIG> is a is a diagram illustrating a workflow of this disclosure, according to the renderer transmission implementations of the techniques of this disclosure. According to the workflow illustrated in <FIG>, the audio encoding device <NUM> may transmit, to the audio decoding device <NUM>, information regarding the type of renderer as well as the rendering matrix itself (as rendering information <NUM>) to be used for rendering the audio data of the bitstream <NUM>. According to the workflow illustrated in <FIG>, the audio decoding device <NUM> may use the signaled information (stored as the audio rendering information <NUM>) to add, if necessary, the signaled rendering matrix to the audio renderers <NUM>, and use the explicitly-signaled rendering matrix to render the audio data <NUM>'.

<FIG> is a flowchart illustrating example operation of the audio encoding device of <FIG> in performing example operation of the rendering techniques described in this disclosure. The audio encoding device <NUM> may store audio data <NUM> to a memory of a device (<NUM>). Next, the audio encoding device <NUM> may encode the audio data <NUM> to form encoded audio data (which is shown as the bitstream <NUM> in the example of <FIG>) (<NUM>). The audio encoding device <NUM> may select a renderer <NUM> associated with the encoded audio data <NUM> (<NUM>), where the selected renderer may include one of an object-based renderer or an ambisonic renderer. The audio encoding device <NUM> may then generate an encoded audio bitstream <NUM> comprising the encoded audio data and data indicative of the selected renderer (e.g., the rendering information <NUM>) (<NUM>).

<FIG> is a flowchart illustrating example operation of the audio decoding device of <FIG> in performing example operation of the rendering techniques described in this disclosure. The audio decoding device <NUM> may first store, to a memory of encoded audio data <NUM>' of an encoded audio bitstream <NUM> (<NUM>). The audio decoding device <NUM> may then parse a portion of the encoded audio data stored to the memory to select a renderer for the encoded audio data <NUM>' (<NUM>), where the selected renderer may include one of an object-based renderer or a ambisonic renderer. In this example it is assumed that the renderers <NUM> are incorporated within the audio decoding device <NUM>. As such, the audio encoding device <NUM> may apply one or more renderers to the encoded audio data <NUM>' to render the encoded audio data <NUM>' using the selected renderer <NUM> to generate one or more rendered speaker feeds <NUM> (<NUM>).

Other examples of context in which the techniques may be performed include an audio ecosystem that may include acquisition elements, and playback elements. The acquisition elements may include wired and/or wireless acquisition devices (e.g., Eigen microphones or EigenMike® microphones), on-device surround sound capture, and mobile devices (e.g., smartphones and tablets). In some examples, wired and/or wireless acquisition devices may be coupled to mobile device via wired and/or wireless communication channel(s).

As such, in some examples, this disclosure is directed to a device for rendering audio data. The device includes a memory and one or more processors in communication with the memory. The memory is configured to store encoded audio data of an encoded audio bitstream. The one or more processors are configured to parse a portion of the encoded audio data stored to the memory to select a renderer for the encoded audio data, the selected renderer comprising one of an object-based renderer or a ambisonic renderer, and to render the encoded audio data using the selected renderer to generate one or more rendered speaker feeds. In some implementations, the device includes an interface in communication with the memory. In these implementations, the interface is configured to receive the encoded audio bitstream. In some implementations, the device includes one or more loudspeakers in communication with the one or more processors. In these implementations, the one or more loudspeakers are configured to output the one or more rendered speaker feeds.

In some examples, the one or more processors comprise processing circuitry. In some examples, the one or more processors comprise an application-specific integrated circuit (ASIC). In some examples, the one or more processors are further configured to parse metadata of the encoded audio data to select the renderer. In some examples, the one or more processors are further configured to select the renderer based on a value of a RendererFlag_OBJ_HOA flag included in the parsed portion of the encoded video data. In some examples, the one or more processors are configured to parse a RendererFlag_ENTIRE_SEPARATE flag, to determine, based on a value of the RendererFlag_ENTIRE_SEPARATE flag being equal to <NUM>, that the value of the RendererFlag_OBJ_HOA applies to all objects of the encoded audio data rendered by the one or more processors, and to determine, based on a value of the RendererFlag_ENTIRE_SEPARATE flag being equal to <NUM>, that the value of the RendererFlag_OBJ_HOA applies to only a single object of the encoded audio data rendered by the one or more processors.

In some examples, the one or more processors are further configured to obtain a rendering matrix from the parsed portion of the encoded audio data, the obtained rendering matrix representing the selected renderer. In some examples, the one or more processors are further configured to obtain a rendererID syntax element from the parsed portion of the encoded audio data. In some examples, the one or more processors are further configured to select the renderer by matching a value of the rendererID syntax element to an entry of multiple entries of a codebook. In some examples, the one or more processors are further configured to obtain a SoftRendererParameter_OBJ_HOA flag from the parsed portion of the encoded audio data, to determine, based on a value of the SoftRendererParameter_OBJ_HOA flag, that portions of the encoded audio data are to be rendered using the object-based renderer and the ambisonic renderer, and to generate the one or more rendered speaker feeds using a weighted combination of rendered object-domain audio data and rendered ambisonic-domain audio data obtained from the portions of the encoded audio data.

In some examples, the one or more processors are further configured to determine a weighting associated with the weighted combination based on a value of an alpha syntax element obtained from the parsed portion of the encoded video data. In some examples, the selected renderer is the ambisonic renderer, and the one or more processors are further configured to decode a portion of the encoded audio data stored to the memory to reconstruct decoded object-based audio data and object metadata associated with the decoded object-based audio data, to convert the decoded object-based audio and the object metadata into an ambisonic domain to form ambisonic-domain audio data, and to render the ambisonic-domain audio data using the ambisonic renderer to generate the one or more rendered speaker feeds.

In some examples, the one or more processors are configured to obtain a rendering matrix from the parsed portion of the encoded audio data, the obtained rendering matrix representing the selected renderer, to parse a RendererFlag_Transmitted_Reference flag, to use, based on a value of the RendererFlag_Transmitted_Reference flag being equal to <NUM>, the obtained rendering matrix to render the encoded audio data, and to use, based on a value of the RendererFlag_Transmitted_Reference flag being equal to <NUM>, a reference renderer to render the encoded audio data.

In some examples, the one or more processors are configured to obtain a rendering matrix from the parsed portion of the encoded audio data, the obtained rendering matrix representing the selected renderer, to parse a RendererFlag_External_Internal flag, to determine, based on a value of the RendererFlag_External_Internal flag being equal to <NUM>, that the selected renderer is an external renderer, and to determine, based on the value of the RendererFlag_External_Internal flag being equal to <NUM>, that the selected renderer is an external renderer. In some examples, the value of the RendererFlag_External_Internal flag is equal to <NUM>, and the one or more processors are configured to determine that the external renderer is unavailable for rendering the encoded audio data, and to determine, based on the external renderer being unavailable for rendering the encoded audio data, that the selected renderer is a reference renderer.

As such, in some examples, this disclosure is directed to a device for encoding audio data. The device includes a memory, and one or more processors in communication with the memory. The memory is configured to store audio data. The one or more processors are configured to encode the audio data to form encoded audio data, to select a renderer associated with the encoded audio data, the selected renderer comprising one of an object-based renderer or a ambisonic renderer, and to generate an encoded audio bitstream comprising the encoded audio data and data indicative of the selected renderer. In some implementations, the device includes one or more microphones in communication with the memory. In these implementations, the one or more microphones are configured to receive the audio data. In some implementations, the device includes and interface in communication with the one or more processors. In these implementations, the interface is configured to signal the encoded audio bitstream.

In some examples, the one or more processors comprise processing circuitry. In some examples, the one or more processors comprise an application-specific integrated circuit (ASIC). In some examples, the one or more processors are further configured to include the data indicative of the selected renderer in metadata of the encoded audio data. In some examples, the one or more processors are further configured to include a RendererFlag_OBJ_HOA flag in the encoded audio bitstream, and wherein a value of a RendererFlag_OBJ_HOA flag is indicative of the selected renderer.

In some examples, the one or more processors are configured to set a value of a RendererFlag_ENTIRE_SEPARATE flag being equal to <NUM>, based on a determination that the value of the RendererFlag_OBJ_HOA applies to all objects of the encoded audio bitstream, to set the value of the RendererFlag_ENTIRE_SEPARATE flag being equal to <NUM>, based on a determination that the value of the RendererFlag_OBJ_HOA applies to only a single object of the encoded audio bitstream, and to include the RendererFlag_OBJ_HOA flag in the encoded audio bitstream. In some examples, the one or more processors are further configured to include a rendering matrix in the encoded audio bitstream, the rendering matrix representing the selected renderer.

In some examples, the one or more processors are further configured to include a rendererID syntax element in the encoded audio bitstream. In some examples, a value of the rendererID syntax element matches an entry of multiple entries of a codebook accessible to the one or more processors. In some examples, the one or more processors are further configured to determine that portions of the encoded audio data are to be rendered using the object-based renderer and the ambisonic renderer, and to include a SoftRendererParameter_OBJ_HOA flag in the encoded audio bitstream based on the determination that the portions of the encoded audio data are to be rendered using the object-based renderer and the ambisonic renderer.

In some examples, the one or more processors are further configured to determine a weighting associated with the SoftRendererParameter_OBJ_HOA flag; and include an alpha syntax element indicative of the weighting in the encoded audio bitstream. In some examples, the one or more processors are configured to include a RendererFlag_Transmitted_Reference flag in the encoded audio bitstream, and to include, based on a value of the RendererFlag_Transmitted_Reference flag being equal to <NUM>, a rendering matrix in the encoded audio bitstream, the rendering matrix representing the selected renderer. In some examples, the one or more processors are configured to set a value of a RendererFlag_External_Internal flag equal to <NUM>, based on a determination that the selected renderer is an external renderer, to set the value of the RendererFlag_External_Internal flag equal to <NUM>, based on a determination that the selected renderer is an external renderer, and to include the RendererFlag_External_Internal flag in the encoded audio bitstream.

In accordance with one or more techniques of this disclosure, the mobile device may be used to acquire a soundfield. For instance, the mobile device may acquire a soundfield via the wired and/or wireless acquisition devices and/or the on-device surround sound capture (e.g., a plurality of microphones integrated into the mobile device). The mobile device may then code the acquired soundfield into the ambisonic coefficients for playback by one or more of the playback elements. For instance, a user of the mobile device may record (acquire a soundfield of) a live event (e.g., a meeting, a conference, a play, a concert, etc.), and code the recording into ambisonic coefficients.

The mobile device may also utilize one or more of the playback elements to playback the ambisonic coded soundfield. For instance, the mobile device may decode the ambisonic coded soundfield and output a signal to one or more of the playback elements that causes the one or more of the playback elements to recreate the soundfield. As one example, the mobile device may utilize the wireless and/or wireless communication channels to output the signal to one or more speakers (e.g., speaker arrays, sound bars, etc.). As another example, the mobile device may utilize docking solutions to output the signal to one or more docking stations and/or one or more docked speakers (e.g., sound systems in smart cars and/or homes). As another example, the mobile device may utilize headphone rendering to output the signal to a set of headphones, e.g., to create realistic binaural sound.

In some examples, a particular mobile device may both acquire a 3D soundfield and playback the same 3D soundfield at a later time. In some examples, the mobile device may acquire a 3D soundfield, encode the 3D soundfield into ambisonic coefficients, and transmit the encoded 3D soundfield to one or more other devices (e.g., other mobile devices and/or other non-mobile devices) for playback.

Yet another context in which the techniques may be performed includes an audio ecosystem that may include audio content, game studios, coded audio content, rendering engines, and delivery systems. In some examples, the game studios may include one or more DAWs which may support editing of ambisonic signals. For instance, the one or more DAWs may include ambisonic plugins and/or tools which may be configured to operate with (e.g., work with) one or more game audio systems. In some examples, the game studios may output new stem formats that support ambisonic. In any case, the game studios may output coded audio content to the rendering engines which may render a soundfield for playback by the delivery systems.

The techniques may also be performed with respect to exemplary audio acquisition devices. For example, the techniques may be performed with respect to an EigenMike® microphone which may include a plurality of microphones that are collectively configured to record a 3D soundfield. In some examples, the plurality of microphones of EigenMike® microphone may be located on the surface of a substantially spherical ball with a radius of approximately <NUM>. In some examples, the audio encoding device <NUM> may be integrated into the Eigen microphone so as to output a bitstream <NUM> directly from the microphone.

Another exemplary audio acquisition context may include a production truck which may be configured to receive a signal from one or more microphones, such as one or more EigenMike® microphones. The production truck may also include an audio encoder, such as the audio encoding device <NUM> of <FIG> and <FIG>.

The mobile device may also, in some instances, include a plurality of microphones that are collectively configured to record a 3D soundfield. In other words, the plurality of microphone may have X, Y, Z diversity. In some examples, the mobile device may include a microphone which may be rotated to provide X, Y, Z diversity with respect to one or more other microphones of the mobile device. The mobile device may also include an audio encoder, such as the audio encoding device <NUM> of <FIG> and <FIG>.

A ruggedized video capture device may further be configured to record a 3D soundfield. In some examples, the ruggedized video capture device may be attached to a helmet of a user engaged in an activity. For instance, the ruggedized video capture device may be attached to a helmet of a user whitewater rafting. In this way, the ruggedized video capture device may capture a 3D soundfield that represents the action all around the user (e.g., water crashing behind the user, another rafter speaking in front of the user, etc..

The techniques may also be performed with respect to an accessory enhanced mobile device, which may be configured to record a 3D soundfield. In some examples, the mobile device may be similar to the mobile devices discussed above, with the addition of one or more accessories. For instance, an Eigen microphone may be attached to the above noted mobile device to form an accessory enhanced mobile device. In this way, the accessory enhanced mobile device may capture a higher quality version of the 3D soundfield than just using sound capture components integral to the accessory enhanced mobile device.

Example audio playback devices that may perform various aspects of the techniques described in this disclosure are further discussed below. In accordance with one or more techniques of this disclosure, speakers and/or sound bars may be arranged in any arbitrary configuration while still playing back a 3D soundfield. Moreover, in some examples, headphone playback devices may be coupled to a decoder <NUM> via either a wired or a wireless connection. In accordance with one or more techniques of this disclosure, a single generic representation of a soundfield may be utilized to render the soundfield on any combination of the speakers, the sound bars, and the headphone playback devices.

A number of different example audio playback environments may also be suitable for performing various aspects of the techniques described in this disclosure. For instance, a <NUM> speaker playback environment, a <NUM> (e.g., stereo) speaker playback environment, a <NUM> speaker playback environment with full height front loudspeakers, a <NUM> speaker playback environment, a <NUM> speaker playback environment, an automotive speaker playback environment, and a mobile device with ear bud playback environment may be suitable environments for performing various aspects of the techniques described in this disclosure.

In accordance with one or more techniques of this disclosure, a single generic representation of a soundfield may be utilized to render the soundfield on any of the foregoing playback environments. Additionally, the techniques of this disclosure enable a rendered to render a soundfield from a generic representation for playback on the playback environments other than that described above. For instance, if design considerations prohibit proper placement of speakers according to a <NUM> speaker playback environment (e.g., if it is not possible to place a right surround speaker), the techniques of this disclosure enable a render to compensate with the other <NUM> speakers such that playback may be achieved on a <NUM> speaker playback environment.

Moreover, a user may watch a sports game while wearing headphones. In accordance with one or more techniques of this disclosure, the 3D soundfield of the sports game may be acquired (e.g., one or more Eigen microphones or EigenMike® microphones may be placed in and/or around the baseball stadium), ambisonic coefficients corresponding to the 3D soundfield may be obtained and transmitted to a decoder, the decoder may reconstruct the 3D soundfield based on the ambisonic coefficients and output the reconstructed 3D soundfield to a renderer, the renderer may obtain an indication as to the type of playback environment (e.g., headphones), and render the reconstructed 3D soundfield into signals that cause the headphones to output a representation of the 3D soundfield of the sports game.

In each of the various instances described above, it should be understood that the audio encoding device <NUM> may perform a method or otherwise comprise means to perform each step of the method for which the audio encoding device <NUM> is configured to perform. In some instances, the means may comprise processing circuitry (e.g., fixed function circuitry and/or programmable processing circuitry) and/or one or more processors. In some instances, the one or more processors may represent a special purpose processor configured by way of instructions stored to a non-transitory computer-readable storage medium. In other words, various aspects of the techniques in each of the sets of encoding examples may provide for a non-transitory computer-readable storage medium having stored thereon instructions that, when executed, cause the one or more processors to perform the method for which the audio encoding device <NUM> has been configured to perform.

Computer-readable media may include computer-readable storage media, which corresponds to a tangible medium such as data storage media.

Likewise, in each of the various instances described above, it should be understood that the audio decoding device <NUM> may perform a method or otherwise comprise means to perform each step of the method for which the audio decoding device <NUM> is configured to perform. In some instances, the means may comprise one or more processors. In some instances, the one or more processors may represent a special purpose processor configured by way of instructions stored to a non-transitory computer-readable storage medium. In other words, various aspects of the techniques in each of the sets of encoding examples may provide for a non-transitory computer-readable storage medium having stored thereon instructions that, when executed, cause the one or more processors to perform the method for which the audio decoding device <NUM> has been configured to perform.

Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), processing circuitry (e.g. fixed function circuitry, programmable processing circuitry, or any combination thereof), or other equivalent integrated or discrete logic circuitry.

Claim 1:
A device (<NUM>, <NUM>; <NUM>) for encoding audio data, the device comprising:
a memory configured to store the audio data; and
one or more processors in communication with the memory, the one or more processors being configured to:
encode the audio data to form encoded audio data;
select a renderer (<NUM>) associated with the encoded audio data; and
generate an encoded audio bitstream (<NUM>) comprising the encoded audio data and data indicative of the selected renderer;
characterized in that
the selected renderer comprises one of an object-based renderer or an ambisonic renderer; and
wherein the one or more processors are further configured to include the data indicative of the selected renderer in metadata of the encoded audio data.