Patent Description:
During production of audio content, the sound engineer may render the audio content using a specific renderer in an attempt to tailor the audio content for target configurations of speakers used to reproduce the audio content. In other words, the sound engineer may render the audio content and playback the rendered audio content using speakers arranged in the targeted configuration. The sound engineer may then remix various aspects of the audio content, render the remixed audio content and again playback the rendered, remixed audio content using the speakers arranged in the targeted configuration. The sound engineer may iterate in this manner until a certain artistic intent is provided by the audio content. In this way, the sound engineer may produce audio content that provides a certain artistic intent or that otherwise provides a certain sound field during playback (e.g., to accompany video content played along with the audio content).

In "<NPL>et al propose a standard for the Ambisonics community. For the case that mixed order or reduced Ambisonic signal sets are beneficial, the format includes a simple matrix - named "adaptor matrix". This matrix can be freely configured to re-order, complete, re-normalize, or embed the transmitted or stored audio channels to a default set of periphonic Ambisonic signals.

In general, techniques are described for specifying audio rendering information in a bitstream representative of audio data. In other words, the techniques may provide for a way by which to signal audio rendering information used during audio content production to a playback device, which may then use the audio rendering information to render the audio content. Providing the rendering information in this manner enables the playback device to render the audio content in a manner intended by the sound engineer, and thereby potentially ensure appropriate playback of the audio content such that the artistic intent is potentially understood by a listener. In other words, the rendering information used during rendering by the sound engineer is provided in accordance with the techniques described in this disclosure so that the audio playback device may utilize the rendering information to render the audio content in a manner intended by the sound engineer, thereby ensuring a more consistent experience during both production and playback of the audio content in comparison to systems that do not provide this audio rendering information.

The invention is defined in the independent claim.

The details of one or more aspects of the techniques are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of these techniques will be apparent from the description and drawings, and from the claims.

The evolution of surround sound has made available many output formats for entertainment nowadays. Examples of such surround sound formats include the popular <NUM> format (which includes the following six channels: front left (FL), front right (FR), center or front center, back left or surround left, back right or surround right, and low frequency effects (LFE)), the growing <NUM> format, and the upcoming <NUM> format (e.g., for use with the Ultra High Definition Television standard). Further examples include formats for a spherical harmonic array.

The input to the future MPEG encoder is optionally one of three possible formats: (i) traditional channel-based audio, which is meant to be played through loudspeakers at pre-specified positions; (ii) object-based audio, which involves discrete pulse-code-modulation (PCM) data for single audio objects with associated metadata containing their location coordinates (amongst other information); and (iii) scene-based audio, which involves representing the sound field using coefficients of spherical harmonic basis functions (also called "spherical harmonic coefficients" or SHC).

There are various 'surround-sound' formats in the market. They range, for example, from the <NUM> home theatre system (which has been the most successful in terms of making inroads into living rooms beyond stereo) to the <NUM> system developed by NHK (Nippon Hoso Kyokai or Japan Broadcasting Corporation). Content creators (e.g., Hollywood studios) would like to produce the soundtrack for a movie once, and not spend the efforts to remix it for each speaker configuration. Recently, standard committees have been considering ways in which to provide an encoding into a standardized bitstream and a subsequent decoding that is adaptable and agnostic to the speaker geometry and acoustic conditions at the location of the renderer.

To provide such flexibility for content creators, a hierarchical set of elements may be used to represent a sound field. The hierarchical set of elements may refer to a set of elements in which the elements are ordered such that a basic set of lower-ordered elements provides a full representation of the modeled sound field. As the set is extended to include higher-order elements, the representation becomes more detailed.

One example of a hierarchical set of elements is a set of spherical harmonic coefficients (SHC). The following expression demonstrates a description or representation of a sound field using SHC: <MAT> This expression shows that the pressure pi at any point {rr, θr, φr} of the sound field can be represented uniquely by the SHC <MAT>. Here, <MAT>, c is the speed of sound (~<NUM>/s), {rr, θr, φr} is a point of reference (or observation point), jn(·) is the spherical Bessel function of order n, and <MAT> are the spherical harmonic basis functions of order n and suborder m. It can be recognized that the term in square brackets is a frequency-domain representation of the signal (i.e., S(ω, rr, θr, φr)) which can be approximated by various time-frequency transformations, such as the discrete Fourier transform (DFT), the discrete cosine transform (DCT), or a wavelet transform. Other examples of hierarchical sets include sets of wavelet transform coefficients and other sets of coefficients of multiresolution basis functions.

<FIG> is a diagram illustrating a zero-order spherical harmonic basis function <NUM>, first-order spherical harmonic basis functions 12A-12C and second-order spherical harmonic basis functions 14A-14E. The order is identified by the rows of the table, which are denoted as rows 16A-16C, with row 16A referring to the zero order, row 16B referring to the first order and row 16C referring to the second order. The sub-order is identified by the columns of the table, which are denoted as columns 18A-18E, with column 18A referring to the zero suborder, column 18B referring to the first suborder, column 18C referring to the negative first suborder, column 18D referring to the second suborder and column 18E referring to the negative second suborder. The SHC corresponding to zero-order spherical harmonic basis function <NUM> may be considered as specifying the energy of the sound field, while the SHCs corresponding to the remaining higher-order spherical harmonic basis functions (e.g., spherical harmonic basis functions 12A-12C and 14A-14E) may specify the direction of that energy.

<FIG> is a diagram illustrating spherical harmonic basis functions from the zero order (n = <NUM>) to the fourth order (n = <NUM>). As can be seen, for each order, there is an expansion of suborders m which are shown but not explicitly noted in the example of <FIG> for ease of illustration purposes.

<FIG> is another diagram illustrating spherical harmonic basis functions from the zero order (n = <NUM>) to the fourth order (n = <NUM>). In <FIG>, the spherical harmonic basis functions are shown in three-dimensional coordinate space with both the order and the suborder shown.

In any event, the SHC <MAT> can either be physically acquired (e.g., recorded) by various microphone array configurations or, alternatively, they can be derived from channel-based or object-based descriptions of the sound field. The former represents scene-based audio input to an encoder. For example, a fourth-order representation involving <NUM>+<NUM><NUM> (<NUM>, and hence fourth order) coefficients may be used.

To illustrate how these SHCs may be derived from an object-based description, consider the following equation. The coefficients <MAT> for the sound field corresponding to an individual audio object may be expressed as <MAT> where i is <MAT> is the spherical Hankel function (of the second kind) of order n, and {rs, θs, φs} is the location of the object. Knowing the source energy g(ω) as a function of frequency (e.g., using time-frequency analysis techniques, such as performing a fast Fourier transform on the PCM stream) allows us to convert each PCM object and its location into the SHC <MAT>. Further, it can be shown (since the above is a linear and orthogonal decomposition) that the <MAT> coefficients for each object are additive. In this manner, a multitude of PCM objects can be represented by the <MAT> coefficients (e.g., as a sum of the coefficient vectors for the individual objects). Essentially, these coefficients contain information about the sound field (the pressure as a function of 3D coordinates), and the above represents the transformation from individual objects to a representation of the overall sound field, in the vicinity of the observation point {rr, θr, φr}. The remaining figures are described below in the context of object-based and SHC-based audio coding.

<FIG> is a block diagram illustrating a system <NUM> that may perform the techniques described in this disclosure to signal rendering information in a bitstream representative of audio data. As shown in the example of <FIG>, system <NUM> includes a content creator <NUM> and a content consumer <NUM>. The content creator <NUM> may represent a movie studio or other entity that may generate multi-channel audio content for consumption by content consumers, such as the content consumer <NUM>. Often, this content creator generates audio content in conjunction with video content. The content consumer <NUM> represents an individual that owns or has access to an audio playback system <NUM>, which may refer to any form of audio playback system capable of playing back multi-channel audio content. In the example of <FIG>, the content consumer <NUM> includes the audio playback system <NUM>.

The content creator <NUM> includes an audio renderer <NUM> and an audio editing system <NUM>. The audio renderer <NUM> may represent an audio processing unit that renders or otherwise generates speaker feeds (which may also be referred to as "loudspeaker feeds," "speaker signals," or "loudspeaker signals"). Each speaker feed may correspond to a speaker feed that reproduces sound for a particular channel of a multi-channel audio system. In the example of <FIG>, the renderer <NUM> may render speaker feeds for conventional <NUM>, <NUM> or <NUM> surround sound formats, generating a speaker feed for each of the <NUM>, <NUM> or <NUM> speakers in the <NUM>, <NUM> or <NUM> surround sound speaker systems. Alternatively, the renderer <NUM> may be configured to render speaker feeds from source spherical harmonic coefficients for any speaker configuration having any number of speakers, given the properties of source spherical harmonic coefficients discussed above. The renderer <NUM> may, in this manner, generate a number of speaker feeds, which are denoted in <FIG> as speaker feeds <NUM>.

The content creator <NUM> may, during the editing process, render spherical harmonic coefficients <NUM> ("SHC <NUM>") to generate speaker feeds, listening to the speaker feeds in an attempt to identify aspects of the sound field that do not have high fidelity or that do not provide a convincing surround sound experience. The content creator <NUM> may then edit source spherical harmonic coefficients (often indirectly through manipulation of different objects from which the source spherical harmonic coefficients may be derived in the manner described above). The content creator <NUM> may employ an audio editing system <NUM> to edit the spherical harmonic coefficients <NUM>. The audio editing system <NUM> represents any system capable of editing audio data and outputting this audio data as one or more source spherical harmonic coefficients.

When the editing process is complete, the content creator <NUM> may generate the bitstream <NUM> based on the spherical harmonic coefficients <NUM>. That is, the content creator <NUM> includes a bitstream generation device <NUM>, which may represent any device capable of generating the bitstream <NUM>. In some instances, the bitstream generation device <NUM> may represent an encoder that bandwidth compresses (through, as one example, entropy encoding) the spherical harmonic coefficients <NUM> and that arranges the entropy encoded version of the spherical harmonic coefficients <NUM> in an accepted format to form the bitstream <NUM>. In other instances, the bitstream generation device <NUM> may represent an audio encoder (possibly, one that complies with a known audio coding standard, such as MPEG surround, or a derivative thereof) that encodes the multi-channel audio content <NUM> using, as one example, processes similar to those of conventional audio surround sound encoding processes to compress the multi-channel audio content or derivatives thereof. The compressed multi-channel audio content <NUM> may then be entropy encoded or coded in some other way to bandwidth compress the content <NUM> and arranged in accordance with an agreed upon format to form the bitstream <NUM>. Whether directly compressed to form the bitstream <NUM> or rendered and then compressed to form the bitstream <NUM>, the content creator <NUM> may transmit the bitstream <NUM> to the content consumer <NUM>.

While shown in <FIG> as being directly transmitted to the content consumer <NUM>, the content creator <NUM> may output the bitstream <NUM> to an intermediate device positioned between the content creator <NUM> and the content consumer <NUM>. This intermediate device may store the bitstream <NUM> for later delivery to the content consumer <NUM>, which may request this bitstream. The intermediate device may comprise a file server, a web server, a desktop computer, a laptop computer, a tablet computer, a mobile phone, a smart phone, or any other device capable of storing the bitstream <NUM> for later retrieval by an audio decoder. Alternatively, the content creator <NUM> may store the bitstream <NUM> to a storage medium, such as a compact disc, a digital video disc, a high definition video disc or other storage mediums, most of which are capable of being read by a computer and therefore may be referred to as computer-readable storage mediums. In this context, the transmission channel may refer to those channels by which content stored to these mediums are transmitted (and may include retail stores and other store-based delivery mechanism). In any event, the techniques of this disclosure should not therefore be limited in this respect to the example of <FIG>.

As further shown in the example of <FIG>, the content consumer <NUM> includes an audio playback system <NUM>. The audio playback system <NUM> may represent any audio playback system capable of playing back multi-channel audio data. The audio playback system <NUM> may include a number of different renderers <NUM>. The renderers <NUM> may each provide for a different form of rendering, where the different forms of rendering may include one or more of the various ways of performing vector-base amplitude panning (VBAP), one or more of the various ways of performing distance based amplitude panning (DBAP), one or more of the various ways of performing simple panning, one or more of the various ways of performing near field compensation (NFC) filtering and/or one or more of the various ways of performing wave field synthesis.

The audio playback system <NUM> may further include an extraction device <NUM>. The extraction device <NUM> may represent any device capable of extracting the spherical harmonic coefficients <NUM>' ("SHC <NUM>'," which may represent a modified form of or a duplicate of the spherical harmonic coefficients <NUM>) through a process that may generally be reciprocal to that of the bitstream generation device <NUM>. In any event, the audio playback system <NUM> may receive the spherical harmonic coefficients <NUM>'. The audio playback system <NUM> may then select one of renderers <NUM>, which then renders the spherical harmonic coefficients <NUM>' to generate a number of speaker feeds <NUM> (corresponding to the number of loudspeakers electrically or possibly wirelessly coupled to the audio playback system <NUM>, which are not shown in the example of <FIG> for ease of illustration purposes).

Typically, the audio playback system <NUM> may select any one the of audio renderers <NUM> and may be configured to select the one or more of audio renderers <NUM> depending on the source from which the bitstream <NUM> is received (such as a DVD player, a Blu-ray player, a smartphone, a tablet computer, a gaming system, and a television to provide a few examples). While any one of the audio renderers <NUM> may be selected, often the audio renderer used when creating the content provides for a better (and possibly the best) form of rendering due to the fact that the content was created by the content creator <NUM> using this one of audio renderers, i.e., the audio renderer <NUM> in the example of <FIG>. Selecting the one of the audio renderers <NUM> that is the same or at least close (in terms of rendering form) may provide for a better representation of the sound field and may result in a better surround sound experience for the content consumer <NUM>.

In accordance with the techniques described in this disclosure, the bitstream generation device <NUM> may generate the bitstream <NUM> to include the audio rendering information <NUM> ("audio rendering info <NUM>"). The audio rendering information <NUM> may include a signal value identifying an audio renderer used when generating the multi-channel audio content, i.e., the audio renderer <NUM> in the example of <FIG>. In some instances, the signal value includes a matrix used to render spherical harmonic coefficients to a plurality of speaker feeds.

In some instances, the signal value includes two or more bits that define an index that indicates that the bitstream includes a matrix used to render spherical harmonic coefficients to a plurality of speaker feeds. In some instances, when an index is used, the signal value further includes two or more bits that define a number of rows of the matrix included in the bitstream and two or more bits that define a number of columns of the matrix included in the bitstream. Using this information and given that each coefficient of the two-dimensional matrix is typically defined by a <NUM>-bit floating point number, the size in terms of bits of the matrix may be computed as a function of the number of rows, the number of columns, and the size of the floating point numbers defining each coefficient of the matrix, i.e., <NUM>-bits in this example.

In some instances, the signal value specifies a rendering algorithm used to render spherical harmonic coefficients to a plurality of speaker feeds. The rendering algorithm may include a matrix that is known to both the bitstream generation device <NUM> and the extraction device <NUM>. That is, the rendering algorithm may include application of a matrix in addition to other rendering steps, such as panning (e.g., VBAP, DBAP or simple panning) or NFC filtering. In some instances, the signal value includes two or more bits that define an index associated with one of a plurality of matrices used to render spherical harmonic coefficients to a plurality of speaker feeds. Again, both the bitstream generation device <NUM> and the extraction device <NUM> may be configured with information indicating the plurality of matrices and the order of the plurality of matrices such that the index may uniquely identify a particular one of the plurality of matrices. Alternatively, the bitstream generation device <NUM> may specify data in the bitstream <NUM> defining the plurality of matrices and/or the order of the plurality of matrices such that the index may uniquely identify a particular one of the plurality of matrices.

In some instances, the signal value includes two or more bits that define an index associated with one of a plurality of rendering algorithms used to render spherical harmonic coefficients to a plurality of speaker feeds. Again, both the bitstream generation device <NUM> and the extraction device <NUM> may be configured with information indicating the plurality of rendering algorithms and the order of the plurality of rendering algorithms such that the index may uniquely identify a particular one of the plurality of matrices. Alternatively, the bitstream generation device <NUM> may specify data in the bitstream <NUM> defining the plurality of matrices and/or the order of the plurality of matrices such that the index may uniquely identify a particular one of the plurality of matrices.

In some instances, the bitstream generation device <NUM> specifies audio rendering information <NUM> on a per audio frame basis in the bitstream. In other instances, bitstream generation device <NUM> specifies the audio rendering information <NUM> a single time in the bitstream.

The extraction device <NUM> may then determine audio rendering information <NUM> specified in the bitstream. Based on the signal value included in the audio rendering information <NUM>, the audio playback system <NUM> may render a plurality of speaker feeds <NUM> based on the audio rendering information <NUM>. As noted above, the signal value may in some instances include a matrix used to render spherical harmonic coefficients to a plurality of speaker feeds. In this case, the audio playback system <NUM> may configure one of the audio renderers <NUM> with the matrix, using this one of the audio renderers <NUM> to render the speaker feeds <NUM> based on the matrix.

In some instances, the signal value includes two or more bits that define an index that indicates that the bitstream includes a matrix used to render the spherical harmonic coefficients <NUM>' to the speaker feeds <NUM>. The extraction device <NUM> may parse the matrix from the bitstream in response to the index, whereupon the audio playback system <NUM> may configure one of the audio renderers <NUM> with the parsed matrix and invoke this one of the renderers <NUM> to render the speaker feeds <NUM>. When the signal value includes two or more bits that define a number of rows of the matrix included in the bitstream and two or more bits that define a number of columns of the matrix included in the bitstream, the extraction device <NUM> may parse the matrix from the bitstream in response to the index and based on the two or more bits that define a number of rows and the two or more bits that define the number of columns in the manner described above.

In some instances, the signal value specifies a rendering algorithm used to render the spherical harmonic coefficients <NUM>' to the speaker feeds <NUM>. In these instances, some or all of the audio renderers <NUM> may perform these rendering algorithms. The audio playback device <NUM> may then utilize the specified rendering algorithm, e.g., one of the audio renderers <NUM>, to render the speaker feeds <NUM> from the spherical harmonic coefficients <NUM>'.

When the signal value includes two or more bits that define an index associated with one of a plurality of matrices used to render the spherical harmonic coefficients <NUM>' to the speaker feeds <NUM>, some or all of the audio renderers <NUM> may represent this plurality of matrices. Thus, the audio playback system <NUM> may render the speaker feeds <NUM> from the spherical harmonic coefficients <NUM>' using the one of the audio renderers <NUM> associated with the index.

When the signal value includes two or more bits that define an index associated with one of a plurality of rendering algorithms used to render the spherical harmonic coefficients <NUM>' to the speaker feeds <NUM>, some or all of the audio renderers <NUM> may represent these rendering algorithms. Thus, the audio playback system <NUM> may render the speaker feeds <NUM> from the spherical harmonic coefficients <NUM>' using one of the audio renderers <NUM> associated with the index.

Depending on the frequency with which this audio rendering information is specified in the bitstream, the extraction device <NUM> may determine the audio rendering information <NUM> on a per audio frame basis or a single time.

By specifying the audio rendering information <NUM> in this manner, the techniques may potentially result in better reproduction of the multi-channel audio content <NUM> and according to the manner in which the content creator <NUM> intended the multi-channel audio content <NUM> to be reproduced. As a result, the techniques may provide for a more immersive surround sound or multi-channel audio experience.

While described as being signaled (or otherwise specified) in the bitstream, the audio rendering information <NUM> may be specified as metadata separate from the bitstream or, in other words, as side information separate from the bitstream. The bitstream generation device <NUM> may generate this audio rendering information <NUM> separate from the bitstream <NUM> so as to maintain bitstream compatibility with (and thereby enable successful parsing by) those extraction devices that do not support the techniques described in this disclosure. Accordingly, while described as being specified in the bitstream, the techniques may allow for other ways by which to specify the audio rendering information <NUM> separate from the bitstream <NUM>.

Moreover, while described as being signaled or otherwise specified in the bitstream <NUM> or in metadata or side information separate from the bitstream <NUM>, the techniques may enable the bitstream generation device <NUM> to specify a portion of the audio rendering information <NUM> in the bitstream <NUM> and a portion of the audio rendering information <NUM> as metadata separate from the bitstream <NUM>. For example, the bitstream generation device <NUM> may specify the index identifying the matrix in the bitstream <NUM>, where a table specifying a plurality of matrixes that includes the identified matrix may be specified as metadata separate from the bitstream. The audio playback system <NUM> may then determine the audio rendering information <NUM> from the bitstream <NUM> in the form of the index and from the metadata specified separately from the bitstream <NUM>. The audio playback system <NUM> may, in some instances, be configured to download or otherwise retrieve the table and any other metadata from a pre-configured or configured server (most likely hosted by the manufacturer of the audio playback system <NUM> or a standards body).

In other words and as noted above, Higher-Order Ambisonics (HOA) may represent a way by which to describe directional information of a sound-field based on a spatial Fourier transform. Typically, the higher the Ambisonics order N, the higher the spatial resolution, the larger the number of spherical harmonics (SH) coefficients (N+<NUM>)^<NUM>, and the larger the required bandwidth for transmitting and storing the data.

A potential advantage of this description is the possibility to reproduce this soundfield on most any loudspeaker setup (e.g., <NUM>, <NUM><NUM>,. The conversion from the soundfield description into M loudspeaker signals may be done via a static rendering matrix with (N+<NUM>)<NUM> inputs and M outputs. Consequently, every loudspeaker setup may require a dedicated rendering matrix. Several algorithms may exist for computing the rendering matrix for a desired loudspeaker setup, which may be optimized for certain objective or subjective measures, such as the Gerzon criteria. For irregular loudspeaker setups, algorithms may become complex due to iterative numerical optimization procedures, such as convex optimization. To compute a rendering matrix for irregular loudspeaker layouts without waiting time, it may be beneficial to have sufficient computation resources available. Irregular loudspeaker setups may be common in domestic living room environments due to architectural constrains and aesthetic preferences. Therefore, for the best soundfield reproduction, a rendering matrix optimized for such scenario may be preferred in that it may enable reproduction of the soundfield more accurately.

Because an audio decoder usually does not require much computational resources, the device may not be able to compute an irregular rendering matrix in a consumer-friendly time. Various aspects of the techniques described in this disclosure may provide for the use a cloud-based computing approach as follows:.

This approach may allow the manufacturer to keep manufacturing costs of an audio decoder low (because a powerful processor may not be needed to compute these irregular rendering matrices), while also facilitating a more optimal audio reproduction in comparison to rendering matrices usually designed for regular speaker configurations or geometries. The algorithm for computing the rendering matrix may also be optimized after an audio decoder has shipped, potentially reducing the costs for hardware revisions or even recalls. The techniques may also, in some instances, gather a lot of information about different loudspeaker setups of consumer products which may be beneficial for future product developments.

<FIG> is a block diagram illustrating another system <NUM> that may perform other aspects of the techniques described in this disclosure. While shown as a separate system from system <NUM>, both system <NUM> and system <NUM> may be integrated within or otherwise performed by a single system. In the example of <FIG> described above, the techniques were described in the context of spherical harmonic coefficients. However, the techniques may likewise be performed with respect to any representation of a sound field, including representations that capture the sound field as one or more audio objects. An example of audio objects may include pulse-code modulation (PCM) audio objects. Thus, system <NUM> represents a similar system to system <NUM>, except that the techniques may be performed with respect to audio objects <NUM> and <NUM>' instead of spherical harmonic coefficients <NUM> and <NUM>'.

In this context, audio rendering information <NUM> may, in some instances, specify a rendering algorithm, i.e., the one employed by audio renderer <NUM> in the example of <FIG>, used to render audio objects <NUM> to speaker feeds <NUM>. In other instances, audio rendering information <NUM> includes two or more bits that define an index associated with one of a plurality of rendering algorithms, i.e., the one associated with audio renderer <NUM> in the example of <FIG>, used to render audio objects <NUM> to speaker feeds <NUM>.

When audio rendering information <NUM> specifies a rendering algorithm used to render audio objects <NUM>' to the plurality of speaker feeds, some or all of audio renderers <NUM> may represent or otherwise perform different rendering algorithms. Audio playback system <NUM> may then render speaker feeds <NUM> from audio objects <NUM>' using the one of audio renderers <NUM>.

In instances where audio rendering information <NUM> includes two or more bits that define an index associated with one of a plurality of rendering algorithms used to render audio objects <NUM> to speaker feeds <NUM>, some or all of audio renderers <NUM> may represent or otherwise perform different rendering algorithms. Audio playback system <NUM> may then render speaker feeds <NUM> from audio objects <NUM>' using the one of audio renderers <NUM> associated with the index.

While described above as comprising two-dimensional matrices, the techniques may be implemented with respect to matrices of any dimension. In some instances, the matrices may only have real coefficients. In other instances, the matrices may include complex coefficients, where the imaginary components may represent or introduce an additional dimension. Matrices with complex coefficients may be referred to as filters in some contexts.

The following is one way to summarize the foregoing techniques. With object or Higher-order Ambisonics (HoA)-based 3D/2D soundfield reconstruction, there may be a renderer involved. There may be two uses for the renderer. The first use may be to take into account the local conditions (such as the number and geometry of loudspeakers) to optimize the soundfield reconstruction in the local acoustic landscape. The second use may be to provide it to the sound-artist, at the time of the content-creation, e.g., such that he/she may provide the artistic intent of the content. One potential problem being addressed is to transmit, along with the audio content, information on which renderer was used to create the content.

The techniques described in this disclosure may provide for one or more of: (i) transmission of the renderer (in a typical HoA embodiment- this is a matrix of size NxM, where N is the number of loudspeakers and M is the number of HoA coefficients) or (ii) transmission of an index to a table of renderers that is universally known.

Again, while described as being signaled (or otherwise specified) in the bitstream, the audio rendering information <NUM> may be specified as metadata separate from the bitstream or, in other words, as side information separate from the bitstream. The bitstream generation device <NUM> may generate this audio rendering information <NUM> separate from the bitstream <NUM> so as to maintain bitstream compatibility with (and thereby enable successful parsing by) those extraction devices that do not support the techniques described in this disclosure. Accordingly, while described as being specified in the bitstream, the techniques may allow for other ways by which to specify the audio rendering information <NUM> separate from the bitstream <NUM>.

<FIG> is a block diagram illustrating another system <NUM> that may perform various aspects of the techniques described in this disclosure. While shown as a separate system from the system <NUM> and the system <NUM>, various aspects of the systems <NUM>, <NUM> and <NUM> may be integrated within or otherwise performed by a single system. The system <NUM> may be similar to systems <NUM> and <NUM> except that the system <NUM> may operate with respect to audio content <NUM>, which may represent one or more of audio objects similar to audio objects <NUM> and SHC similar to SHC <NUM>. Additionally, the system <NUM> may not signal the audio rendering information <NUM> in the bitstream <NUM> as described above with respect to the examples of <FIG> and <FIG>, but instead signal this audio rendering information <NUM> as metadata <NUM> separate from the bitstream <NUM>.

<FIG> is a block diagram illustrating another system <NUM> that may perform various aspects of the techniques described in this disclosure. While shown as a separate system from the systems <NUM>, <NUM> and <NUM>, various aspects of the systems <NUM>, <NUM>, <NUM> and <NUM> may be integrated within or otherwise performed by a single system. The system <NUM> may be similar to system <NUM> except that the system <NUM> may signal a portion of the audio rendering information <NUM> in the bitstream <NUM> as described above with respect to the examples of <FIG> and <FIG> and signal a portion of this audio rendering information <NUM> as metadata <NUM> separate from the bitstream <NUM>. In some examples, the bitstream generation device <NUM> may output metadata <NUM>, which may then be uploaded to a server or other device. The audio playback system <NUM> may then download or otherwise retrieve this metadata <NUM>, which is then used to augment the audio rendering information extracted from the bitstream <NUM> by the extraction device <NUM>.

<FIG> are diagram illustrating bitstreams 31A-31D formed in accordance with the techniques described in this disclosure. In the example of <FIG>, bitstream 31A may represent one example of bitstream <NUM> shown in <FIG>, <FIG> and <FIG> above. The bitstream 31A includes audio rendering information 39A that includes one or more bits defining a signal value <NUM>. This signal value <NUM> may represent any combination of the below described types of information. The bitstream 31A also includes audio content <NUM>, which may represent one example of the audio content <NUM>.

In the example of <FIG>, the bitstream 31B may be similar to the bitstream 31A where the signal value <NUM> comprises an index 54A, one or more bits defining a row size 54B of the signaled matrix, one or more bits defining a column size 54C of the signaled matrix, and matrix coefficients 54D. The index 54A may be defined using two to five bits, while each of row size 54B and column size 54C may be defined using two to sixteen bits.

The extraction device <NUM> may extract the index 54A and determine whether the index signals that the matrix is included in the bitstream 31B (where certain index values, such as <NUM> or <NUM>, may signal that the matrix is explicitly specified in bitstream 31B). In the example of <FIG>, the bitstream 31B includes an index 54A signaling that the matrix is explicitly specified in the bitstream 31B. As a result, the extraction device <NUM> may extract the row size 54B and the column size 54C. The extraction device <NUM> may be configured to compute the number of bits to parse that represent matrix coefficients as a function of the row size 54B, the column size 54C and a signaled (not shown in <FIG>) or implicit bit size of each matrix coefficient. Using these determined number of bits, the extraction device <NUM> may extract the matrix coefficients 54D, which the audio playback device <NUM> may use to configure one of the audio renderers <NUM> as described above. While shown as signaling the audio rendering information 39B a single time in the bitstream 31B, the audio rendering information 39B may be signaled multiple times in bitstream 31B or at least partially or fully in a separate out-of-band channel (as optional data in some instances).

In the example of <FIG>, the bitstream 31C represents one example of bitstream <NUM> shown in <FIG>, <FIG> and <FIG> above. The bitstream 31C includes the audio rendering information 39C that includes a signal value <NUM>, which in this example specifies an algorithm index 54E. The bitstream 31C also includes audio content <NUM>. The algorithm index 54E may be defined using two to five bits, as noted above, where this algorithm index 54E may identify a rendering algorithm to be used when rendering the audio content <NUM>.

The extraction device <NUM> extracts the algorithm index 50E and determines whether the algorithm index 54E signals that the matrix are included in the bitstream 31C (where certain index values, such as <NUM> or <NUM>, signal that the matrix is explicitly specified in bitstream 31C). In the example of <FIG>, the bitstream 31C includes the algorithm index 54E signaling that the matrix is not explicitly specified in bitstream 31C. As a result, the extraction device <NUM> forwards the algorithm index 54E to audio playback device, which selects the corresponding one (if available) the rendering algorithms (which are denoted as renderers <NUM> in the example of <FIG>). While shown as signaling audio rendering information 39C a single time in the bitstream 31C, in the example of <FIG>, audio rendering information 39C may be signaled multiple times in the bitstream.

In the example of <FIG>, the bitstream 31C may represent one example of bitstream <NUM> shown in <FIG>, <FIG> and <FIG> above. The bitstream 31D includes the audio rendering information 39D that includes a signal value <NUM>, which in this example specifies a matrix index 54F. The bitstream 31D also includes audio content <NUM>. The matrix index 54F may be defined using two to five bits, as noted above, where this matrix index 54F may identify a rendering algorithm to be used when rendering the audio content <NUM>.

The extraction device <NUM> may extract the matrix index 50F and determine whether the matrix index 54F signals that the matrix are included in the bitstream 31D (where certain index values, such as <NUM> or <NUM>, may signal that the matrix is explicitly specified in bitstream 31C). In the example of <FIG>, the bitstream 31D includes the matrix index 54F signaling that the matrix is not explicitly specified in bitstream 31D. As a result, the extraction device <NUM> forwards the matrix index 54F to audio playback device, which selects the corresponding one (if available) the renderes <NUM>. While shown as signaling audio rendering information 39D a single time in the bitstream 31D, in the example of <FIG>, audio rendering information 39D may be signaled multiple times in the bitstream 31D or at least partially or fully in a separate out-of-band channel (as optional data in some instances).

<FIG> is a flowchart illustrating example operation of a system, such as one of systems <NUM>, <NUM>, <NUM> and <NUM> shown in the examples of <FIG>, in performing various aspects of the techniques described in this disclosure. Although described below with respect to system <NUM>, the techniques discussed with respect to <FIG> may also be implemented by any one of system <NUM>, <NUM> and <NUM>.

As discussed above, the content creator <NUM> may employ audio editing system <NUM> to create or edit captured or generated audio content (which is shown as the SHC <NUM> in the example of <FIG>). The content creator <NUM> may then render the SHC <NUM> using the audio renderer <NUM> to generated multi-channel speaker feeds <NUM>, as discussed in more detail above (<NUM>). The content creator <NUM> may then play these speaker feeds <NUM> using an audio playback system and determine whether further adjustments or editing is required to capture, as one example, the desired artistic intent (<NUM>). When further adjustments are desired ("YES" <NUM>), the content creator <NUM> may remix the SHC <NUM> (<NUM>), render the SHC <NUM> (<NUM>), and determine whether further adjustments are necessary (<NUM>). When further adjustments are not desired ("NO" <NUM>), the bitstream generation device <NUM> may generate the bitstream <NUM> representative of the audio content (<NUM>). The bitstream generation device <NUM> may also generate and specify the audio rendering information <NUM> in the bitstream <NUM>, as described in more detail above (<NUM>).

The content consumer <NUM> may then obtain the bitstream <NUM> and the audio rendering information <NUM> (<NUM>). As one example, the extraction device <NUM> may then extract the audio content (which is shown as the SHC <NUM>' in the example of <FIG>) and the audio rendering information <NUM> from the bitstream <NUM>. The audio playback device <NUM> may then render the SHC <NUM>' based on the audio rendering information <NUM> in the manner described above (<NUM>) and play the rendered audio content (<NUM>).

It should be understood that, depending on the example, certain acts or events of any of the methods described herein can be performed in a different sequence, may be added, merged, or left out altogether (e.g., not all described acts or events are necessary for the practice of the method). In addition, while certain aspects of this disclosure are described as being performed by a single device, module or unit for purposes of clarity, it should be understood that the techniques of this disclosure may be performed by a combination of devices, units or modules.

In one or more examples, the functions described may be implemented in hardware or a combination of hardware and software (which may include firmware). If implemented in software, the functions may be stored on or transmitted over as one or more instructions or code on a non-transitory computer-readable medium and executed by a hardware-based processing unit.

Disk and disc, as used herein, includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers.

Claim 1:
An audio playback device (<NUM>) configured to render multi-channel audio content (<NUM>) from a bitstream (31C), the device comprising one or more processors configured to:
extract an algorithm index (54E) from the bitstream, wherein the algorithm index identifies a rendering algorithm to be used when rendering the audio content (<NUM>);
determine whether the algorithm index (54E) indicates that a matrix of the rendering algorithm is explicitly specified in the bitstream (31C); and
if the determination is that the algorithm index (54E) indicates that the matrix of the rendering algorithm is not explicitly specified in the bitstream, select from a plurality of rendering algorithms (<NUM>), a rendering algorithm identified by the algorithm index, and render a plurality of speaker feeds based on the selected rendering algorithm.