Patent Description:
In the present disclosure and the appended claims:.

In last years, the generation, recording, representation, coding, transmission, and reproduction of audio is moving towards enhanced, interactive and immersive experience for the listener. The immersive experience can be described e.g. as a state of being deeply engaged or involved in a sound scene while the sounds are coming from all directions. In immersive audio (also called 3D audio), the sound image is reproduced in all <NUM> dimensions around the listener taking into account a wide range of sound characteristics like timbre, directivity, reverberation, transparency and accuracy of (auditory) spaciousness. Immersive audio is produced for given reproduction systems, i.e. loudspeaker configurations, integrated reproduction systems (sound bars) or headphones. Then interactivity of an audio reproduction system can include e.g. an ability to adjust sound levels, change positions of sounds, or select different languages for the reproduction.

There are three fundamental approaches (also referred below as audio formats) to achieve an immersive audio experience.

A first approach is a channel-based audio where multiple spaced microphones are used to capture sounds from different directions while one microphone corresponds to one audio channel in a specific loudspeaker layout. Each recorded channel is supplied to a loudspeaker in a particular location. Examples of channel-based audio comprise, for example, stereo, <NUM> surround, <NUM>+<NUM> etc..

A second approach is a scene-based audio which represents a desired sound field over a localized space as a function of time by a combination of dimensional components. The signals representing the scene-based audio are independent of the audio sources positions while the sound field has to be transformed to a chosen loudspeakers layout at the rendering reproduction system. An example of scene-based audio is ambisonics.

A third, last immersive audio approach is an object-based audio which represents an auditory scene as a set of individual audio elements (for example singer, drums, guitar) accompanied by information about, for example their position in the audio scene, so that they can be rendered at the reproduction system to their intended locations. This gives an object-based audio a great flexibility and interactivity because each object is kept discrete and can be individually manipulated.

Each of the above described audio formats has its pros and cons. It is thus common that not only one specific format is used in an audio system, but they might be combined in a complex audio system to create an immersive auditory scene. An example can be a system that combines a scene-based or channel-based audio with an object-based audio, e.g. ambisonics with few discrete audio objects. The <CIT> discloses an audio decoding device. An input signal includes a channel-based audio signal and an object-based audio signal, and an audio encoding device includes an audio scene analysis unit configured to determine an audio scene from the input signal and detect audio scene information; a channel-based encoder that encodes the channel-based audio signal output from the audio scene analysis unit; an object-based encoder that encodes the object-based audio signal output from the audio scene analysis unit; and an audio scene encoding unit configured to encode the audio scene information. The Canadian patent application No. <CIT> discloses a method and device for efficiently distributing a bit-budget in a CELP codec. A method and device allocate a bit-budget to a plurality of first parts of a CELP core module of (a) an encoder for encoding a sound signal or (b) a decoder for decoding the sound signal. In the method and device, bit-budget allocation tables assign, for each of a plurality of intermediate bit rates, respective bit-budgets to the first CELP core module parts. A CELP core module bit rate is determined and one of the intermediate bit rates is selected based on the determined CELP core module bit rate. The respective bit-budgets assigned by the bit-budget allocation tables for the selected intermediate bit rate are allocated to the first CELP core module parts. The <CIT> discloses a post-encoding bitrate reduction of multiple object audio. This prior art document more particularly discloses post-encoding bitrate reduction system and method for generating one more scaled compressed bitstreams from a single encoded plenary file. The plenary file contains multiple audio object files that were encoded separately using a scalable encoding process having fine-grained scalability. Activity in the data frames of the encoded audio object files at a time period are compared with each other to obtain a data frame activity comparison. Bits from an available bitpool are assigned to all of the data frames based on the data frame activity comparison and corresponding hierarchical metadata. The plenary file is scaled down by truncating bits in the data frames to conform to the bit allocation. In some embodiments frame activity is compared to a silence threshold and the data frame contains silence if the frame activity is less than or equal to the threshold and minimal bits are used to represent the silent frame.

The present disclosure presents in the following description a framework to encode and decode object-based audio. Such framework can be a standalone system for object-based audio format coding, or it could form part of a complex immersive codec that may contain coding of other audio formats and/or combination thereof.

According to a first aspect, the present disclosure provides a system for coding an object-based audio signal according to claim <NUM>.

The present disclosure also provides a method for coding an object-based audio signal according to claim <NUM>.

Embodiments related to systems and method for decoding audio objects are not encompassed by the wording of the claims but are considered as useful for understanding the invention.

The foregoing and other objects, advantages and features of the system and method for coding an object-based audio signal and the system and method for decoding an object-based audio signal will become more apparent upon reading of the following non-restrictive description of illustrative embodiments thereof, given by way of example only with reference to the accompanying drawings.

The present disclosure provides an example of mechanism for coding the metadata. The present disclosure also provides a mechanism for flexible intra-object and inter-object bitrate adaptation, i.e. a mechanism that distributes the available bitrate as efficiently as possible. In the present disclosure, it is further considered that the bitrate is fixed (constant). However, it is within the scope of the present disclosure to similarly consider an adaptive bitrate, for example (a) in an adaptive bitrate-based codec or (b) as a result of coding a combination of audio formats coded otherwise at a fixed total bitrate.

There is no description in the present disclosure as to how audio streams are actually coded in a so-called "core-encoder. " In general, the core-encoder for coding one audio stream can be an arbitrary mono codec using adaptive bitrate coding. An example is a codec based on the EVS codec as described in Reference [<NUM>] with a fluctuating bit-budget that is flexibly and efficiently distributed between modules of the core-encoder, for example as described in Reference [<NUM>].

As a non-limitative example, the present disclosure considers a framework that supports simultaneous coding of several audio objects (for example up to <NUM> audio objects) while a fixed constant ISm total bitrate, referred to as ism_total_brate, is considered for coding the audio objects, including the audio streams with their associated metadata. It should be noted that the metadata are not necessarily transmitted for at least some of the audio objects, for example in the case of non-diegetic content. Non-diegetic sounds in movies, TV shows and other videos are sound that the characters cannot hear. Soundtracks are an example of non-diegetic sound, since the audience members are the only ones to hear the music.

In the case of coding a combination of audio formats in the framework, for example an ambisonics audio format with two (<NUM>) audio objects, the constant total codec bitrate, referred to as codec_total_brate, then represents a sum of the ambisonics audio format bitrate (i. the bitrate to encode the ambisonics audio format) and the ISm total bitrate ism_total_brate (i.e. the sum of bitrates to code the audio objects, i.e. the audio streams with the associated metadata).

The present disclosure considers a basic non-limitative example of input metadata consisting of two parameters, namely azimuth and elevation, which are stored per audio frame for each object. In this example, an azimuth range of [-<NUM><NUM>, <NUM><NUM>), and an elevation range of [-<NUM><NUM>, <NUM><NUM>], is considered. However, it is within the scope of the present disclosure to consider only one or more than two (<NUM>) metadata parameters.

<FIG> is a schematic block diagram illustrating concurrently the system <NUM>, comprising several processing blocks, for coding an object-based audio signal and the corresponding method <NUM> for coding the object-based audio signal.

Referring to <FIG>, the method <NUM> for coding the object-based audio signal comprises an operation of input buffering <NUM>. To perform the operation <NUM> of input buffering, the system <NUM> for coding the object-based audio signal comprises an input buffer <NUM>.

The input buffer <NUM> buffers a number N of input audio objects <NUM>, i.e. a number N of audio streams with the associated respective N metadata. The N input audio objects <NUM>, including the N audio streams and the N metadata associated to each of these N audio streams are buffered for one frame, for example a <NUM> long frame. As well known in the art of sound signal processing, the sound signal is sampled at a given sampling frequency and processed by successive blocks of these samples called "frames" each divided into a number of "sub-frames.

Still referring to <FIG>, the method <NUM> for coding the object-based audio signal comprises an operation of analysis and front pre-processing <NUM> of the N audio streams. To perform the operation <NUM>, the system <NUM> for coding the object-based audio signal comprises an audio stream processor <NUM> to analyze and front pre-process, for example in parallel, the buffered N audio streams transmitted from the input buffer <NUM> to the audio stream processor <NUM> through a number N of transport channels <NUM>, respectively.

The analysis and front pre-processing operation <NUM> performed by the audio stream processor <NUM> may comprise, for example, at least one of the following sub-operations: time-domain transient detection, spectral analysis, long-term prediction analysis, pitch tracking and voicing analysis, voice/sound activity detection (VAD/SAD), bandwidth detection, noise estimation and signal classification (which may include in a non-limitative embodiment (a) core-encoder selection between, for example, ACELP core-encoder, TCX core-encoder, HQ core-encoder, etc., (b) signal type classification between, for example, inactive core-encoder type, unvoiced core-encoder type, voiced core-encoder type, generic core-encoder type, transition core-encoder type, and audio core-encoder type, etc., (c) speech/music classification, etc.). Information obtained from the analysis and front pre-processing operation <NUM> is supplied to a configuration and decision processor <NUM> through la line <NUM>. Examples of the foregoing sub-operations are described in Reference [<NUM>] in relation to the EVS codec and, therefore, will not be further described in the present disclosure.

The method <NUM> of <FIG>, for coding the object-based audio signal comprises an operation of metadata analysis, quantization and coding <NUM>. To perform the operation <NUM>, the system <NUM> for coding the object-based audio signal comprises a metadata processor <NUM>.

Signal classification information <NUM> (for example VAD or localVAD flag as used in the EVS codec (See Reference [<NUM>]) from the audio stream processor <NUM> is supplied to the metadata processor <NUM>. The metadata processor <NUM> comprises an analyzer (not shown) of the metadata of each of the N audio objects to determine whether the current frame is inactive (for example VAD = <NUM>) or active (for example VAC ≠ <NUM>) with respect to this particular audio object. In inactive frames, no metadata is coded by the metadata processor <NUM> relative of that object. In active frames, the metadata are quantized and coded for this audio object using a variable bitrate. More details about metadata quantization and coding will be provided in the following Sections <NUM>. <NUM> and <NUM>.

The metadata processor <NUM> of <FIG> quantizes and codes the metadata of the N audio objects, in the described non-restrictive illustrative embodiments, sequentially in a loop while a certain dependency can be employed between quantization of audio objects and the metadata parameters of these audio objects.

As indicated herein above, in the present disclosure, two metadata parameters, azimuth and elevation (as included in the N input metadata), are considered. As a non-limitative example, the metadata processor <NUM> comprises a quantizer (not shown) of the following metadata parameter indexes using the following example resolution to reduce the number of bits being used:.

A total metadata bit-budget for coding the N metadata and a total number quantization bits for quantizing the metadata parameter indexes (i.e. the quantization index granularity and thus the resolution) may be made dependent on the bitrate(s) codec_total_brate, ism_total_brate and/or element_brate (the latter resulting from a sum of a metadata bit-budget and/or a core-encoder bit-budget related to one audio object).

The azimuth and elevation parameters can be represented as one parameter, for example by a point on a sphere. In such a case, it is within the scope of the present disclosure to implement different metadata including two or more parameters.

Both azimuth and elevation indexes, once quantized, can be coded by a metadata encoder (not shown) of the metadata processor <NUM> using either absolute or differential coding. As known, absolute coding means that a current value of a parameter is coded. Differential coding means that a difference between a current value and a previous value of a parameter is coded. As the indexes of the azimuth and elevation parameters usually evolve smoothly (i.e. a change in azimuth or elevation position can be considered as continuous and smooth), differential coding is used by default. However, absolute coding may be used, for example in the following instances:.

The metadata encoder produces a <NUM>-bit absolute coding flag, flagabs, to distinguish between absolute and differential coding.

In the case of absolute coding, the coding flag, flagabs, is set to <NUM>, and is followed by the Baz-bit (or Bel-bit) index coded using absolute coding, where Baz and Bel refer to the above mentioned indexes of the azimuth and elevation parameters to be coded, respectively.

In the case of differential coding, the <NUM>-bit coding flag, flagabs, is set to <NUM> and is followed by a <NUM>-bit zero coding flag, flagzero, signaling a difference Δ between the Baz-bit indexes (respectively the Bel-bit indices) in the current and previous frames equal to <NUM>. If the difference Δ is not equal to <NUM>, the metadata encoder continues coding by producing a <NUM>-bit sign flag, flagsign, followed by a difference index, of which the number of bits is adaptive, in a form of, for example, a unary code indicative of the value of the difference Δ.

<FIG> is a diagram showing different scenarios of bit-stream coding of one metadata parameter.

Referring to <FIG>, it is noted that not all metadata parameters are always transmitted in every frame. Some might be transmitted only in every yth frame, some are not sent at all for example when they do not evolve, they are not important or the available bit-budget is low. Referring to <FIG>, for example:.

The logic used to set absolute or differential coding may be further extended by an intra-object metadata coding logic. Specifically, in order to limit a range of metadata coding bit-budget fluctuation between frames and thus to avoid too low a bit-budget left for the core-encoders <NUM>, the metadata encoder limits absolute coding in a given frame to one, or generally to a number as low as possible of, metadata parameters.

In the non-limitative example of azimuth and elevation metadata parameter coding, the metadata encoder uses a logic that avoids absolute coding of the elevation index in a given frame if the azimuth index was already coded using absolute coding in the same frame. In other words, the azimuth and elevation parameters of one audio object are (practically) never both coded using absolute coding in a same frame. As a consequence, the absolute coding flag, flagabs. ele, for the elevation parameter is not transmitted in the audio object bit-stream if the absolute coding flag, flagabs. azi, for the azimuth parameter is equal to <NUM>.

It is also within the scope of the present disclosure to make the intra-object metadata coding logic bitrate dependent. For example, both the absolute coding flag, flagabs. ele, for the elevation parameter and the absolute coding flag, flagabs. azi, for the azimuth parameter can be transmitted in a same frame is the bitrate is sufficiently large.

The metadata encoder may apply a similar logic to metadata coding of different audio objects. The implemented inter-object metadata coding logic minimizes the number of metadata parameters of different audio objects coded using absolute coding in a current frame. This is achieved by the metadata encoder mainly by controlling frame counters of metadata parameters coded using absolute coding chosen from robustness purposes and represented by the parameter β. As a non-limitative example, a scenario where the metadata parameters of the audio objects evolve slowly and smoothly is considered. In order to control decoding in a noisy channel where indexes are coded using absolute coding every β frames, the azimuth Baz-bit index of audio object #<NUM> is coded using absolute coding in frame M, the elevation Bel-bit index of audio object #<NUM> is coded using absolute coding in frame M+<NUM>, the azimuth Baz-bit index of audio object #<NUM> is encoded using absolute coding in frame M+<NUM>, the elevation Bel-bit index of object #<NUM> is coded using absolute coding in frame M+<NUM>, etc..

<FIG> is a graph showing values of the absolute coding flag, flagabs, for metadata parameters of three (<NUM>) audio objects without using the inter-object metadata coding logic, and <FIG> is a graph showing values of the absolute coding flag, flagabs, for the metadata parameters of the three (<NUM>) audio objects using the inter-object metadata coding logic. In <FIG>, the arrows indicate frames where the value of several absolute coding flags is equal to <NUM>.

More specifically, <FIG> shows the values of the absolute coding flag, flagabs, for two metadata parameters (azimuth and elevation in this particular example) for the audio objects without using the inter-object metadata coding logic, while <FIG> shows the same values but with the inter-object metadata coding logic implemented. The graphs of <FIG> correspond to (from top to bottom):.

It can be seen from <FIG> that several flagabs may have a value equal to <NUM> (see the arrows) in a same frame when the inter-object metadata coding logic is not used. In contrast, <FIG> shows that only one absolute flag, flagabs, may have a value equal to <NUM> in a given frame when the inter-object metadata coding logic is used.

The inter-object metadata coding logic may also be made bitrate dependent. In this case, for example, more that one absolute flag, flagabs, may have a value equal to <NUM> in a given frame even when the inter-object metadata coding logic is used, if the bitrate is sufficiently large.

A technical advantage of the inter-object metadata coding logic and the intra-object metadata coding logic is to limit a range of fluctuation of the metadata coding bit-budget between frames. Another technical advantage is to increase robustness of the codec in a noisy channel; when a frame is lost, then only a limited number of metadata parameters from the audio objects coded using absolute coding is lost. Consequently, any error propagated from a lost frame affects only a small number of metadata parameters across the audio objects and thus does not affect the whole audio scene (or several different channels).

A global technical advantage of analyzing, quantizing and coding the metadata separately from the audio streams is, as described hereinabove, to enable processing specially adapted to the metadata and more efficient in terms of metadata coding bitrate, metadata coding bit-budget fluctuation, robustness in noisy channel, and error propagation due to lost frames.

The quantized and coded metadata <NUM> from the metadata processor <NUM> are supplied to a multiplexer <NUM> for insertion into an output bit-stream <NUM> transmitted to a distant decoder <NUM> (<FIG>).

Once the metadata of the N audio objects are analyzed, quantized and encoded, information <NUM> from the metadata processor <NUM> about the bit-budget for the coding of the metadata per audio object is supplied to a configuration and decision processor <NUM> (bit-budget allocator) described in more detail in the following section <NUM>. When the configuration and bitrate distribution between the audio streams is completed in processor <NUM> (bit-budget allocator), the coding continues with further pre-processing <NUM> to be described later. Finally, the N audio streams are encoded using an encoder comprising, for example, N fluctuating bitrate core-encoders <NUM>, such as mono core-encoders.

The method <NUM> of <FIG>, for coding the object-based audio signal comprises an operation <NUM> of configuration and decision about bitrates per transport channel <NUM>. To perform the operation <NUM>, the system <NUM> for coding the object-based audio signal comprises the configuration and decision processor <NUM> forming a bit-budget allocator.

The configuration and decision processor <NUM> (herein after bit-budget allocator <NUM>) uses a bitrate adaptation algorithm to distribute the available bit-budget for core-encoding the N audio streams in the N transport channels <NUM>.

The bitrate adaptation algorithm of the configuration and decision operation <NUM> comprises the following sub-operations <NUM>-<NUM> performed by the bit-budget allocator <NUM>:.

When the audio streams are all in an inactive segment (or are without meaningful content), the above last two sub-operations <NUM> and <NUM> may be skipped. Accordingly, the bitrate adaptation algorithms described in following sections <NUM>. <NUM> and <NUM>. <NUM> are employed when at least one audio stream has active content.

In inactive frames (VAD = <NUM>), the total bitrate, total_brate, is lowered and the saved bit-budget is redistributed, for example equally between the audio streams in active frames (VAD ≠ <NUM>). The assumption is that waveform coding of an audio stream in frames which are classified as inactive is not required; the audio object may be muted. The logic, used in every frame, can be expressed by the following sub-operations <NUM>-<NUM>:.

<FIG> is a graph illustrating an example of bitrate adaptation for three (<NUM>) core-encoders. Specifically, In <FIG>, the first line shows the core-encoder total bitrate, total_brate, for audio stream #<NUM>, the second line shows the core-encoder total bitrate, total_brate, for audio stream #<NUM>, the third line shows the core-encoder total bitrate, total_brate, for audio stream #<NUM>, line <NUM> is the audio stream #<NUM>, line <NUM> is the audio stream #<NUM>, and line <NUM> is the audio stream #<NUM>.

In the example of <FIG>, the adaptation of the total bitrate, total_brate, for the three (<NUM>) core-encoder is based on VAD activity (active/inactive frames). As can be seen from <FIG>, most of the time there is a small fluctuation of the core-encoder total bitrate, total_brate, as a result of the fluctuating side bit-budget bitsside. Then, there are infrequent substantial changes of the core-encoder total bitrate, total_brate, as a result of the VAD activity.

For example, referring to <FIG>, instance A) corresponds to a frame where the audio stream #<NUM> VAD activity changes from <NUM> (active) to <NUM> (inactive). According to the logic, a minimum core-encoder total bitrate, total_brate, is assigned to audio object #<NUM> while the core-encoder total bitrates, total_brate, for active audio objects #<NUM> and #<NUM> are increased. Instance B) corresponds to a frame where the VAD activity of the audio stream #<NUM> changes from <NUM> (active) to <NUM> (inactive) while the VAD activity of the audio stream #<NUM> remains to <NUM>. Accordingly to the logic, a minimum core-encoder total bitrate, total_brate, is assigned to audio streams #<NUM> and #<NUM> while the core-encoder total bitrate, total_brate, of the active audio stream #<NUM> is further increased.

The above logic of section <NUM>. <NUM> can be made dependent from the total bitrate ism_total_brate. For example, the bit-budget BVAD<NUM> in the above sub-operation <NUM> can be set higher for a higher total bitrate ism_total_brate, and lower for a lower total bitrate ism_total_brate.

The logic described in previous section <NUM>. <NUM> results in about a same core-encoder bitrate in every audio stream with active content (VAD = <NUM>) in a given frame. However, it may be beneficial to introduce an inter-object core-encoder bitrate adaptation based on a classification of ISm importance (or, more generally, on a metric indicative of how critical coding of a particular audio object in a current frame to obtain a given (decent) quality of the decoded synthesis is).

The classification of ISm importance can be based on several parameters and/or combination of parameters, for example core-encoder type (coder_type), FEC (Forward Error Correction), sound signal classification (class), speech/music classification decision, and/or SNR (Signal-to-Noise Ratio) estimate from the open-loop ACELP/TCX (Algebraic Code-Excited Linear Prediction/Transform-Coded eXcitation) core decision module (snr_celp, snr_tcx) as described in Reference [<NUM>]. Other parameters can possibly be used for determining the classification of ISm importance.

In a non-restrictive example, a simple classification of ISm importance is based on the core-encoder type as defined in Reference [<NUM>] is implemented. For that purpose, the bit-budget allocator <NUM> of <FIG> comprises a classifier (not shown) for rating the importance of a particular ISm stream. As a result, four (<NUM>) distinct ISm importance classes, classISm, are defined:.

The ISm importance class is then used by the bit-budget allocator <NUM>, in the bitrate adaptation algorithm (See above Section <NUM>, sub-operation <NUM>) to assign a higher bit-budget to audio streams with a higher ISm importance and a lower bit-budget to audio streams with a lower ISm importance. Thus for every audio stream n, n = <NUM>,. ,N-<NUM>, the following bitrate adaptation algorithm is used by the bit-budget allocator <NUM>:.

<FIG> is a graph illustrating an example of bitrate adaptation based on ISm importance logic. From top to bottom, the graph of <FIG> illustrates, in time:.

In the non-limitative example of <FIG>, with two audio objects (N=<NUM>) and a fixed constant total bitrate, ism_total_brate, equal to <NUM> kbps, the core-encoder total bitrate, total_brate, in active frames of audio object #<NUM> fluctuates between <NUM> kbps and <NUM> kbps when the bitrate adaptation algorithm is not used while it fluctuates between <NUM> kbps and <NUM> kbps when the bitrate adaptation algorithm is used. Similarly, the core-encoder total bitrate, total_brate, in active frames of audio object #<NUM> fluctuates between <NUM> kbps and <NUM> kbps without using the bitrate adaptation algorithm and between <NUM> kbps and <NUM> kbps with the bitrate adaptation algorithm. A better, more efficient distribution of the available bit-budget between the audio streams is thereby obtained.

Referring to <FIG>, the method <NUM> for coding the object-based audio signal comprises an operation of pre-processing <NUM> of the N audio streams conveyed through the N transport channels <NUM> from the configuration and decision processor <NUM> (bit-budget allocator). To perform the operation <NUM>, the system <NUM> for coding the object-based audio signal comprises a pre-processor <NUM>.

Once the configuration and bitrate distribution between the N audio streams is completed by the configuration and decision processor <NUM> (bit-budget allocator), the pre-processor <NUM> performs sequential further pre-processing <NUM> on each of the N audio streams. Such pre-processing <NUM> may comprise, for example, further signal classification, further core-encoder selection (for example selection between ACELP core, TCX core, and HQ core), other resampling at a different internal sampling frequency Fs adapted to the bitrate to be used for core-encoding, etc. Examples of such pre-processing can be found, for example, in Reference [<NUM>] in relation to the EVS codec and, therefore, will not be further described in the present disclosure.

Referring to <FIG>, the method <NUM> for coding the object-based audio signal comprises an operation of core-encoding <NUM>. To perform the operation <NUM>, the system <NUM> for coding the object-based audio signal comprises the above mentioned encoder of the N audio streams including, for example, a number N of core-encoders <NUM> to respectively code the N audio streams conveyed through the N transport channels <NUM> from the pre-processor <NUM>.

Specifically, the N audio streams are encoded using N fluctuating bitrate core-encoders <NUM>, for example mono core-encoders. The bitrate used by each of the N core-encoders is the bitrate selected by the configuration and decision processor <NUM> (bit-budget allocator) for the corresponding audio stream. For example, core-encoders as described in Reference [<NUM>] can be used as core-encoders <NUM>.

Referring to <FIG>, the method <NUM> for coding the object-based audio signal comprises an operation of multiplexing <NUM>. To perform the operation <NUM>, the system <NUM> for coding the object-based audio signal comprises a multiplexer <NUM>.

<FIG> is a schematic diagram illustrating, for a frame, the structure of the bit-stream <NUM> produced by the multiplexer <NUM> and transmitted from the coding system <NUM> of <FIG> to the decoding system <NUM> of <FIG>. Regardless whether metadata are present and transmitted or not, the structure of the bit-stream <NUM> may be structured as illustrated in <FIG>.

Referring to <FIG>, the multiplexer <NUM> writes the indices of the N audio streams from the beginning of the bit-stream <NUM> while the indices of ISm common signaling <NUM> from the configuration and decision processor <NUM> (bit-budget allocator) and metadata <NUM> from the metadata processor <NUM> are written from the end of the bit-stream <NUM>.

The multiplexer writes the ISm common signaling <NUM> from the end of the bit-stream <NUM>. The ISm common signaling is produced by the configuration and decision processor <NUM> (bit-budget allocator) and comprises a variable number of bits representing:.

The multiplexer <NUM> is supplied with the coded metadata <NUM> from the metadata processor <NUM> and writes the metadata payload sequentially from the end of the bit-stream for the audio objects for which the metadata are coded (flagmeta = <NUM>, respectively classISm ≠ ISM_NO_META) in the current frame. The metadata bit-budget for each audio object is not constant but rather inter-object and inter-frame adaptive. Different metadata format scenarios are shown in <FIG>.

In the case that metadata are not present or are not transmitted for at least some of the N audio objects, the metadata flag is set to <NUM>, i.e. flagmeta = <NUM>, respectively classISm = ISM_NO_META, for these audio objects. Then, no metadata indices are sent in relation to those audio objects, i.e. bitsmeta[n] = <NUM>.

The multiplexer <NUM> receives the N audio streams <NUM> coded by the N core encoders <NUM> through the N transport channels <NUM>, and writes the audio streams payload sequentially for the N audio streams in chronological order from the beginning of the bit-stream <NUM> (See <FIG>). The respective bit-budgets of the N audio streams are fluctuating as a result of the bitrate adaptation algorithm described in section <NUM>.

<FIG> is a schematic block diagram illustrating concurrently the system <NUM> for decoding audio objects in response to audio streams with associated metadata and the corresponding method <NUM> for decoding the audio objects.

Referring to <FIG>, the method <NUM> for decoding audio objects in response to audio streams with associated metadata comprises an operation of demultiplexing <NUM>. To perform the operation <NUM>, the system <NUM> for decoding audio objects in response to audio streams with associated metadata comprises a demultiplexer <NUM>.

The demultiplexer receive a bit-stream <NUM> transmitted from the coding system <NUM> of <FIG> to the decoding system <NUM> of <FIG>. Specifically, the bit-stream <NUM> of <FIG> corresponds to the bit-stream <NUM> of <FIG>.

The demultiplexer <NUM> extracts from the bit-stream <NUM> (a) the coded N audio streams <NUM>, (b) the coded metadata <NUM> for the N audio objects, and (c) the ISm common signaling <NUM> read from the end of the received bit-stream <NUM>.

Referring to <FIG>, the method <NUM> for decoding audio objects in response to audio streams with associated metadata comprises an operation <NUM> of metadata decoding and dequantization. To perform the operation <NUM>, the system <NUM> for decoding audio objects in response to audio streams with associated metadata comprises a metadata decoding and dequantization processor <NUM>.

The metadata decoding and dequantization processor <NUM> is supplied with the coded metadata <NUM> for the transmitted audio objects, the ISm common signaling <NUM>, and an output set-up <NUM> to decode and dequantize the metadata for the audio streams/objects with active contents. The output set-up <NUM> is a command line parameter about the number M of decoded audio objects/transport channels and/or audio formats, which can be equal to or different from the number N of coded audio objects/transport channels. The metadata decoding and dequantization processor <NUM> produces decoded metadata <NUM> for the M audio objects/transport channels, and supplies information about the respective bit-budgets for the M decoded metadata on line <NUM>. Obviously, the decoding and dequantization performed by the processor <NUM> is the inverse of the quantization and coding performed by the metadata processor <NUM> of <FIG>.

Referring to <FIG>, the method <NUM> for decoding audio objects in response to audio streams with associated metadata comprises an operation <NUM> of configuration and decision about bitrates per channel. To perform the operation <NUM>, the system <NUM> for decoding audio objects in response to audio streams with associated metadata comprises a configuration and decision processor <NUM> (bit-budget allocator).

The bit-budget allocator <NUM> receives (a) the information about the respective bit-budgets for the M decoded metadata on line <NUM> and (b) the ISm importance class, classISm, from the common signaling <NUM>, and determine the core-decoder bitrates per audio stream, total_brate[n]. The bit-budget allocator <NUM> uses the same procedure as in the bit-budget allocator <NUM> of <FIG> to determine the core-decoder bitrates (see section <NUM>).

Referring to <FIG>, the method <NUM> for decoding audio objects in response to audio streams with associated metadata comprises an operation of core-decoding <NUM>. To perform the operation <NUM>, the system <NUM> for decoding audio objects in response to audio streams with associated metadata comprises a decoder of the N audio streams <NUM> including a number N of core-decoders <NUM>, for example N fluctuating bitrate core-decoders.

The N audio streams <NUM> from the demultiplexer <NUM> are decoded, for example sequentially decoded in the number N of fluctuating bitrate core decoders <NUM> at their respective core-decoder bitrates as determined by the bit-budget allocator <NUM>. When the number of decoded audio objects, M, as requested by the output set-up <NUM> is lower than the number of transport channels, i. e M < N, a lower number of core-decoders are used. Similarly, not all metadata payloads may be decoded in such a case.

In response to the N audio streams <NUM> from the demultiplexer <NUM>, the core-decoder bitrates as determined by the bit-budget allocator <NUM>, and the output set-up <NUM>, the core-decoders <NUM> produces a number M of decoded audio streams <NUM> on respective M transport channels.

In an operation of audio channel rendering <NUM>, a renderer <NUM> of audio objects transforms the M decoded metadata <NUM> and the M decoded audio streams <NUM> into a number of output audio channels <NUM>, taking into consideration an output set-up <NUM> indicative of the number and contents of output audio channels to be produced. Again, the number of output audio channels <NUM> may be equal to or different from the number M.

The renderer <NUM> may be designed in a variety of different structures to obtain the desired output audio channels. For that reason, the renderer will not be further described in the present disclosure.

According to a non-limitative illustrative embodiment, the system and method for coding an object-based audio signal as disclosed in the foregoing description may be implemented by the following source code (expressed in C-code) given herein below as additional disclosure. <IMG>
<IMG>
<IMG>
<IMG>
<IMG>
<IMG>
<IMG>
<IMG>
<IMG>
<IMG>
<IMG>
<IMG>
<IMG>
<IMG>.

<FIG> is a simplified block diagram of an example configuration of hardware components forming the above described coding and decoding systems and methods.

Each of the coding and decoding systems may be implemented as a part of a mobile terminal, as a part of a portable media player, or in any similar device. Each of the coding and decoding systems (identified as <NUM> in <FIG>) comprises an input <NUM>, an output <NUM>, a processor <NUM> and a memory <NUM>.

The input <NUM> is configured to receive the input signal(s), e.g. the N audio objects <NUM> (N audio streams with the corresponding N metadata) of <FIG> or the bit-stream <NUM> of <FIG>, in digital or analog form. The output <NUM> is configured to supply the output signal(s), e.g. the bit-stream <NUM> of <FIG> or the M decoded audio channels <NUM> and the M decoded metadata <NUM> of <FIG>. The input <NUM> and the output <NUM> may be implemented in a common module, for example a serial input/output device.

The processor <NUM> is operatively connected to the input <NUM>, to the output <NUM>, and to the memory <NUM>. The processor <NUM> is realized as one or more processors for executing code instructions in support of the functions of the various processors and other modules of <FIG> and <FIG>.

The memory <NUM> may comprise a non-transient memory for storing code instructions executable by the processor(s) <NUM>, specifically, a processor-readable memory comprising non-transitory instructions that, when executed, cause a processor(s) to implement the operations and processors/modules of the coding and decoding systems and methods as described in the present disclosure. The memory <NUM> may also comprise a random access memory or buffer(s) to store intermediate processing data from the various functions performed by the processor(s) <NUM>.

Those of ordinary skill in the art will realize that the description of the coding and decoding systems and methods are illustrative only and are not intended to be in any way limiting. Other embodiments will readily suggest themselves to such persons with ordinary skill in the art having the benefit of the present disclosure. Furthermore, the disclosed coding and decoding systems and methods may be customized to offer valuable solutions to existing needs and problems of encoding and decoding sound.

In the interest of clarity, not all of the routine features of the implementations of the coding and decoding systems and methods are shown and described. It will, of course, be appreciated that in the development of any such actual implementation of the coding and decoding systems and methods, numerous implementation-specific decisions may need to be made in order to achieve the developer's specific goals, such as compliance with application-, system-, network- and business-related constraints, and that these specific goals will vary from one implementation to another and from one developer to another. Moreover, it will be appreciated that a development effort might be complex and time-consuming, but would nevertheless be a routine undertaking of engineering for those of ordinary skill in the field of sound processing having the benefit of the present disclosure.

In accordance with the present disclosure, the processors/modules, processing operations, and/or data structures described herein may be implemented using various types of operating systems, computing platforms, network devices, computer programs, and/or general purpose machines. In addition, those of ordinary skill in the art will recognize that devices of a less general purpose nature, such as hardwired devices, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), or the like, may also be used. Where a method comprising a series of operations and sub-operations is implemented by a processor, computer or a machine and those operations and sub-operations may be stored as a series of non-transitory code instructions readable by the processor, computer or machine, they may be stored on a tangible and/or non-transient medium.

The coding and decoding systems and methods as described herein may use software, firmware, hardware, or any combination(s) of software, firmware, or hardware suitable for the purposes described herein.

In the coding and decoding systems and methods as described herein, the various operations and sub-operations may be performed in various orders and some of the operations and sub-operations may be optional.

Claim 1:
A system (<NUM>) for coding an object-based audio signal comprising audio objects (<NUM>) in response to audio streams with associated metadata, comprising:
a metadata processor (<NUM>) for coding the metadata, the metadata processor (<NUM>) generating information (<NUM>) about bit-budgets for the coding of the metadata of the audio objects (<NUM>);
an encoder (<NUM>) for coding the audio streams; and
a bit-budget allocator (<NUM>) responsive to the information (<NUM>) about the bit-budgets for the coding of the metadata of the audio objects (<NUM>) from the metadata processor (<NUM>) to allocate bitrates for the coding of the audio streams by the encoder (<NUM>);
characterized in that the metadata processor (<NUM>) codes the metadata prior to and separately from the coding of the audio streams and generates, after the metadata are coded, the information (<NUM>) about the bit-budgets used by the metadata processor (<NUM>) for the coding of the metadata of the audio objects (<NUM>).