Patent Description:
Audio signals may be generally categorized into two types: channel-based audio and object-based audio.

In channel-based audio, the audio signal includes a number of channel signals, and each channel signal corresponds to a loudspeaker. Example channel-based audio signals include stereo audio, <NUM>-channel surround audio, <NUM>-channel surround audio, etc. Stereo audio includes two channels, a left channel for a left loudspeaker and a right channel for a right loudspeaker. <NUM>-channel surround audio includes six channels: a front left channel, a front right channel, a center channel, a left surround channel, a right surround channel, and a low-frequency effects channel. <NUM>-channel surround audio includes eight channels: a front left channel, a front right channel, a center channel, a left surround channel, a right surround channel, a left rear channel, a right rear channel, and a low-frequency effects channel.

In object-based audio, the audio signal includes audio objects, and each audio object includes position information on where the audio of that audio object is to be output. This position information may thus be agnostic with respect to the configuration of the loudspeakers. A rendering system then renders the audio object using the position information to generate the particular signals for the particular configuration of the loudspeakers. Examples of object-based audio include Dolby® Atmos™ audio, DTS:X™ audio, etc..

Both channel-based systems and object-based systems may include renderers that generate the loudspeaker signals from the channel signals or the object signals. Renderers may be categorized into various types, including wave field renderers, beamformers, panners, binaural renderers, etc..

Document <CIT> discloses an audio processing apparatus comprising: a receiver for receiving audio data and render configuration data, the audio data comprising audio data for a plurality of audio components and the render configuration data comprising audio transducer position data for a set of audio transducers; a renderer for generating audio transducer signals for the set of audio transducers from the audio data, the renderer being capable of rendering audio components in accordance with a plurality of rendering modes; a render controller arranged to select rendering modes for the renderer out of the plurality of rendering modes in response the audio transducer position data; and wherein the renderer is arranged to employ different rendering modes for different subsets of the set of audio transducers, and to independently select rendering modes for each of the different subsets of the set of audio transducers, and wherein the render controller is arranged to select the rendering mode for a first audio transducer of the set of audio transducers in response to a position of the first audio transducer relative to a predetermined position for the audio transducer.

Although many existing systems combine multiple renderers, they do not recognize that the selection of renderers may be made based on the desired perceived location of the sound. In many listening environments, the listening experience may be improved by accounting for the desired perceived location of the sound when selecting the renderers. Thus, there is a need for a system that accounts for the desired perceived location of the sound when selecting the renderers, and when assigning the weights to be used between the selected renderers.

Given the above problems and lack of solutions, the embodiments described herein are directed toward using the desired perceived position of an audio object to control two or more renderers, optionally having a single category or different categories.

According to an embodiment, a method of audio processing includes receiving one or more audio objects, wherein each of the one or more audio objects respectively includes position information. The method further includes, for a given audio object of the one or more audio objects, selecting, based on the position information of the given audio object, at least two renderers of a plurality of renderers, for example the at least two renderers having at least two categories; determining, based on the position information of the given audio object, at least two weights; rendering, based on the position information, the given audio object using the at least two renderers weighted according to the at least two weights, to generate a plurality of rendered signals; and combining the plurality of rendered signals to generate a plurality of loudspeaker signals. The method further includes outputting, from a plurality of loudspeakers, the plurality of loudspeaker signals.

The at least two categories may include a sound field renderer, a beamformer, a panner, and a binaural renderer.

A given rendered signal of the plurality of rendered signals may include at least one component signal, wherein each of the at least one component signal is associated with a respective one of the plurality of loudspeakers, and wherein a given loudspeaker signal of the plurality of loudspeaker signals corresponds to combining, for a given loudspeaker of the plurality of loudspeakers, all of the at least one component signal that are associated with the given loudspeaker.

A first renderer may generate a first rendered signal, wherein the first rendered signal includes a first component signal associated with a first loudspeaker and a second component signal associated with a second loudspeaker. A second renderer may generate a second rendered signal, wherein the second rendered signal includes a third component signal associated with the first loudspeaker and a fourth component signal associated with the second loudspeaker. A first loudspeaker signal associated with the first loudspeaker may correspond to combining the first component signal and the third component signal. A second loudspeaker signal associated with the second loudspeaker may correspond to combining the second component signal and the fourth component signal.

Rendering the given audio object may include, for a given renderer of the plurality of renderers, applying a gain based on the position information to generate a given rendered signal of the plurality of rendered signals.

The plurality of loudspeakers may include a dense linear array of loudspeakers.

The at least two categories may include a sound field renderer, wherein the sound field renderer performs a wave field synthesis process.

The plurality of loudspeakers may be arranged in a first group that is directed in a first direction and a second group that is directed in a second direction that differs from the first direction. The first direction may include a forward component and the second direction may include a vertical component. The second direction may include a vertical component, wherein the at least two renderers includes a wave field synthesis renderer and an upward firing panning renderer, and wherein the wave field synthesis renderer and the upward firing panning renderer generate the plurality of rendered signals for the second group. The second direction may include a vertical component, wherein the at least two renderers includes a wave field synthesis renderer, an upward firing panning renderer and a beamformer, and wherein the wave field synthesis renderer, the upward firing panning renderer and the beamformer generate the plurality of rendered signals for the second group. The second direction may include a vertical component, wherein the at least two renderers includes a wave field synthesis renderer, an upward firing panning renderer and a side firing panning renderer, and wherein the wave field synthesis renderer, the upward firing panning renderer and the side firing panning renderer generate the plurality of rendered signals for the second group. The first direction may include a forward component and the second direction may include a side component. The first direction may include a forward component, wherein the at least two renderers includes a wave field synthesis renderer, and wherein the wave field synthesis renderer generates the plurality of rendered signals for the first group. The second direction may include a side component, wherein the at least two renderers includes a wave field synthesis renderer and a beamformer, and wherein the wave field synthesis renderer and the beamformer generate the plurality of rendered signals for the second group. The second direction may include a side component, wherein the at least two renderers includes a wave field synthesis renderer and a side firing panning renderer, and wherein the wave field synthesis renderer and the side firing panning renderer generate the plurality of rendered signals for the second group.

The method may further include combining the plurality of rendered signals for the one or more audio objects to generate the plurality of loudspeaker signals.

The at least two renderers may include renderers in series.

The at least two renderers may include an amplitude panner, a plurality of binaural renderers, and a plurality of beamformers. The amplitude panner may be configured to render, based on the position information, the given audio object to generate a first plurality of signals. The plurality of binaural renderers may be configured to render the first plurality of signals to generate a second plurality of signals. The plurality of beamformers may be configured to render the second plurality of signals to generate a third plurality of signals. The third plurality of signals may be combined to generate the plurality of loudspeaker signals.

According to another embodiment, a non-transitory computer readable medium stores a computer program that, when executed by a processor, controls an apparatus to execute processing including one or more of the method steps discussed herein.

According to another embodiment, an apparatus for processing audio includes a plurality of loudspeakers, a processor, and a memory. The processor is configured to control the apparatus to receive one or more audio objects, wherein each of the one or more audio objects respectively includes position information. For a given audio object of the one or more audio objects, the processor is configured to control the apparatus to select, based on the position information of the given audio object, at least two renderers of a plurality of renderers, wherein the at least two renderers have at least two categories; the processor is configured to control the apparatus to determine, based on the position information of the given audio object, at least two weights; the processor is configured to control the apparatus to render, based on the position information, the given audio object using the at least two renderers weighted according to the at least two weights, to generate a plurality of rendered signals; and the processor is configured to control the apparatus to combine the plurality of rendered signals to generate a plurality of loudspeaker signals. The processor is configured to control the apparatus to output, from the plurality of loudspeakers, the plurality of loudspeaker signals.

The apparatus may include further details similar to those of the methods described herein.

According to another embodiment, a method of audio processing includes receiving one or more audio objects, wherein each of the one or more audio objects respectively includes position information. For a given audio object of the one or more audio objects, the method further includes rendering, based on the position information, the given audio object using a first category of renderer to generate a first plurality of signals; rendering the first plurality of signals using a second category of renderer to generate a second plurality of signals; rendering the second plurality of signals using a third category of renderer to generate a third plurality of signals; and combining the third plurality of signals to generate a plurality of loudspeaker signals. The method further includes outputting, from a plurality of loudspeakers, the plurality of loudspeaker signals.

The first category of renderer may correspond to an amplitude panner, the second category of renderer may correspond to a plurality of binaural renderers, and the third category of renderer may correspond to a plurality of beamformers.

The method may include further details similar to those described regarding the other methods discussed herein.

According to another embodiment, an apparatus for processing audio includes a plurality of loudspeakers, a processor, and a memory. The processor is configured to control the apparatus to receive one or more audio objects, wherein each of the one or more audio objects respectively includes position information. For a given audio object of the one or more audio objects, the processor is configured to control the apparatus to render, based on the position information, the given audio object using a first category of renderer to generate a first plurality of signals; the processor is configured to control the apparatus to render the first plurality of signals using a second category of renderer to generate a second plurality of signals; the processor is configured to control the apparatus to render the second plurality of signals using a third category of renderer to generate a third plurality of signals; and the processor is configured to control the apparatus to combine the third plurality of signals to generate a plurality of loudspeaker signals. The processor is configured to control the apparatus to output, from the plurality of loudspeakers, the plurality of loudspeaker signals.

The following detailed description and accompanying drawings provide a further understanding of the nature and advantages of various implementations.

Described herein are techniques for audio rendering. In the following description, for purposes of explanation, numerous examples and specific details are set forth in order to provide a thorough understanding of the present invention.

In the following description, various methods, processes and procedures are detailed.

In this document, the terms "and", "or" and "and/or" are used. Such terms are to be read as having an inclusive meaning. For example, "A and B" may mean at least the following: "both A and B", "at least both A and B". As another example, "A or B" may mean at least the following: "at least A", "at least B", "both A and B", "at least both A and B". As another example, "A and/or B" may mean at least the following: "A and B", "A or B". When an exclusive-or is intended, such will be specifically noted (e.g., "either A or B", "at most one of A and B").

<FIG> is a block diagram of a rendering system <NUM>. The rendering system <NUM> includes a distribution module <NUM>, a number of renderers <NUM> (three shown: 120a, 120b and 120c), and a routing module <NUM>. The renderers <NUM> are categorized into a number of different categories, which are discussed in more detail below. The rendering system <NUM> receives an audio signal <NUM>, renders the audio signal <NUM>, and generates a number of loudspeaker signals <NUM>. Each of the loudspeaker signals <NUM> drives a loudspeaker (not shown).

The audio signal <NUM> is an object audio signal and includes one or more audio objects. Each of the audio objects includes object metadata <NUM> and object audio data <NUM>. The object metadata <NUM> includes position information for the audio object. The position information corresponds to the desired perceived position for the object audio data <NUM> of the audio object. The object audio data <NUM> corresponds to the audio data that is to be rendered by the rendering system <NUM> and output by the loudspeakers (not shown). The audio signal <NUM> may be in one or more of a variety of formats, including the Dolby® Atmos™ format, the Ambisonics format (e.g., B-format), the DTS:X™ format from Xperi Corp. , etc. For brevity, the following refers to a single audio object in order to describe the operation of the rendering system <NUM>, with the understanding that multiple audio objects may be processed concurrently, for example by instantiating multiple instances of one or more of the renderers <NUM>. For example, an implementation of the Dolby® Atmos™ system may reproduce up to <NUM> simultaneous audio objects in the audio signal <NUM>.

The distribution module <NUM> receives the object metadata <NUM> from the audio signal <NUM>. The distribution module <NUM> also receives loudspeaker configuration information <NUM>. The loudspeaker configuration information <NUM> generally indicates the configuration of the loudspeakers connected to the rendering system <NUM>, such as their numbers, configurations or physical positions. When the loudspeaker positions are fixed (e.g., being components physically attached to a device that includes the rendering system <NUM>), the loudspeaker configuration information <NUM> may be static, and when the loudspeaker positions may be adjusted, the loudspeaker configuration information <NUM> may be dynamic. The dynamic information may be updated as desired, e.g. when the loudspeakers are moved. The loudspeaker configuration information <NUM> may be stored in a memory (not shown).

Based on the object metadata <NUM> and the loudspeaker configuration information <NUM>, the distribution module <NUM> determines selection information <NUM> and position information <NUM>. The selection information <NUM> selects two or more of the renderers <NUM> that are appropriate for rendering the audio object for the given position information in the object metadata <NUM>, given the arrangement of the loudspeakers according to the loudspeaker configuration information <NUM>. The position information <NUM> corresponds to the source position to be rendered by each of the selected renderers <NUM>. In general, the position information <NUM> may be considered to be a weighting function that weights the object audio data <NUM> among the selected renderers <NUM>.

The renderers <NUM> receive the object audio data <NUM>, the loudspeaker configuration information <NUM>, the selection information <NUM> and the position information <NUM>. The renderers <NUM> use the loudspeaker configuration information <NUM> to configure their outputs. The selection information <NUM> selects two or more of the renderers <NUM> to render the object audio data <NUM>. Based on the position information <NUM>, each of the selected renderers <NUM> renders the object audio data <NUM> to generate rendered signals <NUM>. , the renderer 120a generates the rendered signals 166a, the renderer 120b generates the rendered signals 166b, etc.). Each of the rendered signals <NUM> from each of the renderers <NUM> corresponds to a driver signal for one of the loudspeakers (not shown), as configured according to the loudspeaker configuration information <NUM>. For example, if the rendering system <NUM> is connected to <NUM> loudspeakers, the renderer 120a generates up to <NUM> rendered signals 166a. (If a given audio object is rendered such that it is not to be output from a particular loudspeaker, then that one of the rendered signals <NUM> may be considered to be zero or not present, as indicated by the loudspeaker configuration information <NUM>.

The routing module <NUM> receives the rendered signals <NUM> from each of the renderers <NUM> and the loudspeaker configuration information <NUM>. Based on the loudspeaker configuration information <NUM>, the routing module <NUM> combines the rendered signals <NUM> to generate the loudspeaker signals <NUM>. To generate each of the loudspeaker signals <NUM>, the routing module <NUM> combines, for each loudspeaker, each one of the rendered signals <NUM> that correspond to that loudspeaker. For example, a given loudspeaker may be related to one of the rendered signals 166a, one of the rendered signals 166b, and one of the rendered signals 166c; the routing module <NUM> combines these three signals to generate the corresponding one of the loudspeaker signals <NUM> for that given loudspeaker. In this manner, the routing module <NUM> performs a mixing function of the appropriate rendered signals <NUM> to generate the respective loudspeaker signals <NUM>.

Due to the linearity of acoustics, the principle of superposition allows the rendering system <NUM> to use any given loudspeaker concurrently for any number of the renderers <NUM>. The routing module <NUM> implements this by summing, for each loudspeaker, the contribution from each of the renderers <NUM>. As long as the sum of those signals does not overload the loudspeaker, the result corresponds to a situation where independent loudspeakers are allocated to each renderer, in terms of impression for the listener.

When multiple audio objects are rendered to be output concurrently, the routing module <NUM> combines the rendered signals <NUM> in a manner similar to the single audio object case discussed above.

<FIG> is a flowchart of a method <NUM> of audio processing. The method <NUM> may be performed by the rendering system <NUM> (see <FIG>). The method <NUM> may be implemented by one or more computer programs, for example that the rendering system <NUM> executes to control its operation.

At <NUM>, one or more audio objects are received. Each of the audio objects respectively includes position information. (For example, two audio objects A and B may have respective position information PA and PB. ) As an example, the rendering system <NUM> (see <FIG>) may receive one or more audio objects in the audio signal <NUM>. For each of the audio objects, the method continues with <NUM>.

At <NUM>, for a given audio object, at least two renderers are selected based on the position information of the given audio object. The at least two renderers have at least two categories. (Of course, a particular audio object may be rendered using a single category of renderer; such a situation operates similarly to the multiple category situation discussed herein. ) For example, when the position information indicates that a particular two renderers (having a particular two categories) would be appropriate for rendering that audio object, then those two renderers are selected. The renderers may be selected based on the loudspeaker configuration information <NUM> (see <FIG>). As an example, the distribution module <NUM> may generate the selection information <NUM> to select at least two of the renderers <NUM>, based on the position information in the object metadata <NUM> and the loudspeaker configuration information <NUM>.

At <NUM>, for the given audio object, at least two weights are determined based on the position information. The weights are related to the renderers selected at <NUM>. As an example, the distribution module <NUM> (see <FIG>) may generate the position information <NUM> (corresponding to the weights) based on the position information in the object metadata <NUM> and the loudspeaker configuration information <NUM>.

At <NUM>, the given audio object is rendered, based on the position information, using the selected renderers (see <NUM>) weighted according to the weights (see <NUM>), to generate a plurality of rendered signals. As an example, the renderers <NUM> (see <FIG>, selected according to the selection information <NUM>) generate the rendered signals <NUM> from the object audio data <NUM>, weighted according to the position information <NUM>. Continuing the example, when the renderers 120a and 120b are selected, the rendered signals 166a and 166b are generated.

At <NUM>, the plurality of rendered signals (see <NUM>) are combined to generate a plurality of loudspeaker signals. For a given loudspeaker, the corresponding rendered signals <NUM> are summed to generate the loudspeaker signal. The loudspeaker signals may be attenuated when above a maximum signal level, in order to prevent overloading a given loudspeaker. As an example, the routing module <NUM> may combine the rendered signals <NUM> to generate the loudspeaker signals <NUM>.

At <NUM>, the plurality of loudspeaker signals (see <NUM>) are output from a plurality of loudspeakers.

When multiple audio objects are to be output concurrently, the method <NUM> operates similarly. For example, multiple given audio objects may be processed using multiple paths of <NUM>-<NUM>-<NUM> in parallel, with the rendered signals corresponding to the multiple audio objects being combined (see <NUM>) to generate the loudspeaker signals.

<FIG> is a block diagram of a rendering system <NUM>. The rendering system <NUM> may be used to implement the rendering system <NUM> (see <FIG>) or to perform one or more of the steps of the method <NUM> (see <FIG>). The rendering system <NUM> may store and execute one or more computer programs to implement the rendering system <NUM> or to perform the method <NUM>. The rendering system <NUM> includes a memory <NUM>, a processor <NUM>, an input interface <NUM>, and an output interface <NUM>, connected by a bus <NUM>. The rendering system <NUM> may include other components that (for brevity) are not shown.

The memory <NUM> generally stores data used by the rendering system <NUM>. The memory <NUM> may also store one or more computer programs that control the operation of the rendering system <NUM>. The memory <NUM> may include volatile components (e.g., random access memory) and non-volatile components (e.g., solid state memory). The memory <NUM> may store the loudspeaker configuration information <NUM> (see <FIG>) or the data corresponding to the other signals in <FIG>, such as the object metadata <NUM>, the object audio data <NUM>, the rendered signals <NUM>, etc..

The processor <NUM> generally controls the operation of the rendering system <NUM>. When the rendering system <NUM> implements the rendering system <NUM> (see <FIG>), the processor <NUM> implements the functionality corresponding to the distribution module <NUM>, the renderers <NUM>, and the routing module <NUM>.

The input interface <NUM> receives the audio signal <NUM>, and the output interface <NUM> outputs the loudspeaker signals <NUM>.

<FIG> is a block diagram of a loudspeaker system <NUM>. The loudspeaker system <NUM> includes a rendering system <NUM> and a number of loudspeakers <NUM> (six shown, 404a, 404b, 404c, 404d, 404e and 404f). The loudspeaker system <NUM> may be configured as a single device that includes all of the components (e.g., a soundbar form factor). The loudspeaker system <NUM> may be configured as separate devices (e.g., the rendering system <NUM> is one component, and the loudspeakers <NUM> are one or more other components).

The rendering system <NUM> may correspond to the rendering system <NUM> (see <FIG>), receiving the audio signal <NUM>, and generating loudspeaker signals <NUM> that correspond to the loudspeaker signals <NUM> (see <FIG>). The components of the rendering system <NUM> may be similar to those of the rendering system <NUM> (see <FIG>).

The loudspeakers <NUM> output auditory signals (not shown) corresponding to the loudspeaker signals <NUM> (six shown, 406a, 406b, 406c, 406d, 406e and 406f). The loudspeaker signals <NUM> may correspond to the loudspeaker signals <NUM> (see <FIG>). The loudspeakers <NUM> may output the loudspeaker signals as discussed above regarding <NUM> in <FIG>.

As mentioned above, the renderers (e.g., the renderers <NUM> of <FIG>) are classified into various categories. Four general categories of renderers include sound field renderers, binaural renderers, panning renderers, and beamforming renderers. As discussed above (see <NUM> in <FIG>), for a given audio object, the selected renderers have at least two categories. For example, based on the object metadata <NUM> and the loudspeaker configuration information <NUM> (see <FIG>), the distribution module <NUM> may select a sound field renderer and a beamforming renderer (of the renderers <NUM>) to render a given audio object.

Additional details of the four general categories of renderers are provided below. Note that where a category includes sub-categories of renderers, it is to be understood that the references to different categories of renderers are similar applicable to different sub-categories of renderers. The rendering systems described herein (e.g., the rendering system <NUM> of <FIG>) may implement one or more of these categories of renderers.

In general, sound field rendering aims to reproduce a specific acoustic pressure (sound) field in a given volume of space. Sub-categories of sound field renderers include wave field synthesis, near-field compensated high-order Ambisonics, and spectral division.

One important capability of sound field rendering methods is the ability to project virtual sources in the near field, meaning generate sources that the listener will be localized at a position between himself and the speakers. While such effect is possible also for binaural renderers (see below), the particularity here is that the correct localization impression can be generated over a wide listening area.

Binaural rendering methods focus on delivering to the listener's ears a signal carrying along the source signal processed to mimic the binaural cues associated with the source location. While the simpler way to deliver such signals is commonly over headphones, it can be successfully done over a speaker system as well, through the use of crosstalk cancellers in order to deliver individual left and right ear feeds to the listener.

Panning methods make direct use of the basic auditory mechanisms (e.g., changing interaural loudness and temporal differences) to move sound images around through delay and/or gain differentials applied to the source signal before being fed to multiple speakers. Amplitude panners, which use only gain differentials, are popular due to their simple implementation and stable perceptual impressions. They have been deployed in many consumer audio systems such as stereo systems and traditional cinema content rendering. (An example of a suitable amplitude panner design for arbitrary speaker arrays is provided by <NPL>. ) Finally, methods that use reflections from the reproduction environment generally rely on similar principles to manipulate the spatial impression from the system.

Beamforming was originally designed for sensor arrays (e.g., microphone arrays), as a means to amplify the signal coming from a set of preferred directions. Thanks to the principle of reciprocity in acoustics, the same principle can be used to create directional acoustic signals. <CIT> describes the use of beamforming to create virtual speakers through the use of focused sources.

The rendering system categories discussed above have a number of considerations regarding the sweet spot and the source location to be rendered.

The sweet spot generally corresponds to the space where the rendering is considered acceptable according to a listener perception metric. While the exact extent of such area is generally imperfectly defined due to the absence of analytic metrics capturing well the perceptual quality of the rendering, it is generally possible to derive qualitative information from typical error metrics (e.g., square error), and compare different systems in different configurations. For example, a common observation is that the sweet spot is smaller (for all categories of renderers) at higher frequencies. Generally, it can also be observed that the sweet spot grows with the number of speakers available in the system, except for panning methods, for which the addition of speakers has different advantages.

The different rendering system categories may also vary in the way and capabilities they have to deliver audio to be perceived at various source locations. Sound field rendering methods generally allow for the creation of virtual sources anywhere in the direction of the speaker array from the point of view of the listener. One aspect of those methods is that they allow for the manipulation of the perceived distance of the source in a transparent way and from the perspective of the entire listening area. Binaural rendering methods can theoretically deliver any source locations in the sweet spot, as long as the binaural information related to those positions has been previously stored. Finally, the panning methods can deliver any source direction for which a pair/trio of speakers sufficiently close (e.g., approximately <NUM> degree angle such as between <NUM>-<NUM> degrees) is available from the point of view of the listener. (However, panning methods generally do not define specific ways to handle source distance, so additional strategies need to be used if a distance component is desired.

In addition, some rendering system categories exhibit an interdependence between the source location and the sweet spot. For example, for a linear array of loudspeakers implementing a wave field synthesis process (in the sound field rendering category), a source location in the center behind the array may be perceived in a large sweet spot in front of the array, whereas a source location in front of the array and displaced to the side may be perceived in a smaller, off-center sweet spot.

Given the above considerations, embodiments are directed toward using two or more rendering methods in combination, where the relative weight between the selected rendering methods depends on the audio object location.

With the increasing availability of hardware allowing for the use of large number of speakers in consumer applications, the possibility of using complex rendering strategies becomes more and more appealing. Indeed, the number of speakers still remains limited so that using a single rendering method generally leads to strong limitations, generally with regard to the sweet spot extent. Additionally, complex strategies can potentially deal with complex speaker setups, for example some missing surround coverage in some region, or just lacking speaker density. However, the standard limitations of those reproduction methods remain, leading to the necessary compromise between coverage (the largest array possible to have a wider range of possible source locations) and density (the densest array possible to avoid as much as possible high frequency distortion due to aliasing) for a given number of channels.

In view of the above issues, embodiments are directed to using multiple types of renderers driven together to render object-based audio content. For example, in the rendering system <NUM> (see <FIG>), the distribution module <NUM> processes the object-based audio content based on the object metadata <NUM> and the loudspeaker configuration information <NUM> in order to determine (<NUM>) which of the renderers <NUM> to activate (the selection information <NUM>), and (<NUM>) the source position to be rendered by each activated renderer (the position information <NUM>). Each selected renderer then renders the object audio data <NUM> according to the position information <NUM> and generates the rendered signals <NUM> that the routing module <NUM> routes to the appropriate loudspeaker in the system. The routing module <NUM> allows the use of a given loudspeaker by multiple renderers. In this manner, the rendering system <NUM> uses the distribution module <NUM> to distribute each audio object to the renderers <NUM> that will effectively convey the intended spatial impression in the desired listening area.

For a system at K speakers (k = <NUM>. K), rendering O objects (o = <NUM>. O) with R distinct renderers (r = <NUM>. R), the output s of each speaker k is given by: <MAT>.

The type of renderer for renderer r is reflected in the driving function <MAT>. The specific behavior of a given renderer is determined by its type and the available setup of speakers it is driving (as determined by δk∈r). The distribution of a given object among the renderers is controlled by the distribution algorithm, through the activation coefficient wr and the mapping <MAT> of a given object o in the space controlled by renderer r.

Applying the above equation to the rendering system <NUM> (see <FIG>), each sk corresponds to one of the loudspeaker signals <NUM>, so corresponds to the object audio data <NUM> for a given audio object, wr corresponds to the selection information <NUM>, δk∈r corresponds to the loudspeaker configuration information <NUM> (e.g., configuring the routings performed by the routing module <NUM>), <MAT> corresponds to a rendering function for each of the renderers <NUM>, and xo and <MAT> correspond to the position information <NUM>. The combination of wr and <MAT> may be considered to be weights that provide the relative weight between the selected renderers for the given audio object.

Although the above equation is written in the time domain, an example implementation may operate in the frequency domain, for example using a filter bank. Such an implementation may transform the object audio data <NUM> to the frequency domain, perform the operations of the above equation in the frequency domain (e.g., the convolutions become multiplications, etc.), and then inverse transform the results to generate the rendered signals <NUM> or the loudspeaker signals <NUM>.

<FIG> are respectively a top view and a side view of a soundbar <NUM>. The soundbar <NUM> may implement the rendering system <NUM> (see <FIG>). The soundbar <NUM> includes a number of loudspeakers including a linear array <NUM> (having <NUM> loudspeakers 502a, 502b, 502c, 502d, 502e, 502f, <NUM>, <NUM>, 502i, 502j, <NUM> and <NUM>) and an upward firing group <NUM> (including <NUM> loudspeakers 504a and 504b). The loudspeaker 502a may be referred to as the far left loudspeaker, the loudspeaker <NUM> may be referred to as the far right loudspeaker, the loudspeaker 504a may be referred to as the upward left loudspeaker, and the loudspeaker 504b may be referred to as the upward right loudspeaker. The number of loudspeakers and their arrangement may be adjusted as desired.

The soundbar <NUM> is suitable for consumer use, for example in a home theater configuration, and may receive its input from a connected television or audio/video receiver. The soundbar <NUM> may be placed above or below the television screen, for example.

<FIG>, <FIG> are respectively a first top view, a second top view and a side view showing the output coverage for the soundbar <NUM> (see <FIG>) in a room. <FIG> shows a near field output <NUM> generated by the linear array <NUM>. The near field output <NUM> is generally projected outward from the front of the linear array <NUM>. <FIG> shows a virtual side outputs 604a and 604b generated by the linear array <NUM> using beamforming. The virtual side outputs 604a and 604b result from beamforming against the walls. <FIG> shows a virtual top output <NUM> generated by the upward firing group <NUM>. (Also shown is the near field output <NUM> of <FIG>, generally in the plane of the listener. ) The virtual top output <NUM> results from reflecting against the ceiling. For a given audio object, the soundbar <NUM> may combine two or more of these outputs together, e.g. using a routing module such as the routing module <NUM> (see <FIG>), in order to conform the audio object's perceived position with its position metadata.

<FIG> is a block diagram of a rendering system <NUM>. The rendering system <NUM> is a specific embodiment of the rendering system <NUM> (see <FIG>) suitable for the soundbar <NUM> (see <FIG>). The rendering system <NUM> may be implemented using the components of the rendering system <NUM> (see <FIG>). As with the rendering system <NUM>, the rendering system <NUM> receives the audio signal <NUM>. The rendering system <NUM> includes a distribution module <NUM>, four renderers 720a, 720b, 720c and 720d (collectively the renderers <NUM>), and a routing module <NUM>.

The distribution module <NUM>, in a manner similar to the distribution module <NUM> (see <FIG>), receives the object metadata <NUM> and the loudspeaker configuration information <NUM>, and generates the selection information <NUM> and the position information <NUM>.

The renderers <NUM> receive the object audio data <NUM>, the loudspeaker configuration information <NUM>, the selection information <NUM> and the position information <NUM>, and generate rendered signals 766a, 766b, 766c and 766d (collectively the rendered signals <NUM>). The renderers <NUM> otherwise function similarly to the renderers <NUM> (see <FIG>). The renderers <NUM> include a wave field renderer 720a, a left beamformer 720b, a right beamformer 720c, and a vertical panner 720d. The wave field renderer 720a generates the rendered signals 766a corresponding to the near field output <NUM> (see <FIG>). The left beamformer 720b generates the rendered signals 766b corresponding to the virtual side output 604a (see <FIG>). The right beamformer 720c generates the rendered signals 766c corresponding to the virtual side output 604b (see <FIG>). The vertical panner 720d generates the rendered signals 766d corresponding to the virtual top output <NUM> (see <FIG>).

The routing module <NUM> receives the loudspeaker configuration information <NUM> and the rendered signals <NUM>, and combines the rendered signals <NUM> in a manner similar to the routing module <NUM> (see <FIG>) to generate loudspeaker signals 770a and 770b (collectively the loudspeaker signals <NUM>). The routing module <NUM> combines the rendered signals 766a, 766b and 766c to generate the loudspeaker signals 770a that are provided to the loudspeakers of the linear array <NUM> (see <FIG>). The routing module <NUM> routes the rendered signals 766d to the loudspeakers of the upward firing group <NUM> (see <FIG>) as the loudspeaker signals 770b.

As an audio object's perceived position changes across the listening environment, the distribution module <NUM> performs cross-fading (using the position information <NUM>) among the various renderers <NUM> to result in smooth perceived source motion between the different regions of <FIG>, <FIG>.

<FIG> are respectively a top view and a side view showing an example of the source distribution for the soundbar <NUM> (see <FIG>). For a particular audio object in the audio signal <NUM> (see <FIG>), the object metadata <NUM> defines a desired perceived position within a virtual cube of size <NUM>×<NUM>×<NUM>. This virtual cube is mapped to a cube in the listening environment, e.g. by the distribution module <NUM> (see <FIG>) or the distribution module <NUM> (see <FIG>) using the position information <NUM>.

<FIG> shows the horizontal plane (x,y), with the point <NUM> at (<NUM>,<NUM>), point <NUM> at (<NUM>,<NUM>), point <NUM> at (<NUM>,-<NUM>), and point <NUM> at (<NUM>,-<NUM>). (These points are marked with the "X". ) The perceived position of the audio object is then mapped from the virtual cube to the rectangular area <NUM> defined by these four points. Note that this plane is only half the virtual cube in this dimension, and that sources where y><NUM> (e.g., behind the listener positions <NUM>) are placed on the line between the points <NUM> and <NUM>, in front of the listener positions <NUM>. The points <NUM> and <NUM> may be considered to be at the front wall of the listening environment. The width of the area <NUM> (e.g., between points <NUM> and <NUM>) is roughly aligned with (or slightly inside of) the sides of the linear array <NUM> (see also <FIG>).

<FIG> shows the vertical plane (x,z), with the point <NUM> at (<NUM>,<NUM>), point <NUM> at (-<NUM>,<NUM>), point <NUM> at (<NUM>,<NUM>), and point <NUM> at (-<NUM>,<NUM>). The perceived position of the audio object is then mapped from the virtual cube to the rectangular area <NUM> defined by these four points. As with <FIG>, in <FIG> sources where y><NUM> (e.g., behind the listener positions <NUM>) are placed on the line between the points <NUM> and <NUM>. The points <NUM> and <NUM> may be considered to be at the ceiling of the listening environment. The bottom of the area <NUM> is aligned at the level of the linear array <NUM>.

In <FIG>, note the trapezoid <NUM> in the horizontal plane, with its wide base aligned with one side of the area <NUM> between points <NUM> and <NUM>, and its narrow base aligned in front of the listener positions <NUM> (on the line between points <NUM> and <NUM>). The system distinguishes sources with desired perceived positions inside the trapezoid <NUM> from those outside the trapezoid <NUM> (but still within the area <NUM>). Within the trapezoid <NUM>, the source is reproduced without using the beamformers (e.g., 720b and 720c in <FIG>); instead, the sound field renderer (e.g., 720a in <FIG>) is used to reproduce the source. Outside the trapezoid <NUM>, the source may be reproduced using both the beamformers (e.g., 720b and 720c) and the sound field renderer (e.g., 720a) in the horizontal plane. In particular, the sound field renderer 720a places a source at the same coordinate y, at the very left of the trapezoid <NUM>, if the source is located on the left (or the very right if the source is located on the right), while the two beamformers 720b and 720c create a stereo phantom source between each other through panning. The left-right panning factor between the two beamformers 720b and 720c may follow a constant energy amplitude panning rule mapping x = <NUM> to the left beamformer 720b only and x = <NUM> to the right beamformer 720c only. (The distribution module <NUM> may use the position information <NUM> to implement this amplitude panning rule, e.g., using the weights. ) The system applies a constant-energy cross-fading rule between the sound field renderer 720a and the pair of beamformers 720b-720c, so that the sound energy from the beamformers 720b-720c increases while the sound energy from the sound field renderer 720a decreases as the source is placed further from the trapezoid <NUM>. (The distribution module <NUM> may use the position information <NUM> to implement this cross-fading rule.

In the z dimension (see <FIG>), the system applies a constant-energy cross-fade rule between the signal fed to the combination of the beamformers 720b-720c and the sound field renderer 720a, and the rendered signals 766d rendered by the vertical panner 720d that are fed to the upward firing group <NUM> (see <FIG>). The cross-fade factor is proportional to the z coordinate, with z = <NUM> corresponding to all of the signal being rendered through the beamformers 720b-720c and the sound field renderer 720a, and z = <NUM> corresponding to all of the signal being rendered using the vertical panner 720d. The rendered signal 766d produced by the vertical panner 720d is distributed between the two channels (to the two loudspeakers 504a and 504b) using a constant-energy amplitude panning rule, mapping x = <NUM> to the left loudspeaker 504a only and x = <NUM> to the right loudspeaker 504b only. (The distribution module <NUM> may use the position information <NUM> to implement this amplitude panning rule.

<FIG> are top views showing a mapping of object-based audio (<FIG>) to a loudspeaker array (<FIG> shows a horizontal square region <NUM> defined by point <NUM> at (<NUM>,<NUM>), point <NUM> at (<NUM>,<NUM>), point <NUM> at (<NUM>,<NUM>), and point <NUM> at (<NUM>,<NUM>). Point <NUM> is at (<NUM>,<NUM>), at the midpoint between points <NUM> and <NUM>, and point <NUM> is at (<NUM>,<NUM>), at the midpoint between points <NUM> and <NUM>. Point <NUM> is at (<NUM>,<NUM>), the center of the square region <NUM>. Points <NUM>, <NUM>, <NUM> and <NUM> define a trapezoid <NUM>. Adjacent to the sides of the trapezoid <NUM> are two zones <NUM> and <NUM>, which have a width of <NUM> units in the specified x direction. Adjacent to the sides of the zones <NUM> and <NUM> are the triangles <NUM> and <NUM>. An audio object may have a desired perceived position within the square region <NUM> according to its metadata (e.g., the object metadata <NUM> of <FIG>). An example object audio system that uses the horizontal square <NUM> is the Dolby Atmos® system.

<FIG> shows the mapping of a portion of the square region <NUM> (see <FIG>) to a region <NUM> defined by points <NUM>, <NUM>, <NUM> and <NUM>. Note that only half of the square region <NUM> (defined by the points <NUM>, <NUM>, <NUM> and <NUM>) is mapped to the region <NUM>; the perceived positions in the other half of the square region <NUM> are mapped on the line between points <NUM> and <NUM>. (This is similar to what was described above in <FIG>. ) A loudspeaker array <NUM> is within the region <NUM>; the width of the loudspeaker array <NUM> corresponds to the width L of the region <NUM>. Similarly to the square region <NUM> (see <FIG>), the region <NUM> includes a trapezoid <NUM>, two zones <NUM> and <NUM> adjacent to the sides of the trapezoid <NUM>, and two triangles <NUM> and <NUM>. The zones <NUM> and <NUM> correspond to the zones <NUM> and <NUM> (see <FIG>), and the triangles <NUM> and <NUM> correspond to the triangles <NUM> and <NUM> (see <FIG>). A wide base of the trapezoid <NUM> corresponds to the width L of the region <NUM>, and a narrow base corresponds to a width l. The height of the trapezoid <NUM> is (H - h), where H corresponds to a large triangle that includes the trapezoid <NUM> and extends from the wide base (having width L) to a point <NUM>, and h corresponds to the height of a small triangle that extends from the narrow base (having width l) to the point <NUM>. As will be detailed more below, within the zones <NUM> and <NUM>, the system implements a constant-energy cross-fading rule between the categories of renderers.

More precisely, the output of the loudspeaker array <NUM> (see <FIG>) may be described as follows. The loudspeaker array <NUM> has M speakers (m = <NUM>,. , M from left to right). Those speakers are driven as follows: <MAT>.

The factor θNF/B(xo,yo) drives the balance between the near-field wave field synthesis renderer 720a and the beamformers 720b-720c (see <FIG>). It is defined using the notation presented in <FIG> for the trapezoid <NUM>, so that for <MAT>: <MAT>.

The positioning of the sources in the near-field, using the wave field renderer 720a, follows the rule: <MAT>.

The driving functions are written in the frequency domain. For sources behind the array plane (e.g., behind the loudspeaker array <NUM> such as on the line between points <NUM> and <NUM>): <MAT> with <MAT> and c speed of sound.

And in front of the array plane (e.g., in front of the loudspeaker array <NUM>), note that only the last term changes: <MAT> with <MAT>.

In these expressions, the last term corresponds to the amplitude and delay control values in the <NUM>. 5D Wave Field Synthesis theory for a localized sources in front and behind the array plane (e.g., defined by the loudspeaker array <NUM>). (An overview of Wave Field Synthesis theory is provided by <NPL>. ) The other coefficients are defined as follows:.

Regarding the beamformers 720b-720c, the system pre-computes a set of M/<NUM> speaker delays and amplitudes adapted to the configuration of the left half of the linear loudspeaker array <NUM>. In the frequency domain, it gives us filter coefficients Bm(ω) for each speaker m and frequency ω. The beamformer driving function for the left half of the speaker array (m = <NUM>. M/<NUM>) is then a filter defined in the frequency domain as: <MAT>.

In the above equation, EQm is the equalization filter compensating for speaker response distortion (same filter as in Equations (<NUM>) and (<NUM>)). The system is designed for a symmetric setup, so that we can just flip the beam filters for the right half of the array to obtain the other beam, so that for m = M/<NUM>. M, we have: <MAT>.

The rendered signals 766d (see <FIG>), which correspond to the loudspeaker signals 770b provided to the two upward firing speakers 504a-504b (see <FIG>), correspond to the signals sUL and sUR as follows: <MAT>.

According to an embodiment, the vertical panner 720d (see <FIG>) includes a pre-filtering stage. The pre-filtering stage applies a height perceptual filter H proportionally to the height coordinate z<NUM>. In such a case, the applied filter for a given z<NUM> is <MAT>.

<FIG> is a block diagram of a rendering system <NUM>. The rendering system <NUM> is a modification of the rendering system <NUM> (see <FIG>) suitable for implementation in the soundbar <NUM> (see <FIG>). The rendering system <NUM> may be implemented using the components of the rendering system <NUM> (see <FIG>). The components of the rendering system <NUM> are similar to those of the rendering system <NUM> and use similar reference numbers. The rendering system <NUM> also includes a second pair of beamformers 1120e and 1120f. The left beamformer 1120e generates rendered signals 1166d, and the right beamformer 1120f generates rendered signals 1166e, which the routing module <NUM> combines with the other rendered signals 766a, 766b and 766c to generate the loudspeaker signals 770a. When their output is considered on its own, the left beamformer 1120e creates a virtual left rear source, and the right beamformer 1120f creates a virtual right rear source, as shown in <FIG>.

<FIG> is a top view of showing the output coverage for the beamformers 1120e and 1120f, implemented in the soundbar <NUM> (see <FIG>) in a room. (The output coverage for the other renderers of the rendering system <NUM> is as shown in <FIG>. ) The virtual left rear output 1206a results from the left beamformer 1120e (see <FIG>) generating signals that are reflected from the left wall and back wall of the room. The virtual right rear output 1206b results from the right beamformer 1120f (see <FIG>) generating signals that are reflected from the right wall and back wall of the room. (Note the triangular area where 1206a and 1206b overlap behind the listeners. ) For a given audio object, the soundbar <NUM> may combine the output coverage of <FIG> with one or more of the output coverage of <FIG>, e.g. using a routing module such as the routing module <NUM> (see <FIG>).

The output coverages of <FIG> and <FIG> show how the soundbar <NUM> (see <FIG>) may be used in place of the loudspeakers in a traditional <NUM>-channel (or <NUM>. <NUM>-channel) surround sound system. The left, center and right loudspeakers of the <NUM>-channel system may be replaced by the linear array <NUM> driven by the sound field renderer 720a (see <FIG>), resulting in the output coverage shown in <FIG>. The top loudspeakers of the <NUM>. <NUM>-channel system may be replaced by the upward firing group <NUM> driven by the vertical panner 720d, resulting in the output coverage shown in <FIG>. The left and right surround loudspeakers of the <NUM>-channel system may be replaced by the linear array <NUM> driven by the beamformers 720b and 720c, resulting in the output coverage shown in <FIG>. The left and right rear surround loudspeakers of the <NUM>-channel system may be replaced by the linear array <NUM> driven by the beamformers 1120e and 1120f (see <FIG>), resulting in the output coverage shown in <FIG>. As discussed above, the system enables multiple renderers to render an audio object, according to their combined output coverages, in order to generate an appropriate perceived position for the audio object.

In summary, the systems described herein have an advantage of having the rendering system with the most resolution (e.g., the near field renderer) at the front where most of the cinematographic content is expected to be located (as it matches the screen location) and where human localization accuracy is maximal, while rear, lateral and height rendering remains coarser, which may be less critical for typical cinematographic content. Many of these systems also remain relatively compact and can sensibly be integrated alongside typical visual devices (e.g., above or below the television screen). One feature to keep in mind is that the speaker array can be used to generate concurrently a large number of beams thanks to the superposition principle (e.g., combined using the routing module), to create much more complex systems.

Beyond the output coverages shown above, further configurations may model other loudspeaker setups using other combinations of renderers.

<FIG> is a top view of a soundbar <NUM>. The soundbar <NUM> may implement the rendering system <NUM> (see <FIG>). The soundbar <NUM> is similar to the soundbar <NUM> (see <FIG>), and includes the linear array <NUM> (having <NUM> loudspeakers 502a, 502b, 502c, 502d, 502e, 502f, <NUM>, <NUM>, 502i, 502j, <NUM> and <NUM>) and the upward firing group <NUM> (including <NUM> loudspeakers 504a and 504b). The soundbar <NUM> also includes two side firing loudspeakers 1202a and 1202b, with the loudspeaker 1202a referred to as the left side firing loudspeaker and the loudspeaker 1202b referred to as the right side firing loudspeaker.

As compared to the soundbar <NUM> (see <FIG>), the soundbar <NUM> uses the side firing loudspeakers 1202a and 1202b to generate the virtual side outputs 604a and 604b (see <FIG>).

<FIG> is a block diagram of a rendering system <NUM>. The rendering system <NUM> is a modification of the rendering system <NUM> (see <FIG>) suitable for implementation in the soundbar <NUM> (see <FIG>). The rendering system <NUM> may be implemented using the components of the rendering system <NUM> (see <FIG>). The components of the rendering system <NUM> are similar to those of the rendering system <NUM> and use similar reference numbers. As compared to the rendering system <NUM>, the rendering system <NUM> replaces the beamformers 720b and 720c with a binaural renderer <NUM>.

The binaural renderer <NUM> receives the loudspeaker configuration information <NUM>, the object audio data <NUM>, the selection information <NUM>, and the position information <NUM>. The binaural renderer <NUM> performs binaural rendering on the object audio data <NUM> and generates a left binaural signal 1366b and a right binaural signal 1366c. Considering only the side firing loudspeakers 1202a and 1202b (see <FIG>), the left binaural signal 1366b generally corresponds to the output from the left side firing loudspeaker 1202a, and the right binaural signal 1366c generally corresponds to the output from the right side firing loudspeaker 1202b. (Recall that the routing module <NUM> will then combine the binaural signals 1366b and 1366c with the other rendered signals <NUM> to generate the loudspeaker signals <NUM> to the full set of loudspeakers <NUM>, <NUM> and <NUM>.

<FIG> is a block diagram of a renderer <NUM>. The renderer <NUM> may correspond to one or more of the renderers discussed above, such as the renderers <NUM> (see <FIG>), the renderers <NUM> (see <FIG>), the renderers <NUM> (see <FIG>), etc. The renderer <NUM> illustrates that a renderer may include more than one renderer as components thereof. As shown here, the renderer <NUM> includes a renderer <NUM> in series with a renderer <NUM>. Although two renderers <NUM> and <NUM> are shown, the renderer <NUM> may include additional renderers, in assorted serial and parallel configurations. The renderer <NUM> receives the loudspeaker configuration information <NUM>, the selection information <NUM>, and the position information <NUM>; the renderer <NUM> may provide these signals to one or more of the renderers <NUM> and <NUM>, depending upon their particular configurations.

The renderer <NUM> receives the object audio data <NUM>, and one or more of the loudspeaker configuration information <NUM>, the selection information <NUM>, and the position information <NUM>. The renderer <NUM> performs rendering on the object audio data <NUM> and generates rendered signals <NUM>. The rendered signals <NUM> generally correspond to intermediate rendered signals. For example, the rendered signals <NUM> may be virtual speaker feed signals.

The renderer <NUM> receives the rendered signals <NUM>, and one or more of the loudspeaker configuration information <NUM>, the selection information <NUM>, and the position information <NUM>. The renderer <NUM> performs rendering on the rendered signals <NUM> and generates rendered signals <NUM>. The rendered signals <NUM> correspond to the rendered signals discussed above, such as the rendered signals <NUM> (see <FIG>), the rendered signals <NUM> (see <FIG>), the rendered signals <NUM> (see <FIG>), etc. The renderer <NUM> may then provide the rendered signals <NUM> to a routing module (e.g., the routing module <NUM> of <FIG>, the routing module <NUM> of <FIG> or <FIG> or <FIG>), etc. in a manner similar to that discussed above.

In general, the renderers <NUM> and <NUM> have different types in a manner similar to that discussed above. For example, the types may include amplitude panners, vertical panners, wave field renderers, binaural renderers, and beamformers. A specific example configuration is shown in <FIG>.

<FIG> is a block diagram of a renderer <NUM>. The renderer <NUM> may correspond to one or more of the renderers discussed above, such as the renderers <NUM> (see <FIG>), the renderers <NUM> (see <FIG>), the renderers <NUM> (see <FIG>), the renderer <NUM> (see <FIG>), etc. The renderer <NUM> includes an amplitude panner <NUM>, a number N of binaural renderers <NUM> (three shown: 1504a, 1504b and 1504c), and a number M of beamformer sets that include a number of left beamformers <NUM> (three shown: 1506a, 1506b and 1506c) and right beamformers <NUM> (three shown: 1508a, 1508b and 1508c).

The amplitude panner <NUM> receives the object audio data <NUM>, the selection information <NUM>, and the position information <NUM>. The amplitude panner <NUM> performs rendering on the object audio data <NUM> and generates virtual speaker feeds <NUM> (three shown: 1520a, 1520b and 1520c), in a manner similar to the other amplitude panners described herein. The virtual speaker feeds <NUM> may correspond to canonical loudspeaker feed signals such as <NUM>-channel surround signals, <NUM>-channel surround signals, <NUM>. <NUM>-channel surround signals, <NUM>. <NUM>-channel surround signals, <NUM>-channel surround signals, etc. The virtual speaker feeds <NUM> are referred to as "virtual" since they need not be provided directly to actual loudspeakers, but instead may be provided to the other renderers in the renderer <NUM> for further processing.

The specifics of the virtual speaker feeds <NUM> may differ among the various embodiments and implementations of the renderer <NUM>. For example, when the virtual speaker feeds <NUM> include a low-frequency effects channel signal, the amplitude panner <NUM> may provide that channel signal to one or more loudspeakers directly (e.g., bypassing the binaural renderers <NUM> and the beamformers <NUM> and <NUM>). As another example, when the virtual speaker feeds <NUM> include a center channel signal, the amplitude panner <NUM> may provide that channel signal to one or more loudspeakers directly, or may provide that signal directly to a set of one of the left beamformers <NUM> and one of the right beamformers <NUM> (e.g., bypassing the binaural renderers <NUM>).

The binaural renderers <NUM> receive the virtual speaker feeds <NUM> and the loudspeaker configuration information <NUM>. (In general, the number N of binaural renderers <NUM> depends upon the specifics of the embodiments of the renderer <NUM>, such as the number of virtual speaker feeds <NUM>, the type of virtual speaker feed, etc., as discussed above. ) The binaural renderers <NUM> perform rendering on the virtual speaker feeds <NUM> and generate left binaural signals <NUM> (three shown: 1522a, 1522b and 1522c) and right binaural signals <NUM> (three shown: 1524a, 1524b and 1524c), in a manner similar to the other binaural renderers described herein.

The left beamformers <NUM> receive the left binaural signals <NUM> and the loudspeaker configuration information <NUM>, and the right beamformers <NUM> receive the right binaural signals <NUM> and the loudspeaker configuration information <NUM>. Each of the left beamformers <NUM> may receive one or more of the left binaural signals <NUM>, and each of the right beamformers <NUM> may receive one or more of the right binaural signals <NUM>, again depending on the specifics of the embodiments of the renderer <NUM> as discussed above. (These one-or-more relationships are indicated by the dashed lines for <NUM> and <NUM> in <FIG>. ) The left beamformers <NUM> perform rendering on the left binaural signals <NUM> and generate rendered signals <NUM> (three shown: 1566a, 1566b and 1566c). The right beamformers <NUM> perform rendering on the right binaural signals <NUM> and generate rendered signals <NUM> (three shown: 1568a, 1568b and 1568c). The beamformers <NUM> and <NUM> otherwise operate in a manner similar to the other beamformers described herein. The rendered signals <NUM> and <NUM> correspond to the rendered signals discussed above, such as the rendered signals <NUM> (see <FIG>), the rendered signals <NUM> (see <FIG>), the rendered signals <NUM> (see <FIG>), the rendered signals <NUM> (see <FIG>), etc..

The renderer <NUM> may then provide the rendered signals <NUM> and <NUM> to a routing module (e.g., the routing module <NUM> of <FIG>, the routing module <NUM> of <FIG> or <FIG> or <FIG>), etc. in a manner similar to that discussed above.

The number M of left beamformers <NUM> and right beamformers <NUM> depends upon the specifics of the embodiments of the renderer <NUM>, as discussed above. For example, the number M may be varied based on the form factor of the device that includes the renderer <NUM>, on the number of loudspeaker arrays that are connected to the renderer <NUM>, on the capabilities and arrangement of those loudspeaker arrays, etc. As a general guideline, the number M (of beamformers <NUM> and <NUM>) may be less than or equal to the number N (of binaural renderers <NUM>). As another general guideline, the number of separate loudspeaker arrays may be less than or equal to twice the number N (of binaural renderers <NUM>). As one example form factor, a device may have physically separate left and right loudspeaker arrays, where the left loudspeaker array produces all the left beams and the right loudspeaker array produces all the right beams. An another example form factor, a device may have physically separate front and rear loudspeaker arrays, where the front loudspeaker array produces the left and right beams for all front binaural signals, and the rear loudspeaker array produces the left and right beams for all rear binaural signals.

<FIG> is a block diagram of a rendering system <NUM>. The rendering system <NUM> is similar to the rendering system <NUM> (see <FIG>), with the renderers <NUM> (see <FIG>) replaced by a renderer arrangement similar to that of the renderer <NUM> (see <FIG>); there are also differences relating to the distribution module <NUM> (see <FIG>). The rendering system <NUM> includes an amplitude panner <NUM>, a number N of binaural renderers <NUM> (three shown: 1604a, 1604b and 1604c), a number M of beamformer sets that include a number of left beamformers <NUM> (three shown: 1606a, 1606b and 1606c) and right beamformers <NUM> (three shown: 1608a, 1608b and 1508c), and a routing module <NUM>.

The amplitude panner <NUM> receives the object metadata <NUM> and the object audio data <NUM>, performs rendering on the object audio data <NUM> according to the position information in the object metadata <NUM>, and generates virtual speaker feeds <NUM> (three shown: 1620a, 1620b and 1620c), in a manner similar to the other amplitude panners described herein. Similarly, the specifics of the virtual speaker feeds <NUM> may differ among the various embodiments and implementations of the rendering system <NUM>, in a manner similar to that described above regarding the renderer <NUM> (see <FIG>). (As compared to the rendering system <NUM> (see <FIG>), the rendering system <NUM> omits the distribution module <NUM>, but uses the amplitude panner <NUM> to weight the virtual speaker feeds <NUM> among the binaural renderers <NUM>.

The binaural renderers <NUM> receive the virtual speaker feeds <NUM> and the loudspeaker configuration information <NUM>. (In general, the number N of binaural renderers <NUM> depends upon the specifics of the embodiments of the rendering system <NUM>, such as the number of virtual speaker feeds <NUM>, the type of virtual speaker feed, etc., as discussed above. ) The binaural renderers <NUM> perform rendering on the virtual speaker feeds <NUM> and generate left binaural signals <NUM> (three shown: 1622a, 1622b and 1622c) and right binaural signals <NUM> (three shown: 1624a, 1624b and 1624c), in a manner similar to the other binaural renderers described herein.

The left beamformers <NUM> receive the left binaural signals <NUM> and the loudspeaker configuration information <NUM>, and the right beamformers <NUM> receive the right binaural signals <NUM> and the loudspeaker configuration information <NUM>. Each of the left beamformers <NUM> may receive one or more of the left binaural signals <NUM>, and each of the right beamformers <NUM> may receive one or more of the right binaural signals <NUM>, again depending on the specifics of the embodiments of the rendering system <NUM> as discussed above. (These one-or-more relationships are indicated by the dashed lines for <NUM> and <NUM> in <FIG>. ) The left beamformers <NUM> perform rendering on the left binaural signals <NUM> and generate rendered signals <NUM> (three shown: 1666a, 1666b and 1666c). The right beamformers <NUM> perform rendering on the right binaural signals <NUM> and generate rendered signals <NUM> (three shown: 1668a, 1668b and 1668c). The beamformers <NUM> and <NUM> otherwise operate in a manner similar to the other beamformers described herein.

The routing module <NUM> receives the loudspeaker configuration information <NUM>, the rendered signals <NUM> and the rendered signals <NUM>. The routing module <NUM> generates loudspeaker signals <NUM>, in a manner similar to the other routing modules described herein.

At <NUM>, one or more audio objects are received. Each of the audio objects respectively includes position information. As an example, the rendering system <NUM> (see <FIG>) may receive the audio signal <NUM>, which includes the object metadata <NUM> and the object audio data <NUM>. For each of the audio objects, the method continues with <NUM>.

At <NUM>, for a given audio object, the given audio object is rendered, based on the position information, using a first category of renderer to generate a first plurality of signals. For example, the amplitude panner <NUM> (see <FIG>) may render the given audio object (in the object audio data <NUM>) based on the position information (in the object metadata <NUM>) to generate the virtual loudspeaker signals <NUM>.

At <NUM>, for the given audio object, the first plurality of signals are rendered using a second category of renderer to generate a second plurality of signals. For example, the binaural renderers <NUM> (see <FIG>) may render the virtual speaker feeds <NUM> to generate the left binaural signals <NUM> and the right binaural signals <NUM>.

At <NUM>, for the given audio object, the second plurality of signals are rendered using a third category of renderer to generate a third plurality of signals. For example, the left beamformers <NUM> may render the left binaural signals <NUM> to generate the rendered signals <NUM>, and the right beamformers <NUM> may render the right binaural signals <NUM> to generate the rendered signals <NUM>.

At <NUM>, the third plurality of signals are combined to generate a plurality of loudspeaker signals. For example, the routing module <NUM> (see <FIG>) may combine the rendered signals <NUM> and the rendered signals <NUM> to generate the loudspeaker signals <NUM>.

As another example, multiple given audio objects may be processed by combining the rendered signal for each audio object at the output one or more of the rendering stages. Applying this example to the rendering system <NUM> (see <FIG>), the amplitude panner <NUM> may render the multiple given audio objects, each of the virtual loudspeaker signals <NUM> corresponds to a combined rendering that combines the multiple given audio objects, and the binaural renderers <NUM> and the beamformers <NUM> and <NUM> operate on the combined rendering.

An embodiment may be implemented in hardware, executable modules stored on a computer readable medium, or a combination of both (e.g., programmable logic arrays). In particular, various general-purpose machines may be used with programs written in accordance with the teachings herein, or it may be more convenient to construct more specialized apparatus (e.g., integrated circuits) to perform the required method steps. Thus, embodiments may be implemented in one or more computer programs executing on one or more programmable computer systems each comprising at least one processor, at least one data storage system (including volatile and non-volatile memory and/or storage elements), at least one input device or port, and at least one output device or port. Program code is applied to input data to perform the functions described herein and generate output information. The output information is applied to one or more output devices, in known fashion.

Each such computer program is preferably stored on or downloaded to a storage media or device (e.g., solid state memory or media, or magnetic or optical media) readable by a general or special purpose programmable computer, for configuring and operating the computer when the storage media or device is read by the computer system to perform the procedures described herein. The inventive system may also be considered to be implemented as a computer-readable storage medium, configured with a computer program, where the storage medium so configured causes a computer system to operate in a specific and predefined manner to perform the functions described herein. (Software per se and intangible or transitory signals are excluded to the extent that they are unpatentable subject matter.

Claim 1:
A method (<NUM>) of audio processing, the method comprising:
receiving (<NUM>) one or more audio objects, wherein each of the one or more audio objects respectively includes position information;
for a given audio object of the one or more audio objects:
selecting (<NUM>), based on the position information of the given audio object, at least two renderers of a plurality of renderers, wherein the at least two renderers are classified in at least two categories;
characterised by determining (<NUM>), based on the position information of the given audio object, at least two weights;
rendering (<NUM>), based on the position information, the given audio object using the at least two renderers weighted according to the at least two weights, to generate a plurality of rendered signals; and
combining (<NUM>) the plurality of rendered signals to generate a plurality of loudspeaker signals; and
outputting (<NUM>) , from a plurality of loudspeakers, the plurality of loudspeaker signals.