Patent Description:
Head tracking (or headtracking) generally refers to tracking the pose (e.g., the position and orientation) of a user's head to adjust the input to, or output of, a system. For audio, headtracking refers to changing an audio signal according to the head orientation/position of a listener.

Binaural audio generally refers to audio that is recorded, or played back, in such a way that accounts for the natural ear spacing and head shadow of the ears and head of a listener. The listener thus perceives the sounds to originate in one or more spatial locations. Binaural audio may be recorded by using two microphones placed at the two ear locations of a dummy head. Binaural audio may be rendered from audio that was recorded non-binaurally by using a head-related transfer function (HRTF) or a binaural room impulse response (BRIR). Binaural audio may be played back using headphones. Binaural audio generally includes a left channel (to be output by the left headphone), and a right channel (to be output by the right headphone). Binaural audio differs from stereo in that stereo audio may involve loudspeaker crosstalk between the loudspeakers. If binaural audio is to be output from loudspeakers, it is often desirable to perform crosstalk cancellation; an example is described in <CIT>.

Quad binaural generally refers to binaural that has been recorded as four pairs of binaural (e.g., left and right channels for each of the four directions: north at <NUM> degrees, east at <NUM> degrees, south at <NUM> degrees, and west at <NUM> degrees). During playback, if the listener is facing one of the four directions, the binaural signal recorded from that direction is played back. If the listener is facing between two directions, the signal played back is a mixture of the two signals recorded from those two directions.

Binaural audio is often output from headsets or other head-mounted systems. A number of publications describe head-mounted audio systems (that in various ways differ from standard audio headsets). Examples include <CIT>; <CIT>; <CIT>; <CIT>; <CIT>; <CIT>; <CIT>; <CIT>; International Application Pub. No. <CIT>; European Application Pub. No. <CIT>; and Japanese Application <CIT>.

International Application Pub. No. <CIT> at FIG. <NUM> and related description discusses upmixing a two channel binaural signal into four channels: left and right channels for both a front binaural signal and a rear binaural signal. As the orientation of the listener's head changes, the front and rear signals are remixed to convert back to a two channel binaural signal for output.

A number of headsets include visual display elements for virtual reality (VR) or augmented reality (AR). Examples include the Oculus Go™ headset and the Microsoft Hololens™ headset.

A number of publications describe signal processing features for binaural audio. Examples include <CIT>; <CIT>; <CIT>; <CIT>; and <CIT>.

Finally, <CIT> discusses the near-field effect in a speaker array system.

<CIT> describes a method of synthesizing a three dimensional sound-field using a pair of front and a pair of rear loudspeakers, including: a) determining the desired position of a sound source; b) providing a binaural pair of signals corresponding to the sound source using an HRTF filter; c) providing the left signal of said binaural pair of signals to both the left front and left rear loudspeakers through a front and a rear gain control means respectively; d) providing the right signal of said binaural pair of signals to both the right front and right rear loudspeakers through a front and a rear gain control means respectively; e) controlling the ratio of the front signal gains to the rear signal gains as a function of the azimuth angle of the sound source; and f) performing transaural crosstalk cancellation on the front and rear signal pairs through respective transaural crosstalk cancellation means.

<CIT> describes that acoustic near-field transfer functions indicating acoustic near-field propagation channels between loudspeakers and ears of a listener can be employed to pre-process audio signals. Therefore, acoustic near-field distortions of the audio signals can be mitigated. The pre-processed audio signals can be presented to the listener using a wearable frame, wherein the wearable frame comprises the loudspeakers for audio presentation. By means of a loudspeaker selection as a function of a spatial audio source direction, cues related to the listener's ears can be generated. The approach can be extended to an arbitrary number of loudspeaker pairs.

One problem with many binaural audio systems is that it is often difficult for listeners to perceive front-back differentiation of the binaural outputs.

Given the above problems and lack of solutions, the embodiments described herein are directed toward splitting a binaural signal into multiple binaural signals for output by multiple loudspeakers (e.g., front and rear loudspeaker pairs).

The invention is defined by independent claims <NUM>, <NUM> and <NUM>. Further embodiments are set forth in the dependent claims.

According to an embodiment, a method of rendering audio includes receiving a spatial audio signal, where the spatial audio signal includes position information for rendering audio. The method further includes processing the spatial audio signal to determine a plurality of weights based on the position information. The method further includes rendering the spatial audio signal to form a plurality of rendered signals, where the plurality of rendered signals are amplitude weighted according to the plurality of weights, and where the plurality of rendered signals includes a plurality of binaural signals that are amplitude weighted according to the plurality of weights.

Rendering the spatial audio signal to form the plurality of rendered signals includes rendering the spatial audio signal to generate an interim rendered signal, and weighting the interim rendered signal according to the plurality of weights to generate the plurality of rendered signals, or splitting the spatial audio signal, on an amplitude weighting basis, according to the plurality of weights.

The plurality of weights may correspond to a front-back perspective applied to the position information.

The spatial audio signal may include a plurality of audio objects, where each of the plurality of audio objects is associated with a respective position of the position information. Processing the spatial audio signal may include processing the plurality of audio objects to extract the position information. The plurality of weights may correspond to the respective position of each of the plurality of audio objects.

Each of the plurality of rendered signals is a binaural signal that includes a left channel and a right channel.

The plurality of rendered signals may include a front signal and a rear signal, where the front signal includes a left front channel and a right front channel, and where the rear signal includes a left rear channel and a right rear channel.

The plurality of rendered signals may include a front signal, a rear signal, and another signal, where the front signal includes a left front channel and a right front channel, where the rear signal includes a left rear channel and a right rear channel, and where the other signal is an unpaired channel.

The method may further include outputting, from a plurality of loudspeakers, the plurality of rendered signals.

The method includes combining the plurality of rendered signals into a joint rendered signal, generating metadata that relates the joint rendered signal to the plurality of rendered signals, and providing the joint rendered signal and the metadata to a loudspeaker system.

The method may further include generating, by the loudspeaker system, the plurality of rendered signals from the joint rendered signal using the metadata, and outputting, from a plurality of loudspeakers, the plurality of rendered signals.

The method may further include generating headtracking data, and computing, based on the headtracking data, a front delay, a first front set of filter parameters, a second front set of filter parameters, a rear delay, a first rear set of filter parameters, and a second rear set of filter parameters. For a front binaural signal that includes a first channel signal and a second channel signal, the method may further include generating a first modified channel signal by applying the front delay and the first front set of filter parameters to the first channel signal, and generating a second modified channel signal by applying the second front set of filter parameters to the second channel signal. For a rear binaural signal that includes a third channel signal and a fourth channel signal, the method may further include generating a third modified channel signal by applying the second rear set of filter parameters to the third channel signal, and generating a fourth modified channel signal by applying the rear delay and the first rear set of filter parameters to the fourth channel signal. The method may further include outputting, from a first front loudspeaker, the first modified channel signal, outputting, from a second front loudspeaker, the second modified channel signal, outputting, from a first rear loudspeaker, the third modified channel signal, and outputting, from a second rear loudspeaker, the fourth modified channel signal.

According to an embodiment, a non-transitory computer readable medium may store a computer program that, when executed by a processor, controls an apparatus to execute processing including the method steps described herein.

According to an embodiment, an apparatus for rendering audio includes a processor and a memory. The processor is configured to receive a spatial audio signal, where the spatial audio signal includes position information for rendering audio. The processor is configured to process the spatial audio signal to determine a plurality of weights based on the position information. The processor is configured to render the spatial audio signal to form a plurality of rendered signals, where each of the plurality of rendered signals is a binaural signal that includes a left channel and a right channel, wherein rendering the spatial audio signal to form the plurality of rendered signals comprises rendering the spatial audio signal to generate an interim rendered signal and weighting the interim rendered signal according to the plurality of weights to generate the plurality of rendered signals, or splitting the spatial audio signal, on an amplitude weighting basis, according to the plurality of weights. The processor is configured to combine the plurality of rendered signals into a joint rendered signal, to generate metadata that relates the joint rendered signal to the plurality of rendered signals; and to provide the joint rendered signal and the metadata to a loudspeaker system.

The apparatus may further include a left front loudspeaker, a right front loudspeaker, a left rear loudspeaker, and a right rear loudspeaker. The left front loudspeaker is configured to output a left channel of a front binaural signal of the plurality of binaural signals. The right front loudspeaker is configured to output a right channel of the front binaural signal. The left rear loudspeaker is configured to output a left channel of a rear binaural signal of the plurality of binaural signals. The right rear loudspeaker is configured to output a right channel of the rear binaural signal. The plurality of weights correspond to a front-back perspective applied to the left front loudspeaker and the left rear loudspeaker, and applied to the right front loudspeaker and the right rear loudspeaker.

The apparatus may further include a mounting structure that is adapted to position the left front loudspeaker, the left rear loudspeaker, the right front loudspeaker, and the right rear loudspeaker around a head of a listener.

When the spatial audio signal includes a plurality of audio objects, where each of the plurality of audio objects is associated with a respective position of the position information, the processor may be configured configured to process the plurality of audio objects to extract the position information, where the plurality of weights correspond to the respective position of each of the plurality of audio objects.

The apparatus may include further details similar to those described above regarding the method.

The following detailed description and accompanying drawings provide a further understanding of the nature and advantages of various implementations.

Described herein are techniques for binaural audio processing. In the following description, for purposes of explanation, numerous examples and specific details are set forth in order to provide a thorough understanding of the present invention. It will be evident, however, to one skilled in the art that the present invention as defined by the claims may include some or all of the features in these examples alone or in combination with other features described below, and may further include modifications and equivalents of the features and concepts described herein.

In the following description, various methods, processes and procedures are detailed. Although particular steps may be described in a certain order, such order is mainly for convenience and clarity. A particular step may be repeated more than once, may occur before or after other steps (even if those steps are otherwise described in another order), and may occur in parallel with other steps. A second step is required to follow a first step only when the first step must be completed before the second step is begun. Such a situation will be specifically pointed out when not clear from the context.

In this document, the terms "and", "or" and "and/or" are used. Such terms are to be read as having an inclusive meaning. For example, "A and B" may mean at least the following: "both A and B", "at least both A and B". As another example, "A or B" may mean at least the following: "at least A", "at least B", "both A and B", "at least both A and B". As another example, "A and/or B" may mean at least the following: "A and B", "A or B". When an exclusive-or is intended, such will be specifically noted (e.g., "either A or B", "at most one of A and B").

<FIG> is a block diagram of an audio processing system <NUM> not covered by the present invention. The audio processing system <NUM> includes a rendering system <NUM> and a loudspeaker system <NUM>. The rendering system <NUM> receives a spatial audio signal <NUM> and renders the spatial audio signal <NUM> to generate a number of rendered signals 120a,. , 120n (collectively, the rendered signals <NUM>). The loudspeaker system <NUM> receives the rendered signals <NUM> and generates auditory outputs 130a,. , <NUM> (collectively, the auditory outputs <NUM>). (When the rendered signals <NUM> are binaural signals, each of the auditory outputs <NUM> corresponds to two channels of one of the rendered signals <NUM>, so m is twice n.

In general, the spatial audio signal <NUM> includes position information, and the rendering system <NUM> uses the position information when generating the rendered signals <NUM> in order for a listener to perceive the audio as originating from the various positions indicated by the position information. The spatial audio signal <NUM> may include audio objects, such as in the Dolby Atmos™ system or the DTS:X™ system. The spatial audio signal <NUM> may include B-format signals (e.g., using four component channels: W for the sound pressure, X for the front-minus-back sound pressure gradient, Y for left-minus-right, and Z for up-minus-down), such as in the Ambisonics™ system. The spatial audio signal <NUM> may be a surround sound signal, such as a <NUM>-channel or <NUM>-channel stereo signal. For channel signals (such as <NUM>-channel), each channel may be assigned to a defined position, and may be referred to as bed channels. For example, the left bed channel may be provided to the left loudspeaker, etc..

According to an embodiment, the rendering system <NUM> generates the rendered signals <NUM> corresponding to front and rear binaural signals, each with left and right channels; and the loudspeaker system <NUM> includes four speakers that respectively output a left front channel, a right front channel, a left rear channel, and a right rear channel. Further details of the rendering system <NUM> and the loudspeaker system <NUM> are provided below.

<FIG> is a block diagram of a rendering system <NUM>. The rendering system <NUM> may be used as the rendering system <NUM> (see <FIG>). The rendering system <NUM> includes a weight calculator <NUM> and a number of renderers 204a,. , 204n (collectively, the renderers <NUM>). The weight calculator <NUM> receives the spatial audio signal <NUM> and calculates a number of weights <NUM> based on the position information in the spatial audio signal <NUM>. The weights <NUM> correspond to a front-back perspective applied to the position information. The renderers <NUM> render the spatial audio signal <NUM> using the weights <NUM> to generate the rendered signals <NUM>. In general, the renderers <NUM> use the weights <NUM> to perform amplitude weighting of the rendered signals <NUM>. In effect, the renderers <NUM> use the weights <NUM> to split the spatial signal <NUM> on an amplitude weighting basis when generating the rendered signals <NUM>.

For example, an embodiment of the rendering system <NUM> includes two renderers <NUM> (e.g., a front renderer and a rear renderer) that respectively render a front binaural signal and a rear binaural signal (collectively forming the rendered signals <NUM>). When the position information of a particular object indicates the sound is exclusively in the front, the weights <NUM> may be <NUM> provided to the front renderer, and <NUM> provided to the rear renderer, for that particular object. When the position information indicates the sound is exclusively in the rear, the weights <NUM> may be <NUM> provided to the front renderer, and <NUM> provided to the rear renderer, for that particular object. When the position information indicates the sound is exactly between the front and the rear, the weights <NUM> may be <NUM> provided to the front renderer, and <NUM> provided to the rear renderer, for that particular object. When the position information is otherwise between the front and the rear, the weights <NUM> may be similarly apportioned between the front renderer and the rear renderer, for that particular object. The weights <NUM> may be apportioned in an energy preserving manner; for example, when the position information indicates the sound is exactly between the front and the rear, the weights <NUM> may be <NUM>/sqrt(<NUM>) provided to the front renderer, and <NUM>/sqrt(<NUM>) provided to the rear renderer, for that particular object.

<FIG> is a block diagram of a rendering system <NUM>. The rendering system <NUM> may be used as the rendering system <NUM> (see <FIG>). The rendering system <NUM> includes a weight calculator <NUM>, a renderer <NUM>, and a number of weight modules 256a,. , 256n (collectively, the weight modules <NUM>). The weight calculator <NUM> receives the spatial audio signal <NUM> and calculates a number of weights <NUM> based on the position information in the spatial audio signal <NUM>, similarly to the weight calculator <NUM> (see <FIG>). The renderer <NUM> renders the spatial audio signal <NUM> to generate an interim rendered signal <NUM>. When the spatial audio signal <NUM> includes multiple audio objects (or multiple channels) that are to be output at the same time, the renderer <NUM> may process each audio object (or channel) concurrently, for example by assigning processing time shares. The weight modules <NUM> apply the weights <NUM> to the interim rendered signal <NUM> (on a per-object or per-channel basis) to generate the rendered signals <NUM>. Similarly to the rendering system <NUM> (see <FIG>), the weights <NUM> correspond to a front-back perspective applied to the position information, and the weight modules <NUM> use the weights <NUM> to perform amplitude weighting of the interim rendered signal <NUM>.

For example, an embodiment of the rendering system <NUM> includes two weight modules <NUM> (e.g., a front weight module and a rear weight module) that respectively generate a front binaural signal and a rear binaural signal (collectively forming the rendered signals <NUM>), in a manner similar to that described above regarding the weight calculator <NUM> (see <FIG>).

An example of calculating the weights (<NUM> in <FIG> or <NUM> in <FIG>) using Cartesian coordinates is as follows. Given an audio object positioned at a normalized direction V(x,y,z) (with x,y,z values in the range [-<NUM>,<NUM>]) around the head (assuming the head is (<NUM>,<NUM>,<NUM>)) and assuming the positive y-axis is the front direction, the front weight W1 = <NUM>+<NUM>*cos(y) may be used to weight the binaural signal sent to the front speaker pair, and the rear weight W2 = sqrt(<NUM>-W1*W1) can be used for the back speaker pair. In the case of a Dolby Atmos™ presentation where the object's y coordinate in [<NUM>,<NUM>] correspond to a front/back ratio, W1 = cos(y*pi/<NUM>) and W2 = sin(y*pi/<NUM>) may be used.

Continuing the example, further assume four loudspeakers arranged on the front left, the front right, the rear left, and the rear right. The renderer <NUM> (see <FIG>) convolves the audio object signal (e.g., <NUM>) using a left head related transfer function (HRTF) and a right HRTF to generate a left interim rendered signal (e.g., <NUM>) and a right interim rendered signal. The weight modules <NUM> apply the front weight W1 (e.g., <NUM>) to the left interim rendered signal to generate the rendered signal (e.g., 120a) for the front left loudspeaker; the front weight W1 to the right interim rendered signal to generate the rendered signal for the front right loudspeaker; the rear weight W2 to the left interim rendered signal to generate the rendered signal for the rear left loudspeaker; and the rear weight W2 to the right interim rendered signal to generate the rendered signal for the rear right loudspeaker.

Continuing the example for a second audio object, the renderer <NUM> generates a left interim rendered signal and a right interim rendered signal for the signal of the second audio object. The weight modules <NUM> apply the front weight W1 and the rear weight W2 as described above, to generate the rendered signals for the loudspeakers that now include the weighted audio of both audio objects.

For B-format signals (e.g., first order Ambisonics™ or higher order Ambisonics™), the rendering system (e.g., the rendering system <NUM> of <FIG>) may generate a virtual microphone pattern /beam (e.g. cardioid) to first obtain a front and back signals that can be binaurally rendered and sent to the front and back loudspeaker pairs. In such a case, the weighting is achieved by this virtual 'beamforming' process.

For multiple pairs of speakers, a similar approach may be used where cosine lobes pointing towards the direction of each near-field speaker may be used to obtain different input signals or weights suitable for each binaural pair. Generally higher order lobes would be used as the number of speaker pairs increases in a way similar to a higher order Ambisonics™ stream may be decoded on a traditional sound speaker system.

For example, consider four loudspeakers arranged on the front left, the front right, the rear left, and the rear right. Further consider that the spatial audio signal <NUM> is a B-format signal having M basis signals (e.g., <NUM> basis signals w, x, y, z). The renderer <NUM> (see <FIG>) receives the M basis signals and performs a binaural rendering to result in <NUM> interim rendered signals (e.g., a 2x4 matrix of left and right rendered signals for each of the <NUM> basis signals). The weight modules <NUM> implement a weight matrix W of size <NUM> x <NUM> to generate the four output signals to the two speaker pairs. In effect, the weight matrix W performs the 'beamforming' and plays the same role as the weights in the audio object example discussed in the earlier paragraphs.

In summary, for both the audio object case and the B-format case, the rendering of the input signal to binaural need only happen once per object (or soundfield basis signal); the matrixing / beamforming to generate the loudspeaker outputs is an additional matrixing / linear combination operation.

<FIG> is a flowchart of a method <NUM> of rendering audio. The method <NUM> may be performed by the audio processing system <NUM> (see <FIG>), by the rendering system <NUM> (see <FIG>), etc. The method <NUM> may be implemented by to one or more computer programs that are stored or executed by one or more hardware devices.

At <NUM>, a spatial audio signal is received. The spatial audio signal includes position information for rendering audio. For example, the rendering system <NUM> (see <FIG>) or the rendering system <NUM> (see <FIG>) may receive the spatial audio signal <NUM>.

At <NUM>, the spatial audio signal is processed to determine a number of weights based on the position information. For example, the weight calculator <NUM> (see <FIG>) may determine the weights <NUM> based on the position information in the spatial audio signal <NUM>. As another example, the weight calculator <NUM> (see <FIG>) may determine the weights <NUM> based on the position information in the spatial audio signal <NUM>.

At <NUM>, the spatial audio signal is rendered to form a number of rendered signals. The rendered signals are amplitude weighted according to the weights. The rendered signals may include a number of binaural signals that are amplitude weighted according to the weights. As discussed above, generally speaking, these weights may be explicitly based on the x,y,z position of objects, so the system may binauralize each object and then send it to different pairs of speakers with appropriate weights. Alternatively, these weights may be implicitly part of the beamforming pattern. Then several input signals are obtained that can be individually binauralized and sent to their appropriate speaker pairs.

For example, the renderers <NUM> (see <FIG>) may render the spatial audio signal <NUM> to form the rendered signals <NUM>. Each of the renderers <NUM> may use, for a particular audio object, a respective one of the weights <NUM> to perform amplitude weighting when generating its corresponding one of the rendered signals <NUM>. One or more of the renderers <NUM> may be binaural renderers. According to an embodiment, the renderers <NUM> include a front binaural renderer and a rear binaural renderer, and the rendered signals <NUM> include a front binaural signal and a rear binaural signal resulting from rendering one or more audio objects, that have been amplitude weighted according to the weights <NUM>, on a front-back perspective applied to the position information.

As another example, the renderer <NUM> (see <FIG>) renders the spatial audio signal <NUM> to form the interim rendered signal <NUM>, to which the weight modules <NUM> apply the weights <NUM> to form the rendered signals <NUM>. The renderer <NUM> may be a binaural renderer, and the weight modules <NUM> may generate a front binaural signal and a rear binaural signal, using the weights <NUM> to apply a front-back perspective to the interim rendered signal <NUM>.

At <NUM>, a number of loudspeakers output the rendered signals. For example, the loudspeaker system <NUM> (see <FIG>) may output the rendered signals <NUM> as the auditory outputs <NUM>.

<FIG> is a block diagram of a rendering system <NUM>. The rendering system <NUM> includes hardware details for implementing the functions of the rendering system <NUM> (see <FIG>) or the rendering system <NUM> (see <FIG>). The rendering system <NUM> may implement the method <NUM> (see <FIG>), for example by executing one or more computer programs. The rendering system <NUM> includes a processor <NUM>, a memory <NUM>, an input/output interface <NUM>, and an input/output interface <NUM>. A bus <NUM> connects these components. The rendering system <NUM> may include other components that (for brevity) are not shown.

The processor <NUM> generally controls the operation of the rendering system <NUM>. The processor <NUM> may execute one or more computer programs in order to implement the functions of the rendering system <NUM> (see <FIG>), including the weight calculator <NUM> and the renderers <NUM>. Likewise, the processor <NUM> may implement the functions of the rendering system <NUM> (see <FIG>), including the weight calculator <NUM>, the renderer <NUM> and the weight modules <NUM>. The processor <NUM> may include, or be a component of, a programmable logic device or digital signal processor.

The memory <NUM> generally stores the data operated on by the processor <NUM>, such as digital representations of the signals shown in <FIG> such as the spatial audio signal <NUM>, the position information, the weights <NUM> or <NUM>, the interim rendered signal <NUM>, and the rendered signals <NUM>. The memory <NUM> may also store any computer programs executed by the processor <NUM>. The memory <NUM> may include volatile or non-volatile components.

The input/output interfaces <NUM> and <NUM> generally interface the rendering system <NUM> with other components. The input/output interface <NUM> interfaces the rendering system <NUM> with the provider of the spatial audio signal <NUM>. If the spatial audio signal <NUM> is stored locally, the input/output interface <NUM> may communicate with that local component. If the spatial audio signal <NUM> is received from a remote component, the input/output interface <NUM> may communicate with that remote component via a wired or wireless connection.

The input/output interface <NUM> interfaces the rendering system <NUM> with the loudspeaker system <NUM> (see <FIG>) to provide the rendered signals <NUM>. If the loudspeaker system <NUM> and the rendering system <NUM> (see <FIG>) are components of a single device, the input/output interface <NUM> provides a physical interconnection between the components. If the loudspeaker system <NUM> is a separate device from the rendering system <NUM>, the input/output interface <NUM> may provide an interface for a wired or wireless connection (e.g., IEEE <NUM>. <NUM> connection).

<FIG> is a block diagram of a loudspeaker system <NUM>. The loudspeaker system <NUM> includes hardware details for implementing the functions of the loudspeaker system <NUM> (see <FIG>). The loudspeaker system <NUM> may implement <NUM> of the method <NUM> (see <FIG>), for example by executing one or more computer programs. The loudspeaker system <NUM> includes a processor <NUM>, a memory <NUM>, an input/output interface <NUM>, an input/output interface <NUM>, and a number of loudspeakers <NUM> (<NUM> shown, 510a, 510b, 510c and 510d). (Alternatively, a simplified version of the loudspeaker system <NUM> may omit the processor <NUM> and the memory <NUM>, e.g. when the rendering system <NUM> and the loudspeaker system <NUM> are components of a single device. ) A bus <NUM> connects the processor <NUM>, the memory <NUM>, the input/output interface <NUM>, and the input/output interface <NUM>. The loudspeaker system <NUM> may include other components that (for brevity) are not shown.

The processor <NUM> generally controls the operation of the loudspeaker system <NUM>, for example by executing one or more computer programs. The processor <NUM> may include, or be a component of, a programmable logic device or digital signal processor.

The memory <NUM> generally stores the data operated on by the processor <NUM>, such as digital representations of the rendered signals <NUM>. The memory <NUM> may also store any computer programs executed by the processor <NUM>. The memory <NUM> may include volatile or non-volatile components.

The input/output interface <NUM> interfaces the loudspeaker system <NUM> with the rendering system <NUM> (see <FIG>) to receive the rendered signals <NUM>. The input/output interface <NUM> may provide an interface for a wired or wireless connection (e.g., IEEE <NUM>. <NUM> connection). According to an embodiment, the rendered signals <NUM> include a front binaural signal and a rear binaural signal.

The input/output interface <NUM> interfaces the loudspeakers <NUM> with the other components of the loudspeaker system <NUM>.

The loudspeakers <NUM> generally output the auditory signals <NUM> (<NUM> shown, 130a, 130b, 130c and 130d) that correspond to the rendered signals <NUM>. According to an embodiment, the rendered signals <NUM> include a front binaural signal and a rear binaural signal; the loudspeaker 510a outputs a left channel of the front binaural signal, the loudspeaker 510b outputs a right channel of the front binaural signal, the loudspeaker 510c outputs a left channel of the rear binaural signal, and the loudspeaker 510d outputs a right channel of the rear binaural signal.

Since the rendered signals <NUM> have been weighted based on a front-back perspective applied to the position information in the spatial signal <NUM> (as discussed above regarding the rendering system <NUM>), the loudspeakers 510a-510b output the left and right channels of the weighted front binaural signal, and the loudspeakers 510c-510d output the left and right channels of the weighted rear binaural signal. In this manner, the audio processing system <NUM> (see <FIG>) improves the front-back differentiation perceived by a listener.

<FIG> is a top view of a loudspeaker system <NUM>. The loudspeaker system <NUM> corresponds to a specific implementation of the loudspeaker system <NUM> (see <FIG>) or the loudspeaker system <NUM> (see <FIG>). The loudspeaker system <NUM> includes a mounting structure <NUM> that positions the loudspeakers 510a, 510b, 510c and 510d around the head of a listener. The arms of the loudspeakers 510a, 510b, 510c and 510d are positioned <NUM> degrees apart, at <NUM> degrees, <NUM> degrees, <NUM> degrees, and <NUM> degrees (relative to the center of the listener's head, with <NUM> degrees being the listener's front); the loudspeakers themselves may each be angled toward the left ear or right ear of the listener. The loudspeakers 510a, 510b, 510c and 510d are typically positioned close to the listener's head (for example, <NUM> inches away). The loudspeakers 510a, 510b, 510c and 510d are typically low power, e.g. between <NUM> and <NUM> Watts. Given the proximity to the head and the low power, the outputs of the loudspeakers 510a, 510b, 510c and 510d are considered near-field outputs. Near-field outputs have negligible cross-talk interference between the left and right sides of the loudspeakers, so cross-talk cancellation may be omitted in some instances. In addition, the loudspeakers 510a, 510b, 510c and 510d do not obscure the ears of the listener, which allows the listener to also hear ambient sounds and makes the loudspeaker system <NUM> suitable for augmented reality applications.

<FIG> is a right side view of the loudspeaker system <NUM> (see <FIG>), showing the mounting structure <NUM>, the loudspeaker 510b and the loudspeaker 510d. When the helmet structure <NUM> is placed on the head of a listener, the loudspeakers 510b and 510d are horizontally aligned with the listener's right ear. The helmet structure <NUM> may include a solid cap area, straps, etc. for ease of attachment, use and comfort of the wearer.

The configurations of the loudspeakers in the loudspeaker system <NUM> may be varied as desired. For example, the angular separation of the loudspeakers may be adjusted to be greater than, or less than, <NUM> degrees. As another example, the angle of the front loudspeakers may be other than <NUM> and <NUM> degrees (e.g., <NUM> and <NUM> degrees). As a further example, the angle of the rear loudspeakers may be varied to be other than <NUM> and <NUM> degrees (e.g., <NUM> and <NUM> degrees).

The elevations of the loudspeakers in the loudspeaker system <NUM> may also be varied. For example, the loudspeakers may be increased, or decrease, in elevation from the elevations shown in <FIG>.

The quantities of the loudspeakers in the loudspeaker system <NUM> may also be varied. For example, a center loudspeaker may be added between the front loudspeakers 510a and 510b. Since this center loudspeaker outputs an unpaired channel, its corresponding renderer <NUM> (see <FIG>) is not a binaural renderer.

Another option for varying the number of loudspeakers is discussed with regard to <FIG>.

<FIG> is a top view of a loudspeaker system <NUM>. The loudspeaker system <NUM> corresponds to a specific implementation of the loudspeaker system <NUM> (see <FIG>) or the loudspeaker system <NUM> (see <FIG>). The loudspeaker system <NUM> includes a helmet structure <NUM> and loudspeakers 710a, 710b, 710c, 710d, 710e and 710f (collectively the loudspeakers <NUM>). The helmet structure <NUM> positions the loudspeakers 710a, 710b, 710c, 710d similarly to the loudspeakers 510a, 510b, 510c and 510d (see <FIG>). The helmet structure <NUM> positions the loudspeaker 710e adjacent to the listener's left ear (e.g., at <NUM> degrees), and positions the loudspeaker 710f adjacent to the listener's right ear (e.g., at <NUM> degrees).

<FIG> is a right side view of the loudspeaker system <NUM> (see <FIG>), showing the helmet structure <NUM> and the loudspeakers 710b, 710d and 710f.

The configurations, positions, angles, quantities, and elevations of the loudspeakers <NUM> may be varied as desired, similar to the options discussed regarding the loudspeaker <NUM> (see <FIG>).

Embodiments may include a visual display to provide visual VR or AR aspects. For example, the loudspeaker system <NUM> (see <FIG>) may add a visual display system in the form of goggles or a display screen at the front of the helmet structure <NUM>. In such an embodiment, the front loudspeakers 510a and 510b may be attached to the front sides of the visual display system.

As with the other options described above, the configurations, positions, angles, quantities, and elevations of the loudspeakers may be varied as desired.

As an alternative to sending separate rendered signals from the rendering system to the loudspeaker system (e.g., as shown in <FIG> and <FIG>), the rendering system according to the invention combines the rendered signals <NUM> into a combined rendered signal with side chain metadata; the loudspeaker system uses the side chain metadata to un-combine the combined rendered signal into the individual rendered signals <NUM>. Further details are provided with reference to <FIG>.

<FIG> is a block diagram of a rendering system <NUM>. The rendering system <NUM> is similar to the rendering system <NUM> (see <FIG>, including the weight calculator <NUM> and the renderers <NUM>), with the addition of a signal combiner <NUM>. The signal combiner <NUM> combines the rendered signals <NUM> to form a combined signal <NUM>, and generates metadata <NUM> that describes how the rendered signals <NUM> have been combined.

This process of combining may also be referred to as upmixing or forming a joint signal. According to an embodiment, the metadata <NUM> includes front-back amplitude ratios of the left and right channels in various frequency bands (e.g., on a quadrature mirror filter (QMF) sub-band basis).

The rendering system <NUM> may be implemented by components similar to those described above regarding the rendering system <NUM> (see <FIG>).

<FIG> is a block diagram of a rendering system <NUM>. The rendering system <NUM> is similar to the rendering system <NUM> (see <FIG>, including the weight calculator <NUM>, the renderer <NUM> and the weight modules <NUM>), with the addition of a signal combiner <NUM>. The signal combiner <NUM> combines the rendered signals <NUM> to form a combined signal <NUM>, and generates metadata <NUM> that describes how the rendered signals <NUM> have been combined. The signal combiner <NUM>, and the rendering system <NUM>, are otherwise similar to the signal combiner <NUM> and the rendering system <NUM> (see <FIG>).

<FIG> is a block diagram of a loudspeaker system <NUM>. The loudspeaker system <NUM> is similar to the loudspeaker system <NUM> (see <FIG>, including the loudspeakers <NUM> as shown in <FIG>), with the addition of a signal extractor <NUM>. The signal extractor <NUM> receives the combined signal <NUM> and the metadata <NUM> (see <FIG>), and uses the metadata <NUM> to generate the rendered signals <NUM> from the combined signal <NUM>. The loudspeaker system <NUM> then outputs the rendered signals <NUM> from its loudspeakers as the auditory outputs <NUM>, as discussed above.

The loudspeaker system <NUM> may be implemented by components similar to those described above regarding the loudspeaker system <NUM> (see <FIG>).

As mentioned above, the audio processing system <NUM> (see <FIG>) may include headtracking.

<FIG> is a block diagram of a loudspeaker system <NUM> that implements headtracking. The loudspeaker system <NUM> includes a sensor <NUM>, a front headtracking system <NUM>, a rear headtracking system <NUM>, a left front loudspeaker 1010a, a right front loudspeaker 1010b, a left rear loudspeaker 1010c, and a right rear loudspeaker 1010d. The loudspeaker system <NUM> receives two rendered signals <NUM> (see, e.g., <FIG>), which are referred to as a front binaural signal 120a and a rear binaural signal 120b; each include left and right channels. The loudspeaker system <NUM> generates four auditory outputs <NUM>, which are referred to as a left front auditory output 130a, a right front auditory output 130b, a left rear auditory output 130c, and a right rear auditory output 130d.

The sensor <NUM> detects the orientation of the loudspeaker system <NUM> and generates headtracking data <NUM> that corresponds to the detected orientation. The sensor <NUM> may be an accelerometer, a gyroscope, a magnetometer, an infrared sensor, a camera, a radio frequency link, or any other type of sensor that allows for headtracking. The sensor <NUM> may be a multi-axis sensor. The sensor <NUM> may be one of a number of sensors that generate the headtracking data <NUM> (e.g., one sensor generates azimuthal data, another sensor generates elevational data, etc.).

The front headtracking system <NUM> modifies the front binaural signal 120a according to the headtracking data <NUM> to generate a modified front binaural signal 120a'. In general, the modified front binaural signal 120a' corresponds to the front binaural signal 120a, but modified so that the listener perceives the front binaural signal 120a according to the changed orientation of the loudspeaker system <NUM>.

The rear headtracking system <NUM> modifies the rear binaural signal 120b according to the headtracking data <NUM> to generate a modified rear binaural signal 120b'. In general, the modified rear binaural signal 120b' corresponds to the rear binaural signal 120b, but modified so that the listener perceives the rear binaural signal 120b according to the changed orientation of the loudspeaker system <NUM>.

Further details of the front and rear headtracking systems <NUM> and <NUM> are provided with reference to <FIG>.

The left front loudspeaker 1010a outputs a left channel of the modified front binaural signal 120a' as the left front auditory output 130a. The right front loudspeaker 1010b outputs a right channel of the modified front binaural signal 120a' as the right front auditory output 130b. The left rear loudspeaker 1010c outputs a left channel of the modified rear binaural signal 120b' as the left rear auditory output 130c. The right rear loudspeaker 1010d outputs a right channel of the modified rear binaural signal 120b' as the right rear auditory output 130d.

As with the other embodiments described above, the configurations, positions, angles, quantities, and elevations of the loudspeakers in the loudspeaker system <NUM> may be varied as desired.

<FIG> is a block diagram of the front headtracking system <NUM> (see <FIG>). The front headtracking system <NUM> includes a calculation block <NUM>, a delay block <NUM>, a delay block <NUM>, a filter block <NUM>, and a filter block <NUM>. The front headtracking system <NUM> receives as inputs the headtracking data <NUM>, an input left signal L <NUM>, and an input right signal R <NUM>. (The signals <NUM> and <NUM> correspond to left and right channels of the front binaural signal 120a. ) The front headtracking system <NUM> generates as outputs an output left signal L' <NUM> and an output right signal R' <NUM>. (The signals <NUM> and <NUM> correspond to left and right channels of the modified front binaural signal 120a'.

The calculation block <NUM> generates a delay and filter parameters based on the headtracking data <NUM>, provides the delay to the delay blocks <NUM> and <NUM>, and provides the filter parameters to the filter blocks <NUM> and <NUM>. The filter coefficients may be calculated according to the Brown-Duda model (see <NPL>)), and the delay values may be calculated according to the Woodworth approximation (see <NPL>)), or any corresponding system of inter-aural level and time difference.

The delay block <NUM> applies the appropriate delay to the input left signal L <NUM>, and the delay block <NUM> applies the appropriate delay to the input right signal R <NUM>. For example, a leftward turn provides a delay D1 to the delay block <NUM>, and zero delay to the delay block <NUM>. Similarly, a rightward turn provides zero delay to the delay block <NUM>, and a delay D2 to the delay block <NUM>.

The filter block <NUM> applies the appropriate filtering to the delayed signal from the delay block <NUM>, and the filter block <NUM> applies the appropriate filtering to the delayed signal from the delay block <NUM>. The appropriate filtering will be either ipsilateral filtering (for the "near" ear) or contralateral filtering (for the "far" ear), depending upon the headtracking data <NUM>. For example, for a leftward turn, the filter block <NUM> applies a contralateral filter, and the filter block <NUM> applies an ipsilateral filter. Similarly, for a rightward turn, the filter block <NUM> applies an ipsilateral filter, and the filter block <NUM> applies a contralateral filter.

The rear headtracking system <NUM> may be implemented similarly to the front headtracking system <NUM>. Differences include operating on the rear binaural signal 120b (instead of on the front binaural signal 120a), and inverting the headtracking data <NUM> from that used by the front headtracking system <NUM>. For example, when the headtracking data <NUM> indicates a leftward turn of <NUM> degrees (+<NUM> degrees), the front headtracking system <NUM> uses (+<NUM> degrees) for its processing, and the rear headtracking system <NUM> inverts the headtracking data <NUM> as (-<NUM> degrees) for its processing. Another difference is that the delay and the filter coefficients for the rear are slightly different from those for the front. In any event, the front headtracking system <NUM> and the rear headtracking system <NUM> may share the calculation block <NUM>.

The details of the headtracking operations may otherwise be similar to those described in International Application Pub.

An embodiment may be implemented in hardware, executable modules stored on a computer readable medium, or a combination of both (e.g., programmable logic arrays). Unless otherwise specified, the steps executed by embodiments need not inherently be related to any particular computer or other apparatus, although they may be in certain embodiments. In particular, various general-purpose machines may be used with programs written in accordance with the teachings herein, or it may be more convenient to construct more specialized apparatus (e.g., integrated circuits) to perform the required method steps. Thus, embodiments may be implemented in one or more computer programs executing on one or more programmable computer systems each comprising at least one processor, at least one data storage system (including volatile and non-volatile memory and/or storage elements), at least one input device or port, and at least one output device or port. Program code is applied to input data to perform the functions described herein and generate output information. The output information is applied to one or more output devices, in known fashion.

Each such computer program is preferably stored on or downloaded to a storage media or device (e.g., solid state memory or media, or magnetic or optical media) readable by a general or special purpose programmable computer, for configuring and operating the computer when the storage media or device is read by the computer system to perform the procedures described herein. The inventive system may also be considered to be implemented as a computer-readable storage medium, configured with a computer program, where the storage medium so configured causes a computer system to operate in a specific and predefined manner to perform the functions described herein. (Software per se and intangible or transitory signals are excluded to the extent that they are unpatentable subject matter.

Claim 1:
A method of rendering audio, the method comprising:
receiving (<NUM>) a spatial audio signal (<NUM>), wherein the spatial audio signal includes position information for rendering audio;
processing (<NUM>) the spatial audio signal to determine a plurality of weights (<NUM>, <NUM>) based on the position information;
rendering (<NUM>) the spatial audio signal to form a plurality of rendered signals, wherein each of the plurality of rendered signals is a binaural signal that includes a left channel and a right channel,
wherein rendering the spatial audio signal to form the plurality of rendered signals comprises:
- rendering the spatial audio signal to generate an interim rendered signal (<NUM>) and weighting the interim rendered signal according to the plurality of weights to generate the plurality of rendered signals, or
- splitting the spatial audio signal, on an amplitude weighting basis, according to the plurality of weights;
combining the plurality of rendered signals into a joint rendered signal (<NUM>, <NUM>);
generating metadata (<NUM>, <NUM>) that relates the joint rendered signal to the plurality of rendered signals; and
providing the joint rendered signal and the metadata to a loudspeaker system (<NUM>).