Patent Description:
This disclosure pertains to systems and methods for automatically locating audio devices.

Audio devices, including but not limited to smart audio devices, have been widely deployed and are becoming common features of many homes. Although existing systems and methods for locating audio devices provide benefits, improved systems and methods would be desirable.

<CIT> discloses a method for determining spatial relations in a loudspeaker system comprising a central controller, multiple loudspeakers and multiple microphones. <CIT> discloses that the method comprises using the multiple loudspeakers to produce sound and using the multiple microphones to measure the produced sound, and that spatial relations between the loudspeakers are determined based on the measured sound, and the multiple loudspeakers are assigned to loudspeaker zones based on the determined spatial relations. <CIT> discloses that audio of the multiple loudspeakers is controlled in accordance with the assigned loudspeaker zones.

<CIT> discloses an electronic apparatus of an audio output system including a first communicator configured to transmit a first radio signal and receive a first response signal; a second communicator configured to transmit a second radio signal and receive a second response signal; and a processor configured to: determine a first distance between the first communicator and an audio output apparatus based on the first response signal, determine a second distance between the second communicator and the audio output apparatus based on the second response signal, determine a location of the audio output apparatus based on the first distance and the second distance; establish a communication connection with the audio output apparatus based on one from among the first response signal and the second response signal; and set a channel of the audio output apparatus based on the determined location of the audio output apparatus.

<CIT> discloses methods, systems, and apparatuses for determining distances and angles between speakers and other home theater components for performance of calibration operations and functions. <CIT> discloses that distances are determined through timing playback of sweep signals from a reference speaker to the speakers, and also by timing playback the sweep signals between the speakers and a capture device at a desired listening position, and further discloses that angles between the speakers and other theater components are determined based on the distances, and locations or mappings of the speakers and other theater components are determined from the angles and distances.

<CIT> discloses an AV system including a camera-equipped loudspeaker provided with a camera, which camera is united with a loudspeaker body, and captures an image in a direction in which the loudspeaker body outputs a sound.

<CIT> discloses that the recognition unit recognizes a location of a listener from an image of the camera, and detects an orientation of the loudspeaker body relative to the listener. <CIT> further discloses that the sound control unit performs signal processing on a given sound signal for generating an output signal, and outputs the output signal as an acoustic signal to the loudspeaker body.

<CIT> discloses an audio system that adjusts based on the location of a person. <CIT> discloses that instead of relying on fixed speakers, the audio system adjusts the direction of audio output for one or more speakers to optimize the performance of the audio system based on the location of a user or based on the number of users. <CIT> discloses that to do so, the audio system may include a camera and a tracking application which identifies the location of a user and/or the number of users in front of the camera, and using this information, the audio system adjusts one or more actuators coupled to a speaker to change the direction of the audio output of the speaker.

<CIT> discloses that a device in control of a speaker system may receive from one or more sensors information indicative of a user's location and/or body position, and that the device may use the information to the determine the user's location and/or body position and may adjust the outputs of speakers in the sound system. <CIT> discloses that the device may instruct a speaker to switch from outputting a first audio channel at a first volume level to outputting a second audio channel at a second volume level, and that if the user changes location or body position, the device may adjust the speaker's output to accommodate the user's new location and/or body position.

<CIT> discloses a method for calibrating a surround sound system, which method utilizes a microphone array integrated in a front center loudspeaker of the surround sound system or a soundbar facing a listener. <CIT> discloses that positions of each loudspeaker relative to the microphone array can be estimated by playing a test signal at each loudspeaker and measuring the test signal received at the microphone array, and that the listener's position can also be estimated by receiving the listener's voice or other sound cues made by the listener using the microphone array. <CIT> discloses that once the positions of the loudspeakers and the listener's position are estimated, spatial calibrations can be performed for each loudspeaker in the surround sound system so that listening experience is optimized.

Herein, we use the expression "smart audio device" to denote a smart device which is either a single purpose audio device or a virtual assistant (e.g., a connected virtual assistant). A single purpose audio device is a device (e.g., a smart speaker, a television (TV) or a mobile phone) including or coupled to at least one microphone (and which may in some examples also include or be coupled to at least one speaker) and which is designed largely or primarily to achieve a single purpose. Although a TV typically can play (and is thought of as being capable of playing) audio from program material, in most instances a modem TV runs some operating system on which applications run locally, including the application of watching television. Similarly, the audio input and output in a mobile phone may do many things, but these are serviced by the applications running on the phone. In this sense, a single purpose audio device having speaker(s) and microphone(s) is often configured to run a local application and/or service to use the speaker(s) and microphone(s) directly. Some single purpose audio devices may be configured to group together to achieve playing of audio over a zone or user-configured area.

Herein, a "virtual assistant" (e.g., a connected virtual assistant) is a device (e.g., a smart speaker, a smart display or a voice assistant integrated device) including or coupled to at least one microphone (and optionally also including or coupled to at least one speaker) and which may provide an ability to utilize multiple devices (distinct from the virtual assistant) for applications that are in a sense cloud enabled or otherwise not implemented in or on the virtual assistant itself. Virtual assistants may sometimes work together, e.g., in a very discrete and conditionally defined way. For example, two or more virtual assistants may work together in the sense that one of them, i.e., the one which is most confident that it has heard a wakeword, responds to the word. Connected devices may form a sort of constellation, which may be managed by one main application which may be (or include or implement) a virtual assistant.

Herein, "wakeword" is used in a broad sense to denote any sound (e.g., a word uttered by a human, or some other sound), where a smart audio device is configured to awake in response to detection of ("hearing") the sound (using at least one microphone included in or coupled to the smart audio device, or at least one other microphone). In this context, to "awake" denotes that the device enters a state in which it awaits (i.e., is listening for) a sound command.

Herein, the expression "wakeword detector" denotes a device configured (or software that includes instructions for configuring a device) to search continuously for alignment between real-time sound (e.g., speech) features and a trained model. Typically, a wakeword event is triggered whenever it is determined by a wakeword detector that the probability that a wakeword has been detected exceeds a predefined threshold. For example, the threshold may be a predetermined threshold which is tuned to give a good compromise between rates of false acceptance and false rejection. Following a wakeword event, a device might enter a state (which may be referred to as an "awakened" state or a state of "attentiveness") in which it listens for a command and passes on a received command to a larger, more computationally-intensive recognizer.

Throughout this disclosure, including in the claims, "speaker" and "loudspeaker" are used synonymously to denote any sound-emitting transducer (or set of transducers) driven by a single speaker feed. A typical set of headphones includes two speakers. A speaker may be implemented to include multiple transducers (e.g., a woofer and a tweeter), all driven by a single, common speaker feed. The speaker feed may, in some instances, undergo different processing in different circuitry branches coupled to the different transducers.

Throughout this disclosure including in the claims, the expression "system" is used in a broad sense to denote a device, system, or subsystem. For example, a subsystem that implements a decoder may be referred to as a decoder system, and a system including such a subsystem (e.g., a system that generates X output signals in response to multiple inputs, in which the subsystem generates M of the inputs and the other X - M inputs are received from an external source) may also be referred to as a decoder system.

Throughout this disclosure including in the claims, the term "processor" is used in a broad sense to denote a system or device programmable or otherwise configurable (e.g., with software or firmware) to perform operations on data (e.g., audio, or video or other image data). Examples of processors include a field-programmable gate array (or other configurable integrated circuit or chip set), a digital signal processor programmed and/or otherwise configured to perform pipelined processing on audio or other sound data, a programmable general purpose processor or computer, and a programmable microprocessor chip or chip set.

At least some aspects of the present disclosure may be implemented via methods. Some such methods may involve audio device location, i.e. a method of determining a location of a plurality of (e.g. of at least four or more) audio devices in the environment. For example, some methods may involve obtaining direction of arrival (DOA) data for each audio device of a plurality of audio devices and determining interior angles for each of a plurality of triangles based on the DOA data. In some instances, each triangle of the plurality of triangles may have vertices that correspond with audio device locations of three of the audio devices. Some such methods may involve determining a side length for each side of each of the triangles based, at least in part, on the interior angles.

Some such methods may involve performing a forward alignment process of aligning each of the plurality of triangles in a first sequence, to produce a forward alignment matrix. Some such methods may involve performing a reverse alignment process of aligning each of the plurality of triangles in a second sequence that is the reverse of the first sequence, to produce a reverse alignment matrix. Some such methods may involve producing a final estimate of each audio device location based, at least in part, on values of the forward alignment matrix and values of the reverse alignment matrix.

According to some examples, producing the final estimate of each audio device location may involve translating and scaling the forward alignment matrix to produce a translated and scaled forward alignment matrix, and translating and scaling the reverse alignment matrix to produce a translated and scaled reverse alignment matrix. Some such methods may involve producing a rotation matrix based on the translated and scaled forward alignment matrix and the translated and scaled reverse alignment matrix. The rotation matrix may include a plurality of estimated audio device locations for each audio device. In some implementations, producing the rotation matrix may involve performing a singular value decomposition on the translated and scaled forward alignment matrix and the translated and scaled reverse alignment matrix. According to some examples, producing the final estimate of each audio device location may involve averaging the estimated audio device locations for each audio device to produce the final estimate of each audio device location.

In some implementations, determining the side length may involve determining a first length of a first side of a triangle and determining lengths of a second side and a third side of the triangle based on the interior angles of the triangle. Determining the first length may, in some examples, involve setting the first length to a predetermined value. Determining the first length may, in some examples, be based on time-of-arrival data and/or received signal strength data.

According to some examples, obtaining the DOA data may involve determining the DOA data for at least one audio device of the plurality of audio devices. In some instances, determining the DOA data may involve receiving microphone data from each microphone of a plurality of audio device microphones corresponding to a single audio device of the plurality of audio devices and determining the DOA data for the single audio device based, at least in part, on the microphone data. According to some examples, determining the DOA data may involve receiving antenna data from one or more antennas corresponding to a single audio device of the plurality of audio devices and determining the DOA data for the single audio device based, at least in part, on the antenna data.

In some implementations, the method also may involve controlling at least one of the audio devices based, at least in part, on the final estimate of at least one audio device location. In some such examples, controlling at least one of the audio devices may involve controlling a loudspeaker of at least one of the audio devices.

Some or all of the operations, functions and/or methods described herein may be performed by one or more devices according to instructions (e.g., software) stored on one or more non-transitory media. Such non-transitory media may include memory devices such as those described herein, including but not limited to random access memory (RAM) devices, read-only memory (ROM) devices, etc. Accordingly, some innovative aspects of the subject matter described in this disclosure can be implemented in a non-transitory medium having software stored thereon.

For example, the software may include instructions for controlling one or more devices to perform a method that involves audio device location. Some methods may involve obtaining DOA data for each audio device of a plurality of audio devices and determining interior angles for each of a plurality of triangles based on the DOA data. In some instances, each triangle of the plurality of triangles may have vertices that correspond with audio device locations of three of the audio devices. Some such methods may involve determining a side length for each side of each of the triangles based, at least in part, on the interior angles.

At least some aspects of the present disclosure may be implemented via apparatus. For example, one or more devices may be capable of performing, at least in part, the methods disclosed herein. In some implementations, an apparatus may include an interface system and a control system. The control system may include one or more general purpose single- or multi-chip processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic, discrete hardware components, or combinations thereof. In some examples, the apparatus may be one of the above-referenced audio devices. However, in some implementations the apparatus may be another type of device, such as a mobile device, a laptop, a server, etc..

In some aspects of the present disclosure any of the methods describes may be implemented in a computer program product comprising instructions which, when the program is executed by a computer, cause the computer to carry out any of the methods or steps of the methods described in this disclosure.

In some aspect of the present disclosure, there is described a computer-readable medium comprising the computer program product.

The advent of smart speakers, incorporating multiple drive units and microphone arrays, in addition to existing audio devices including televisions and sound bars, and new microphone and loudspeaker-enabled connected devices such as lightbulbs and microwaves, creates a problem in which dozens of microphones and loudspeakers need locating relative to one another in order to achieve orchestration. Audio devices cannot be assumed to lie in canonical layouts (such as a discrete Dolby <NUM> loudspeaker layout). In some instances, the audio devices in an environment may be randomly located, or at least may be distributed within the environment in an irregular and/or asymmetric manner.

Moreover, audio devices cannot be assumed to be heterogeneous or synchronous. As used herein, audio devices may be referred to as "synchronous" or "synchronized" if sounds are detected by, or emitted by, the audio devices according to the same sample clock, or synchronized sample clocks. For example, a first synchronized microphone of a first audio device within an environment may digitally sample audio data according to a first sample clock and a second microphone of a second synchronized audio device within the environment may digitally sample audio data according to the first sample clock. Alternatively, or additionally, a first synchronized speaker of a first audio device within an environment may emit sound according to a speaker set-up clock and a second synchronized speaker of a second audio device within the environment may emit sound according to the speaker set-up clock.

Some previously-disclosed methods for automatic speaker location require synchronized microphones and/or speakers. For example, some previously-existing tools for device localization rely upon sample synchrony between all microphones in the system, requiring known test stimuli and passing full-bandwidth audio data between sensors.

The present assignee has produced several speaker localization techniques for cinema and home that are excellent solutions in the use cases for which they were designed. Some such methods are based on time-of-flight derived from impulse responses between a sound source and microphone(s) that are approximately co-located with each loudspeaker. While system latencies in the record and playback chains may also be estimated, sample synchrony between clocks is required along with the need for a known test stimulus from which to estimate impulse responses.

Recent examples of source localization in this context have relaxed constraints by requiring intra-device microphone synchrony but not requiring inter-device synchrony. Additionally, some such methods relinquish the need for passing audio between sensors by low-bandwidth message passing such as via detection of the time of arrival (TOA) of a direct (non-reflected) sound or via detection of the dominant direction of arrival (DOA) of a direct sound. Each approach has some potential advantages and potential drawbacks. For example, TOA methods can determine device geometry up to an unknown translation, rotation, and reflection about one of three axes. Rotations of individual devices are also unknown if there is just one microphone per device. DOA methods can determine device geometry up to an unknown translation, rotation, and scale. While some such methods may produce satisfactory results under ideal conditions, the robustness of such methods to measurement error has not been demonstrated.

Some implementations of the present disclosure automatically locate the positions of multiple audio devices in an environment (e.g., in a room) by applying a geometrically-based optimization using asynchronous DOA estimates from uncontrolled sound sources observed by a microphone array in each device. Various disclosed audio device location approaches have proven to be robust to large DOA estimation errors.

Some such implementations involve iteratively aligning triangles derived from sets of DOA data. In some such examples, each audio device may contain a microphone array that estimates DOA from an uncontrolled source. In some implementations, microphone arrays may be collocated with at least one loudspeaker. However, at least some disclosed methods generalize to cases in which not all microphone arrays are collocated with a loudspeaker.

According to some disclosed methods, DOA data from every audio device to every other audio device in an environment may be aggregated. The audio device locations may be estimated by iteratively aligning triangles parameterized by pairs of DOAs. Some such methods may yield a result that is correct up to an unknown scale and rotation. In many applications, absolute scale is unnecessary, and rotations can be resolved by placing additional constraints on the solution. For example, some multi-speaker environments may include television (TV) speakers and a couch positioned for TV viewing. After locating the speakers in the environment, some methods may involve finding a vector pointing to the TV and locating the speech of a user sitting on the couch by triangulation. Some such methods may then involve having the TV emit a sound from its speakers and/or prompting the user to walk up to the TV and locating the user's speech by triangulation. Some implementations may involve rendering an audio object that pans around the environment. A user may provide user input (e.g., saying "Stop") indicating when the audio object is in one or more predetermined positions within the environment, such as the front of the environment, at a TV location of the environment, etc. According to some such examples, after locating the speakers within an environment and determining their orientation, the user may be located by finding the intersection of directions of arrival of sounds emitted by multiple speakers. Some implementations involve determining an estimated distance between at least two audio devices and scaling the distances between other audio devices in the environment according to the estimated distance.

<FIG> shows an example of geometric relationships between three audio devices in an environment. In this example, the environment <NUM> is a room that includes a television <NUM>, a sofa <NUM> and five audio devices <NUM>. According to this example, the audio devices <NUM> are in locations <NUM> through <NUM> of the environment <NUM>. In this implementation, each of the audio devices <NUM> includes a microphone system <NUM> having at least three microphones and a speaker system <NUM> that includes at least one speaker. In some implementations, each microphone system <NUM> includes an array of microphones. According to some implementations, each of the audio devices <NUM> may include an antenna system that includes at least three antennas.

As with other examples disclosed herein, the type, number and arrangement of elements shown in <FIG> are merely made by way of example. Other implementations may have different types, numbers and arrangements of elements, e.g., more or fewer audio devices <NUM>, audio devices <NUM> in different locations, etc..

In this example, the triangle 110a has its vertices at locations <NUM>, <NUM> and <NUM>. Here, the triangle 110a has sides <NUM>, 23a and 13a. According to this example, the angle between sides <NUM> and <NUM> is θ<NUM>, the angle between sides <NUM> and 13a is θ<NUM> and the angle between sides 23a and 13a is θ<NUM>. These angles may be determined according to DOA data, as described in more detail below.

In some implementations, only the relative lengths of triangle sides may be determined. In alternative implementations, the actual lengths of triangle sides may be estimated. According to some such implementations, the actual length of a triangle side may be estimated according to TOA data, e.g., according to the time of arrival of sound produced by an audio device located at one triangle vertex and detected by an audio device located at another triangle vertex. Alternatively, or additionally, the length of a triangle side may be estimated according to electromagnetic waves produced by an audio device located at one triangle vertex and detected by an audio device located at another triangle vertex. For example, the length of a triangle side may be estimated according to the signal strength of electromagnetic waves produced by an audio device located at one triangle vertex and detected by an audio device located at another triangle vertex. In some implementations, the length of a triangle side may be estimated according to a detected phase shift of electromagnetic waves.

<FIG> shows another example of geometric relationships between three audio devices in the environment shown in <FIG>. In this example, the triangle 110b has its vertices at locations <NUM>, <NUM> and <NUM>. Here, the triangle 110b has sides 13b, <NUM> and 34a. According to this example, the angle between sides 13b and <NUM> is θ<NUM>, the angle between sides 13b and 34a is θ<NUM> and the angle between sides 34a and <NUM> is θ<NUM>.

By comparing <FIG> and <FIG>, one may observe that the length of side 13a of triangle 110a should equal the length of side 13b of triangle 110b. In some implementations, the side lengths of one triangle (e.g., triangle 110a) may be assumed to be correct, and the length of a side shared by an adjacent triangle will be constrained to this length.

<FIG> shows both of the triangles depicted in <FIG> and <FIG>, without the corresponding audio devices and the other features of the environment. <FIG> shows estimates of the side lengths and angular orientations of triangles 110a and 110b. In the example shown in <FIG>, the length of side 13b of triangle 110b is constrained to be the same length as side 13a of triangle 110a. The lengths of the other sides of triangle 110b are scaled in proportion to the resulting change in the length of side 13b. The resulting triangle 110b' is shown in <FIG>, adjacent to the triangle 110a.

According to some implementations, the side lengths of other triangles adjacent to triangle 110a and 110b may be all determined in a similar fashion, until all of the audio device locations in the environment <NUM> have been determined.

Some examples of audio device location may proceed as follows. Each audio device may report the DOA of every other audio device in an environment (e.g., a room) based on sounds produced by every other audio device in the environment. The Cartesian coordinates of the ith audio device may be expressed as xi = [xi, yi]T, where the superscript T indicates a vector transpose. Given M audio devices in the environment, i = {<NUM>.

<FIG> shows an example of estimating the interior angles of a triangle formed by three audio devices. In this example, the audio devices are i, j and k. The DOA of a sound source emanating from device j as observed from device i may be expressed as θji. The DOA of a sound source emanating from device k as observed from device i may be expressed as θki. In the example shown in <FIG>, θij and θki are measured from axis 305a, the orientation of which is arbitrary and which may, for example, correspond to the orientation of audio device i. Interior angle a of triangle <NUM> may be expressed as a = θki - θji. One may observe that the calculation of interior angle a does not depend on the orientation of the axis 305a.

In the example shown in <FIG>, θij and θkj are measured from axis 305b, the orientation of which is arbitrary and which may correspond to the orientation of audio device j. Interior angle b of triangle <NUM> may be expressed as b = θij - θkj. Similarly, θjk and θik are measured from axis 305c in this example. Interior angle c of triangle <NUM> may be expressed as c = θjk - θik.

In the presence of measurement error, a + b + c ≠ <NUM>°. Robustness can be improved by predicting each angle from the other two angles and averaging, e.g., as follows: <MAT>.

In some implementations, the edge lengths (A, B, C) may be calculated (up to a scaling error) by applying the sine rule. In some examples, one edge length may be assigned an arbitrary value, such as <NUM>. For example, by making A = <NUM> and placing vertex x̂a = [<NUM>,<NUM>]T at the origin, the locations of the remaining two vertices may be calculated as follows: <MAT>.

However, an arbitrary rotation may be acceptable.

According to some implementations, the process of triangle parameterization may be repeated for all possible subsets of three audio devices in the environment, enumerated in superset ζ of size <MAT>. In some examples, Tl may represent the lth triangle. Depending on the implementation, triangles may not be enumerated in any particular order. The triangles may overlap and may not align perfectly, due to possible errors in the DOA and/or side length estimates.

<FIG> is a flow diagram that outlines one example of a method that may be performed by an apparatus such as that shown in <FIG>. The blocks of method <NUM>, like other methods described herein, are not necessarily performed in the order indicated. Moreover, such methods may include more or fewer blocks than shown and/or described. In this implementation, method <NUM> involves estimating a speaker's location in an environment. The blocks of method <NUM> may be performed by one or more devices, which may be (or may include) the apparatus <NUM> shown in <FIG>.

In this example, block <NUM> involves obtaining direction of arrival (DOA) data for each audio device of a plurality of audio devices. In some examples, the plurality of audio devices may include all of the audio devices in an environment, such as all of the audio devices <NUM> shown in <FIG>.

However, in some instances the plurality of audio devices may include only a subset of all of the audio devices in an environment. For example, the plurality of audio devices may include all smart speakers in an environment, but not one or more of the other audio devices in an environment.

The DOA data may be obtained in various ways, depending on the particular implementation. In some instances, determining the DOA data may involve determining the DOA data for at least one audio device of the plurality of audio devices. For example, determining the DOA data may involve receiving microphone data from each microphone of a plurality of audio device microphones corresponding to a single audio device of the plurality of audio devices and determining the DOA data for the single audio device based, at least in part, on the microphone data. Alternatively, or additionally, determining the DOA data may involve receiving antenna data from one or more antennas corresponding to a single audio device of the plurality of audio devices and determining the DOA data for the single audio device based, at least in part, on the antenna data.

In some such examples, the single audio device itself may determine the DOA data. According to some such implementations, each audio device of the plurality of audio devices may determine its own DOA data. However, in other implementations another device, which may be a local or a remote device, may determine the DOA data for one or more audio devices in the environment. According to some implementations, a server may determine the DOA data for one or more audio devices in the environment.

According to this example, block <NUM> involves determining interior angles for each of a plurality of triangles based on the DOA data. In this example, each triangle of the plurality of triangles has vertices that correspond with audio device locations of three of the audio devices. Some such examples are described above.

<FIG> shows an example in which each audio device in an environment is a vertex of multiple triangles. The sides of each triangle correspond with distances between two of the audio devices <NUM>.

In this implementation, block <NUM> involves determining a side length for each side of each of the triangles. (A side of a triangle may also be referred to herein as an "edge. ") According to this example, the side lengths are based, at least in part, on the interior angles. In some instances, the side lengths may be calculated by determining a first length of a first side of a triangle and determining lengths of a second side and a third side of the triangle based on the interior angles of the triangle. Some such examples are described above.

According to some such implementations, determining the first length may involve setting the first length to a predetermined value. The lengths of the second and third sides may be then determined based on the interior angles of the triangle. All sides of the triangles may be determined based on the predetermined value, e.g. a reference value. In order to get actual distances (lengths) between the audio devices in the environment, a standardized scaling may be applied to the geometry resulting from the alignment processes described below with reference to blocks <NUM> and <NUM> of <FIG>. This standardized scaling may include scaling the aligned triangles such that they fit a bounding shape, e.g. a circle, a polygon, etc., of a size corresponding to the environment. The size of the shape may be the size of a typical home environment or an arbitrary size suitable for the specific implementation. However, scaling the aligned triangles is not limited to fitting the geometry to a specific bounding shape, any other scaling criteria may be used which are suitable for the specific implementation.

In some examples, determining the first length may be based on time-of-arrival data and/or received signal strength data. The time-of-arrival data and/or received signal strength data may, in some implementations, correspond to sound waves from a first audio device in an environment that are detected by a second audio device in the environment. Alternatively, or additionally, the time-of-arrival data and/or received signal strength data may correspond to electromagnetic waves (e.g., radio waves, infrared waves, etc.) from a first audio device in an environment that are detected by a second audio device in the environment. When time-of-arrival data and/or received signal strength data are not available, the first length may be set to the predetermined value as described above.

According to this example, block <NUM> involves performing a forward alignment process of aligning each of the plurality of triangles in a first sequence. According to this example, the forward alignment process produces a forward alignment matrix.

According to some such examples, triangles are expected to align in such a way that an edge (xi,xj) is equal to a neighboring edge, e.g., as shown in <FIG> and described above. Let ε be the set of all edges of size <MAT>. In some such implementations, block <NUM> may involve traversing through ε and aligning the common edges of triangles in forward order by forcing an edge to coincide with that of a previously aligned edge.

<FIG> provides an example of part of a forward alignment process. The numbers <NUM> through <NUM> that are shown in bold in <FIG> correspond with the audio device locations shown in <FIG>, <FIG> and <FIG>. The sequence of the forward alignment process that is shown in <FIG> and described herein is merely an example.

In this example, as in <FIG>, the length of side 13b of triangle 110b is forced to coincide with the length of side 13a of triangle 110a. The resulting triangle 110b' is shown in <FIG>, with the same interior angles maintained. According to this example, the length of side 13c of triangle 110c is also forced to coincide with the length of side 13a of triangle 110a. The resulting triangle 110c' is shown in <FIG>, with the same interior angles maintained.

Next, in this example, the length of side 34b of triangle 110d is forced to coincide with the length of side 34a of triangle 110b'. Moreover, in this example, the length of side 23b of triangle 110d is forced to coincide with the length of side 23a of triangle 110a. The resulting triangle 110d' is shown in <FIG>, with the same interior angles maintained. According to some such examples, the remaining triangles shown in <FIG> may be processed in the same manner as triangles 110b, 110c and 110d.

The results of the forward alignment process may be stored in a data structure. According to some such examples, the results of the forward alignment process may be stored in a forward alignment matrix. For example, the results of the forward alignment process may be stored in matrix <MAT>. , where N indicates the total number of triangles.

When the DOA data and/or the initial side length determinations contain errors, multiple estimates of audio device location will occur. The errors will generally increase during the forward alignment process.

<FIG> shows an example of multiple estimates of audio device location that have occurred during a forward alignment process. In this example, the forward alignment process is based on triangles having seven audio device locations as their vertices. Here, the triangles do not align perfectly due to additive errors in the DOA estimates. The locations of the numbers <NUM> through <NUM> that are shown in <FIG> correspond to the estimated audio device locations produced by the forward alignment process. In this example, the audio device location estimates labelled "<NUM>" coincide but the audio device locations estimates for audio devices <NUM> and <NUM> show larger differences, as indicted by the relatively larger areas over which the numbers <NUM> and <NUM> are located.

Returning to <FIG>, in this example block <NUM> involves a reverse alignment process of aligning each of the plurality of triangles in a second sequence that is the reverse of the first sequence. According to some implementations, the reverse alignment process may involve traversing through ε as before, but in reverse order. In alternative examples, the reverse alignment process may not be precisely the reverse of the sequence of operations of the forward alignment process. According to this example, the reverse alignment process produces a reverse alignment matrix, which may be represented herein as <MAT>.

<FIG> provides an example of part of a reverse alignment process. The numbers <NUM> through <NUM> that are shown in bold in <FIG> correspond with the audio device locations shown in <FIG>, <FIG> and <FIG>. The sequence of the reverse alignment process that is shown in <FIG> and described herein is merely an example.

In the example shown in <FIG>, triangle 110e is based on audio device locations <NUM>, <NUM> and <NUM>. In this implementation, the side lengths (or "edges") of triangle 110e are assumed to be correct, and the side lengths of adjacent triangles are forced to coincide with them. According to this example, the length of side 45b of triangle 110f is forced to coincide with the length of side 45a of triangle 110e. The resulting triangle 110f', with interior angles remaining the same, is shown in <FIG>. In this example, the length of side 35b of triangle 110c is forced to coincide with the length of side 35a of triangle 110e. The resulting triangle 110c", with interior angles remaining the same, is shown in <FIG>. According to some such examples, the remaining triangles shown in <FIG> may be processed in the same manner as triangles 110c and 110f, until the reverse alignment process has included all remaining triangles.

<FIG> shows an example of multiple estimates of audio device location that have occurred during a reverse alignment process. In this example, the reverse alignment process is based on triangles having the same seven audio device locations as their vertices that are described above with reference to <FIG>. The locations of the numbers <NUM> through <NUM> that are shown in <FIG> correspond to the estimated audio device locations produced by the reverse alignment process. Here again, the triangles do not align perfectly due to additive errors in the DOA estimates. In this example, the audio device location estimates labelled <NUM> and <NUM> coincide, but the audio device location estimates for audio devices <NUM> and <NUM> show larger differences.

Returning to <FIG>, block <NUM> involves producing a final estimate of each audio device location based, at least in part, on values of the forward alignment matrix and values of the reverse alignment matrix. In some examples, producing the final estimate of each audio device location may involve translating and scaling the forward alignment matrix to produce a translated and scaled forward alignment matrix, and translating and scaling the reverse alignment matrix to produce a translated and scaled reverse alignment matrix.

For example, translation and scaling are fixed by moving the centroids to the origin and forcing unit Frobenius norm, e.g., <MAT> and <MAT>.

According to some such examples, producing the final estimate of each audio device location also may involve producing a rotation matrix based on the translated and scaled forward alignment matrix and the translated and scaled reverse alignment matrix. The rotation matrix may include a plurality of estimated audio device locations for each audio device. An optimal rotation between forward and reverse alignments is can be found, for example, by singular value decomposition. In some such examples, involve producing the rotation matrix may involve performing a singular value decomposition on the translated and scaled forward alignment matrix and the translated and scaled reverse alignment matrix, e.g., as follows: <MAT>.

In the foregoing equation, U represents the left-singular vector and V represents the right-singular vector of matrix <MAT> respectively. Σ represents a matrix of singular values. The foregoing equation yields a rotation matrix R = VUT. The matrix product VUT yields a rotation matrix such that <MAT> is optimally rotated to align with X.

According to some examples, after determining the rotation matrix R = VUT alignments may be averaged, e.g., as follows: <MAT>.

In some implementations, producing the final estimate of each audio device location also may involve averaging the estimated audio device locations for each audio device to produce the final estimate of each audio device location. Various disclosed implementations have proven to be robust, even when the DOA data and/or other calculations include significant errors. For example, X contains <MAT>, i.e. multiple, estimates of the same node due to overlapping vertices from multiple triangles. Averaging across common nodes yields a final estimate <MAT>.

<FIG> shows a comparison of estimated and actual audio device locations. In the example shown in <FIG>, the audio device locations correspond to those that were estimated during the forward and reverse alignment processes that are described above with reference to <FIG> and <FIG>. In these examples, the errors in the DOA estimations had a standard deviation of <NUM> degrees. Nonetheless, the final estimates of each audio device location (each of which is represented by an "x" in <FIG>) correspond well with the actual audio device locations (each of which is represented by a circle in <FIG>). By performing a forward alignment process in a first sequence and a reverse alignment process in a second sequence reversed to the first sequence, errors/inaccuracies in the direction of arrival estimates (data) are averaged out, thereby reducing the overall error of estimates of audio devices locations in the environment. Errors tend to accumulate in the alignment sequence as shown in <FIG> (where larger vertex numbers show larger alignment spread) and <FIG> (where lower vertex numbers show larger spread). The process of traversing the sequence in the reverse order also reverses the alignment error, thereby averaging out the overall error in the final location estimate.

<FIG> is a block diagram that shows examples of components of an apparatus capable of implementing various aspects of this disclosure. According to some examples, the apparatus <NUM> may be, or may include, a smart audio device (such as a smart speaker) that is configured for performing at least some of the methods disclosed herein. In other implementations, the apparatus <NUM> may be, or may include, another device that is configured for performing at least some of the methods disclosed herein. In some such implementations the apparatus <NUM> may be, or may include, a server.

In this example, the apparatus <NUM> includes an interface system <NUM> and a control system <NUM>. The interface system <NUM> may, in some implementations, be configured for receiving input from each of a plurality of microphones in an environment. The interface system <NUM> may include one or more network interfaces and/or one or more external device interfaces (such as one or more universal serial bus (USB) interfaces). According to some implementations, the interface system <NUM> may include one or more wireless interfaces. The interface system <NUM> may include one or more devices for implementing a user interface, such as one or more microphones, one or more speakers, a display system, a touch sensor system and/or a gesture sensor system. In some examples, the interface system <NUM> may include one or more interfaces between the control system <NUM> and a memory system, such as the optional memory system <NUM> shown in <FIG>. However, the control system <NUM> may include a memory system.

The control system <NUM> may, for example, include a general purpose single- or multi-chip processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, and/or discrete hardware components. In some implementations, the control system <NUM> may reside in more than one device. For example, a portion of the control system <NUM> may reside in a device within the environment <NUM> that is depicted in <FIG>, and another portion of the control system <NUM> may reside in a device that is outside the environment <NUM>, such as a server, a mobile device (e.g., a smartphone or a tablet computer), etc. The interface system <NUM> also may, in some such examples, reside in more than one device.

In some implementations, the control system <NUM> may be configured for performing, at least in part, the methods disclosed herein. According to some examples, the control system <NUM> may be configured for implementing the methods described above, e.g., with reference to <FIG> and/or the methods described below with reference to <FIG> et seq. In some such examples, the control system <NUM> may be configured for determining, based at least in part on output from the classifier, an estimate of each of a plurality of audio device locations within an environment.

In some examples, the apparatus <NUM> may include the optional microphone system <NUM> that is depicted in <FIG>. The microphone system <NUM> may include one or more microphones. In some examples, the microphone system <NUM> may include an array of microphones. In some examples, the apparatus <NUM> may include the optional speaker system <NUM> that is depicted in <FIG>. The speaker system <NUM> may include one or more loudspeakers. In some examples, the microphone system <NUM> may include an array of loudspeakers. In some such examples the apparatus <NUM> may be, or may include, an audio device. For example, the apparatus <NUM> may be, or may include, one of the audio devices <NUM> shown in <FIG>.

In some examples, the apparatus <NUM> may include the optional antenna system <NUM> that is shown in <FIG>. According to some examples, the antenna system <NUM> may include an array of antennas. In some examples, the antenna system <NUM> may be configured for transmitting and/or receiving electromagnetic waves. According to some implementations, the control system <NUM> may be configured to estimate the distance between two audio devices in an environment based on antenna data from the antenna system <NUM>. For example, the control system <NUM> may be configured to estimate the distance between two audio devices in an environment according to the time of arrival of the antenna data and/or the received signal strength of the antenna data.

Some or all of the methods described herein may be performed by one or more devices according to instructions (e.g., software) stored on one or more non-transitory media. Such non-transitory media may include memory devices such as those described herein, including but not limited to random access memory (RAM) devices, read-only memory (ROM) devices, etc. The one or more non-transitory media may, for example, reside in the optional memory system <NUM> shown in <FIG> and/or in the control system <NUM>. Accordingly, various innovative aspects of the subject matter described in this disclosure can be implemented in one or more non-transitory media having software stored thereon. The software may, for example, include instructions for controlling at least one device to process audio data. The software may, for example, be executable by one or more components of a control system such as the control system <NUM> of <FIG>.

Much of the foregoing discussion involves audio device auto-location. The following discussion expands upon some methods of determining listener location and listener angular orientation that are described briefly above. In the foregoing description, the term "rotation" is used in essentially the same way as the term "orientation" is used in the following description. For example, the above-referenced "rotation" may refer to a global rotation of the final speaker geometry, not the rotation of the individual triangles during the process that is described above with reference to <FIG> et seq. This global rotation or orientation may be resolved with reference to a listener angular orientation, e.g., by the direction in which the listener is looking, by the direction in which the listener's nose is pointing, etc..

Various satisfactory methods for estimating listener location are known in the art, some of which are described below. However, estimating the listener angular orientation can be challenging. Some relevant methods are described in detail below.

Determining listener location and listener angular orientation can enable some desirable features, such as orienting located audio devices relative to the listener. Knowing the listener position and angular orientation allows a determination of, e.g., which speakers within an environment would be in the front, which are in the back, which are near the center (if any), etc., relative to the listener.

After making a correlation between audio device locations and a listener's location and orientation, some implementations may involve providing the audio device location data, the audio device angular orientation data, the listener location data and the listener angular orientation data to an audio rendering system. Alternatively, or additionally, some implementations may involve an audio data rendering process that is based, at least in part, on the audio device location data, the audio device angular orientation data, the listener location data and the listener angular orientation data.

<FIG> is a flow diagram that outlines one example of a method that may be performed by an apparatus such as that shown in <FIG>. The blocks of method <NUM>, like other methods described herein, are not necessarily performed in the order indicated. Moreover, such methods may include more or fewer blocks than shown and/or described. In this example, the blocks of method <NUM> are performed by a control system, which may be (or may include) the control system <NUM> shown in <FIG>. As noted above, in some implementations the control system <NUM> may reside in a single device, whereas in other implementations the control system <NUM> may reside in two or more devices.

In this example, block <NUM> involves obtaining direction of arrival (DOA) data for each audio device of a plurality of audio devices in an environment. In some examples, the plurality of audio devices may include all of the audio devices in an environment, such as all of the audio devices <NUM> shown in <FIG>.

The DOA data may be obtained in various ways, depending on the particular implementation. In some instances, determining the DOA data may involve determining the DOA data for at least one audio device of the plurality of audio devices. In some examples, the DOA data may be obtained by controlling each loudspeaker of a plurality of loudspeakers in the environment to reproduce a test signal. For example, determining the DOA data may involve receiving microphone data from each microphone of a plurality of audio device microphones corresponding to a single audio device of the plurality of audio devices and determining the DOA data for the single audio device based, at least in part, on the microphone data. Alternatively, or additionally, determining the DOA data may involve receiving antenna data from one or more antennas corresponding to a single audio device of the plurality of audio devices and determining the DOA data for the single audio device based, at least in part, on the antenna data.

According to the example shown in <FIG>, block <NUM> involves producing, via the control system, audio device location data based at least in part on the DOA data. In this example, the audio device location data includes an estimate of an audio device location for each audio device referenced in block <NUM>.

The audio device location data may, for example, be (or include) coordinates of a coordinate system, such as a Cartesian, spherical or cylindrical coordinate system. The coordinate system may be referred to herein as an audio device coordinate system. In some such examples, the audio device coordinate system may be oriented with reference to one of the audio devices in the environment. In other examples, the audio device coordinate system may be oriented with reference to an axis defined by a line between two of the audio devices in the environment. However, in other examples the audio device coordinate system may be oriented with reference to another part of the environment, such as a television, a wall of a room, etc..

In some examples, block <NUM> may involve the processes described above with reference to <FIG>. According to some such examples, block <NUM> may involve determining interior angles for each of a plurality of triangles based on the DOA data. In some instances, each triangle of the plurality of triangles may have vertices that correspond with audio device locations of three of the audio devices. Some such methods may involve determining a side length for each side of each of the triangles based, at least in part, on the interior angles.

Some such methods may involve performing a forward alignment process of aligning each of the plurality of triangles in a first sequence, to produce a forward alignment matrix. Some such methods may involve performing a reverse alignment process of aligning each of the plurality of triangles in a second sequence that is the reverse of the first sequence, to produce a reverse alignment matrix. Some such methods may involve producing a final estimate of each audio device location based, at least in part, on values of the forward alignment matrix and values of the reverse alignment matrix. However, in some implementations of method <NUM> block <NUM> may involve applying methods other than those described above with reference to <FIG>.

In this example, block <NUM> involves determining, via the control system, listener location data indicating a listener location within the environment. The listener location data may, for example, be with reference to the audio device coordinate system. However, in other examples the coordinate system may be oriented with reference to the listener or to a part of the environment, such as a television, a wall of a room, etc..

In some examples, block <NUM> may involve prompting the listener (e.g., via an audio prompt from one or more loudspeakers in the environment) to make one or more utterances and estimating the listener location according to DOA data. The DOA data may correspond to microphone data obtained by a plurality of microphones in the environment. The microphone data may correspond with detections of the one or more utterances by the microphones. At least some of the microphones may be co-located with loudspeakers. According to some examples, block <NUM> may involve a triangulation process. For example, block <NUM> may involve triangulating the user's voice by finding the point of intersection between DOA vectors passing through the audio devices, e.g., as described below with reference to <FIG>. According to some implementations, block <NUM> (or another operation of the method <NUM>) may involve co-locating the origins of the audio device coordinate system and the listener coordinate system, which is after the listener location is determined. Co-locating the origins of the audio device coordinate system and the listener coordinate system may involve transforming the audio device locations from the audio device coordinate system to the listener coordinate system.

According to this implementation, block <NUM> involves determining, via the control system, listener angular orientation data indicating a listener angular orientation. The listener angular orientation data may, for example, be made with reference to a coordinate system that is used to represent the listener location data, such as the audio device coordinate system. In some such examples, the listener angular orientation data may be made with reference to an origin and/or an axis of the audio device coordinate system.

However, in some implementations the listener angular orientation data may be made with reference to an axis defined by the listener location and another point in the environment, such as a television, an audio device, a wall, etc. In some such implementations, the listener location may be used to define the origin of a listener coordinate system. The listener angular orientation data may, in some such examples, be made with reference to an axis of the listener coordinate system.

Various methods for performing block <NUM> are disclosed herein. According to some examples, the listener angular orientation may correspond to a listener viewing direction. In some such examples the listener viewing direction may be inferred with reference to the listener location data, e.g., by assuming that the listener is viewing a particular object, such as a television. In some such implementations, the listener viewing direction may be determined according to the listener location and a television location. Alternatively, or additionally, the listener viewing direction may be determined according to the listener location and a television soundbar location.

However, in some examples the listener viewing direction may be determined according to listener input. According to some such examples, the listener input may include inertial sensor data received from a device held by the listener. The listener may use the device to point at location in the environment, e.g., a location corresponding with a direction in which the listener is facing. For example, the listener may use the device to point to a sounding loudspeaker (a loudspeaker that is reproducing a sound). Accordingly, in such examples the inertial sensor data may include inertial sensor data corresponding to the sounding loudspeaker.

In some such instances, the listener input may include an indication of an audio device selected by the listener. The indication of the audio device may, in some examples, include inertial sensor data corresponding to the selected audio device.

However, in other examples the indication of the audio device may be made according to one or more utterances of the listener (e.g., "the television is in front of me now. " "speaker <NUM> is in front of me now," etc.). Other examples of determining listener angular orientation data according to one or more utterances of the listener are described below.

According to the example shown in <FIG>, block <NUM> involves determining, via the control system, audio device angular orientation data indicating an audio device angular orientation for each audio device relative to the listener location and the listener angular orientation. According to some such examples, block <NUM> may involve a rotation of audio device coordinates around a point defined by the listener location. In some implementations, block <NUM> may involve a transformation of the audio device location data from an audio device coordinate system to a listener coordinate system. Some examples are described below.

<FIG> shows examples of some blocks of <FIG>. According to some such examples, the audio device location data includes an estimate of an audio device location for each of audio devices <NUM>-<NUM>, with reference to the audio device coordinate system <NUM>. In this implementation, the audio device coordinate system <NUM> is a Cartesian coordinate system having the location of the microphone of audio device <NUM> as its origin. Here, the x axis of the audio device coordinate system <NUM> corresponds with a line <NUM> between the location of the microphone of audio device <NUM> and the location of the microphone of audio device <NUM>.

In this example, this example, the listener location is determined by prompting the listener <NUM> who is shown seated on the couch <NUM> (e.g., via an audio prompt from one or more loudspeakers in the environment 1300a) to make one or more utterances <NUM> and estimating the listener location according to time-of-arrival (TOA) data. The TOA data corresponds to microphone data obtained by a plurality of microphones in the environment. In this example, the microphone data corresponds with detections of the one or more utterances <NUM> by the microphones of at least some (e.g., <NUM>, <NUM> or all <NUM> ) of the audio devices <NUM>-<NUM>.

Alternatively, or additionally, the listener location according to DOA data provided by the microphones of at least some (e.g., <NUM>, <NUM>, <NUM> or all <NUM> ) of the audio devices <NUM>-<NUM>. According to some such examples, the listener location may be determined according to the intersection of lines 1309a, 1309b, etc., corresponding to the DOA data.

According to this example, the listener location corresponds with the origin of the listener coordinate system <NUM>. In this example, the listener angular orientation data is indicated by the y' axis of the listener coordinate system <NUM>, which corresponds with a line 1313a between the listener's head <NUM> (and/or the listener's nose <NUM>) and the sound bar <NUM> of the television <NUM>. In the example shown in <FIG>, the line 1313a is parallel to the y' axis. Therefore, the angle Θ represents the angle between the y axis and the y' axis. In this example, block <NUM> of <FIG> may involve a rotation by the angle Θ of audio device coordinates around the origin of the listener coordinate system <NUM>. Accordingly, although the origin of the audio device coordinate system <NUM> is shown to correspond with audio device <NUM> in <FIG>, some implementations involve co-locating the origin of the audio device coordinate system <NUM> with the origin of the listener coordinate system <NUM> prior to the rotation by the angle Θ of audio device coordinates around the origin of the listener coordinate system <NUM>. This co-location may be performed by a coordinate transformation from the audio device coordinate system <NUM> to the listener coordinate system <NUM>.

The location of the sound bar <NUM> and/or the television <NUM> may, in some examples, be determined by causing the sound bar to emit a sound and estimating the sound bar's location according to DOA and/or TOA data, which may correspond detections of the sound by the microphones of at least some (e.g., <NUM>, <NUM> or all <NUM> ) of the audio devices <NUM>-<NUM>. Alternatively, or additionally, the location of the sound bar <NUM> and/or the television <NUM> may be determined by prompting the user to walk up to the TV and locating the user's speech by DOA and/or TOA data, which may correspond detections of the sound by the microphones of at least some (e.g., <NUM>, <NUM> or all <NUM> ) of the audio devices <NUM>-<NUM>. Such methods may involve triangulation. Such examples may be beneficial in situations wherein the sound bar <NUM> and/or the television <NUM> has no associated microphone.

In some other examples wherein the sound bar <NUM> and/or the television <NUM> does have an associated microphone, the location of the sound bar <NUM> and/or the television <NUM> may be determined according to TOA or DOA methods, such as the DOA methods disclosed herein. According to some such methods, the microphone may be co-located with the sound bar <NUM>.

According to some implementations, the sound bar <NUM> and/or the television <NUM> may have an associated camera <NUM>. A control system may be configured to capture an image of the listener's head <NUM> (and/or the listener's nose <NUM>). In some such examples, the control system may be configured to determine a line 1313a between the listener's head <NUM> (and/or the listener's nose <NUM>) and the camera <NUM>. The listener angular orientation data may correspond with the line 1313a. Alternatively, or additionally, the control system may be configured to determine an angle Θ between the line 1313a and the y axis of the audio device coordinate system.

<FIG> shows an additional example of determining listener angular orientation data. According to this example, the listener location has already been determined in block <NUM> of <FIG>. Here, a control system is controlling loudspeakers of the environment 1300b to render the audio object <NUM> to a variety of locations within the environment 1300b. In some such examples, the control system may cause the loudspeakers to render the audio object <NUM> such that the audio object <NUM> seems to rotate around the listener <NUM>, e.g., by rendering the audio object <NUM> such that the audio object <NUM> seems to rotate around the origin of the listener coordinate system <NUM>. In this example, the curved arrow <NUM> shows a portion of the trajectory of the audio object <NUM> as it rotates around the listener <NUM>.

According to some such examples, the listener <NUM> may provide user input (e.g., saying "Stop") indicating when the audio object <NUM> is in the direction that the listener <NUM> is facing. In some such examples, the control system may be configured to determine a line 1313b between the listener location and the location of the audio object <NUM>. In this example, the line 1313b corresponds with the y' axis of the listener coordinate system, which indicates the direction that the listener <NUM> is facing. In alternative implementations, the listener <NUM> may provide user input indicating when the audio object <NUM> is in the front of the environment, at a TV location of the environment, at an audio device location, etc..

<FIG> shows an additional example of determining listener angular orientation data. According to this example, the listener location has already been determined in block <NUM> of <FIG>. Here, the listener <NUM> is using a handheld device <NUM> to provide input regarding a viewing direction of the listener <NUM>, by pointing the handheld device <NUM> towards the television <NUM> or the soundbar <NUM>. The dashed outline of the handheld device <NUM> and the listener's arm indicate that at a time prior to the time at which the listener <NUM> was pointing the handheld device <NUM> towards the television <NUM> or the soundbar <NUM>, the listener <NUM> was pointing the handheld device <NUM> towards audio device <NUM> in this example. In other examples, the listener <NUM> may have pointed the handheld device <NUM> towards another audio device, such as audio device <NUM>. According to this example, the handheld device <NUM> is configured to determine an angle α between audio device <NUM> and the television <NUM> or the soundbar <NUM>, which approximates the angle between audio device <NUM> and the viewing direction of the listener <NUM>.

The handheld device <NUM> may, in some examples, be a cellular telephone that includes an inertial sensor system and a wireless interface configured for communicating with a control system that is controlling the audio devices of the environment 1300c. In some examples, the handheld device <NUM> may be running an application or "app" that is configured to control the handheld device <NUM> to perform the necessary functionality, e.g., by providing user prompts (e.g., via a graphical user interface), by receiving input indicating that the handheld device <NUM> is pointing in a desired direction, by saving the corresponding inertial sensor data and/or transmitting the corresponding inertial sensor data to the control system that is controlling the audio devices of the environment 1300c, etc..

According to this example, a control system (which may be a control system of the handheld device <NUM> or a control system that is controlling the audio devices of the environment 1300c) is configured to determine the orientation of lines 1313c and <NUM> according to the inertial sensor data, e.g., according to gyroscope data. In this example, the line 1313c is parallel to the axis y' and may be used to determine the listener angular orientation. According to some examples, a control system may determine an appropriate rotation for the audio device coordinates around the origin of the listener coordinate system <NUM> according to the angle α between audio device <NUM> and the viewing direction of the listener <NUM>.

<FIG> shows one example of determine an appropriate rotation for the audio device coordinates in accordance with the method described with reference to <FIG>. In this example, the origin of the audio device coordinate system <NUM> is co-located with the origin of the listener coordinate system <NUM>. Co-locating the origins of the audio device coordinate system <NUM> and the listener coordinate system <NUM> is made possible after the process of <NUM>, wherein the listener location is determined. Co-locating the origins of the audio device coordinate system <NUM> and the listener coordinate system <NUM> may involve transforming the audio device locations from the audio device coordinate system <NUM> to the listener coordinate system <NUM>. The angle α has been determined as described above with reference to <FIG>. Accordingly, the angle α corresponds with the desired orientation of the audio device <NUM> in the listener coordinate system <NUM>. In this example, the angle β corresponds with the orientation of the audio device <NUM> in the audio device coordinate system <NUM>. The angle Θ, which is β-α in this example, indicates the necessary rotation to align the y axis of the of the audio device coordinate system <NUM> with the y' axis of the listener coordinate system <NUM>.

In some implementations, the method of <FIG> may involve controlling at least one of the audio devices in the environment based at least in part on a corresponding audio device location, a corresponding audio device angular orientation, the listener location data and the listener angular orientation data.

For example, some implementations may involve providing the audio device location data, the audio device angular orientation data, the listener location data and the listener angular orientation data to an audio rendering system. In some examples, the audio rendering system may be implemented by a control system, such as the control system <NUM> of <FIG>. Some implementations may involve controlling an audio data rendering process based, at least in part, on the audio device location data, the audio device angular orientation data, the listener location data and the listener angular orientation data. Some such implementations may involve providing loudspeaker acoustic capability data to the rendering system. The loudspeaker acoustic capability data may correspond to one or more loudspeakers of the environment. The loudspeaker acoustic capability data may indicate an orientation of one or more drivers, a number of drivers or a driver frequency response of one or more drivers. In some examples, the loudspeaker acoustic capability data may be retrieved from a memory and then provided to the rendering system.

Existing flexible rendering techniques include Center of Mass Amplitude Panning (CMAP) and Flexible Virtualization (FV). From a high level, both these techniques render a set of one or more audio signals, each with an associated desired perceived spatial position, for playback over a set of two or more speakers, where the relative activation of speakers of the set is a function of a model of perceived spatial position of said audio signals played back over the speakers and a proximity of the desired perceived spatial position of the audio signals to the positions of the speakers. The model ensures that the audio signal is heard by the listener near its intended spatial position, and the proximity term controls which speakers are used to achieve this spatial impression. In particular, the proximity term favors the activation of speakers that are near the desired perceived spatial position of the audio signal. For both CMAP and FV, this functional relationship is conveniently derived from a cost function written as the sum of two terms, one for the spatial aspect and one for proximity: <MAT>.

Here, the set {si} denotes the positions of a set of M loudspeakers, o denotes the desired perceived spatial position of the audio signal, and g denotes an M dimensional vector of speaker activations. For CMAP, each activation in the vector represents a gain per speaker, while for FV each activation represents a filter (in this second case g can equivalently be considered a vector of complex values at a particular frequency and a different g is computed across a plurality of frequencies to form the filter). The optimal vector of activations is found by minimizing the cost function across activations: <MAT>.

With certain definitions of the cost function, it is difficult to control the absolute level of the optimal activations resulting from the above minimization, though the relative level between the components of gopt is appropriate. To deal with this problem, a subsequent normalization of gopt may be performed so that the absolute level of the activations is controlled. For example, normalization of the vector to have unit length may be desirable, which is in line with a commonly used constant power panning rules: <MAT>.

The exact behavior of the flexible rendering algorithm is dictated by the particular construction of the two terms of the cost function, Cspatial and Cproximity. For CMAP, Cspatial is derived from a model that places the perceived spatial position of an audio signal playing from a set of loudspeakers at the center of mass of those loudspeakers' positions weighted by their associated activating gains gi (elements of the vector g): <MAT>.

Equation <NUM> is then manipulated into a spatial cost representing the squared error between the desired audio position and that produced by the activated loudspeakers: <MAT>.

With FV, the spatial term of the cost function is defined differently. There the goal is to produce a binaural response b corresponding to the audio object position o at the left and right ears of the listener. Conceptually, b is a 2x1 vector of filters (one filter for each ear) but is more conveniently treated as a 2x1 vector of complex values at a particular frequency. Proceeding with this representation at a particular frequency, the desired binaural response may be retrieved from a set of HRTFs index by object position: <MAT>.

At the same time, the 2x1 binaural response e produced at the listener's ears by the loudspeakers is modelled as a 2xM acoustic transmission matrix H multiplied with the Mx1 vector g of complex speaker activation values: <MAT>.

The acoustic transmission matrix H is modelled based on the set of loudspeaker positions {si} with respect to the listener position. Finally, the spatial component of the cost function is defined as the squared error between the desired binaural response (Equation <NUM>) and that produced by the loudspeakers (Equation <NUM>): <MAT>.

Conveniently, the spatial term of the cost function for CMAP and FV defined in Equations <NUM> and <NUM> can both be rearranged into a matrix quadratic as a function of speaker activations g: <MAT> where A is an M x M square matrix, B is a <NUM> x M vector, and C is a scalar. The matrix A is of rank <NUM>, and therefore when M > <NUM> there exist an infinite number of speaker activations g for which the spatial error term equals zero. Introducing the second term of the cost function, Cproximity, removes this indeterminacy and results in a particular solution with perceptually beneficial properties in comparison to the other possible solutions. For both CMAP and FV, Cproximity is constructed such that activation of speakers whose position si is distant from the desired audio signal position o is penalized more than activation of speakers whose position is close to the desired position. This construction yields an optimal set of speaker activations that is sparse, where only speakers in close proximity to the desired audio signal's position are significantly activated, and practically results in a spatial reproduction of the audio signal that is perceptually more robust to listener movement around the set of speakers.

To this end, the second term of the cost function, Cproximity, may be defined as a distance-weighted sum of the absolute values squared of speaker activations. This is represented compactly in matrix form as: <MAT> where D is a diagonal matrix of distance penalties between the desired audio position and each speaker: <MAT>.

The distance penalty function can take on many forms, but the following is a useful parameterization <MAT> where ∥o - si∥ is the Euclidean distance between the desired audio position and speaker position and α and β are tunable parameters. The parameter α indicates the global strength of the penalty; d<NUM> corresponds to the spatial extent of the distance penalty (loudspeakers at a distance around d<NUM> or futher away will be penalized), and β accounts for the abruptness of the onset of the penalty at distance d<NUM>.

Combining the two terms of the cost function defined in Equations <NUM> and 9a yields the overall cost function <MAT> Setting the derivative of this cost function with respect to g equal to zero and solving for g yields the optimal speaker activation solution: <MAT>.

In general, the optimal solution in Equation <NUM> may yield speaker activations that are negative in value. For the CMAP construction of the flexible renderer, such negative activations may not be desirable, and thus Equation (<NUM>) may be minimized subject to all activations remaining positive.

<FIG> and <FIG> are diagrams which illustrate an example set of speaker activations and object rendering positions, given the speaker positions of <NUM>, <NUM>, <NUM>, -<NUM>, and -<NUM> degrees. <FIG> shows the speaker activations which comprise the optimal solution to Equation <NUM> for these particular speaker positions. <FIG> plots the individual speaker positions as orange, purple, green, gold, and blue dots respectively. <FIG> also shows ideal object positions (i.e., positions at which audio objects are to be rendered) for a multitude of possible object angles as green dots and the corresponding actual rendering positions for those objects as red dots, connected to the ideal object positions by dotted black lines.

Claim 1:
A method (<NUM>) of determining a location of a plurality of at least four audio devices in an environment, each audio device configured to detect signals produced by a different audio device of the plurality of audio devices, the method comprising:
for each audio device of the plurality of audio devices, obtaining (<NUM>) direction of arrival, DOA, data based on a detected direction of the signals produced by another audio device of the plurality of audio devices in the environment;
determining (<NUM>) interior angles for each of a plurality of triangles based on the direction of arrival data, each triangle of the plurality of triangles having vertices that correspond with locations of three of the plurality of audio devices;
determining (<NUM>) a side length for each side of each of the triangles based on the interior angles and on the signals produced by the audio devices separated by the side length to be determined, or
determining the side length for each side of each of the triangles based on the interior angles by setting one side length of the triangle to a predetermined value and then determining the other side lengths of the triangle based on the interior angles;
performing (<NUM>) a forward alignment process of aligning each of the plurality of triangles in a first sequence, to produce a forward alignment matrix, wherein the forward alignment process is performed by forcing a side length of each triangle to coincide with a side length of an adjacent triangle and maintaining the interior angles determined for the adjacent triangle;
performing (<NUM>) a reverse alignment process of aligning each of the plurality of triangles, to produce a reverse alignment matrix, wherein the reverse alignment process is performed as the forward alignment process but in a second sequence that is the reverse of the first sequence; and
producing (<NUM>) a final estimate of each audio device location based, at least in part, on values of the forward alignment matrix and values of the reverse alignment matrix.