Patent ID: 12238505

DETAILED DESCRIPTION

Reference is made in detail to embodiments of the invention, which are illustrated in the accompanying drawings. The same reference numbers may be used throughout the drawings to refer to the same or like parts, components, or operations.

FIG.1shows a functional block diagram of the audio system according to an embodiment of the present invention.

The audio system100mainly comprises a host device130and a plurality of speakers. The host device130may control a plurality of speakers while playing audio. The host device130may be a computer host, a barebones system, an embedded system, or a customized digital audio processing device. The host device130comprises a communication circuit136, so that the host device130may communicate with a user device150with wire or wirelessly, which serves as an input channel for audio signals or data.

The user device150may be a mobile phone, a computer, a television stick, a game console, or other audio source providing device, providing music or sound streaming to the host device130through the communication circuit136. Furthermore, the audio system100may use the communication circuit136to work in conjunction with the user device150or other multimedia devices to form a home theater system having both video functions and audio functions. For example, the target space170may further comprise a projection screen, a screen or display (not drawn), controlled by the user device150to display. For example, the user device150may be a headset-mounted virtual reality device. The user180may stand in the target space170and see the picture through the user device150, and the host device130may be controlled by the user device150. The audio is played synchronously with the screen play. The communication circuit136in the embodiment, may be, but is not limited to, a high-definition multimedia interface (HDMI), Sony/Philips Digital Interface Format (SPDIF), wireless LAN modules, Ethernet modules, shortwave RF transceivers, or an evolutionary application of Bluetooth Low Energy (BLE) version 4 or version 5, or the universal serial bus.

The host device130further comprises an audio transmission circuit135for connecting a plurality of speakers, and respectively outputs a plurality of channels of audio to make the speaker play. The host device130controls a plurality of speakers through the audio transmission circuit135, which may be a one-way digital or analog output, or a two-way synchronous communication protocol. The connection between the audio transmission circuit135and each speaker, may be a wired interface, a wireless interface, or a mixture of both applications. The wired interface may be, but is not limited to, a composite audio-visual terminal, a digital transmission interface, or a high-quality multimedia interface. The wireless interface may be, but is not limited to, a wireless area network, a shortwave RF transceiver, or an evolved application of Bluetooth Low Energy Version 4 or Version 5. In a further derived embodiment, since the audio transmission circuit135and the communication circuit136are both interfaces connected to external components in functional positioning, in the derived implementations, they may be combined to become a multifunctional bidirectional transmission interface module. The audio transmission circuit135and the communication circuit136employ a variety of known telecommunication standards to achieve connection and transmission between components, which may increase forward compatibility for the audio system100, and reduce the replacement cost when the component is damaged.

The target space170inFIG.1may be understood as a three-dimensional stereoscopic space for the user180to use the sound system100. Each of the speakers may be allocated at separate locations in the target space170, respectively playing one of a plurality of audio channels. A plurality of speakers around the configuration, can create a surround sound field environment in a target space170. There are many standard specifications defining the number of speakers and how they are configured. For example, a 5.1-channel surround sound system consists of two front-facing speakers, a center speaker, two surround speakers, and a subwoofer, jointing creating a surround sound field space that surrounds a target listening point, and together plays sound to that target listening point. In a 7.1-channel surround sound system, a pair of rear surround speakers are further configured behind the target listening point to provide a more stereo soundstage. In recent years, new specifications such as 5.1.2 channels and 7.2.2 channels have emerged, including more speakers and direction-specific channel configurations, which can achieve more realistic “panoramic sound”, “sky sound” or “floor sound”. In the embodiments, to facilitate the description of the technical features of the sound system100, only the first speaker110and the second speaker120are depicted inFIG.1as representatives. Wherein, the first speaker110receives and plays the first channel audio112provided by the host device130, and the second speaker120receives and plays the second channel audio122provided by the host device130. It must be understood that, in practice, the sound system100of the embodiment is not limited to the application of only two speakers, but may be applied to 2.1 channels, 4.1 channels, 5.1 channels, 7.2 channels or more channel specification configuration. Each speaker in the target space170may have a different audio output specification. For example, some speakers are good at producing bass effects, and some are good at producing mid-to-high notes. The host device130may deploy a sound field of various environmental characteristics in accordance with different speaker specifications in the target space170.

The term “audio channel” elaborated in the specification and the scope of the patent application generally refers to physical channel and logical channel. A logic channel is referred to as audio data streams transmitted within the system, while a physical channel is referred to as the signal source played by a speaker. In the embodiment, the first channel audio signal112and the second channel audio signal122played by respective speakers are physical channels, which may be down-mix results from one or more logical channels. For example, a pair of headphones has only two speakers, but can hear the sound produced by multiple applications at the same time. In other words, the sound effects data of multiple applications can be down-mixed by the system into two physical channels and respectively played as audible sounds through the two speakers. That is, the first channel audio signal112and the second channel audio signal122in the embodiment are not limited to be audio signals of a single logic channel, but may be down-mixed results from a plurality of logic channels synthesized with appropriate ratios.

InFIG.1, the first speaker110and the second speaker120are configured on both sides of a target space170, play sounds toward a target listening spot in the target space170. The target listening spot may be understood as a position of the audio system100where the received playback effect is optimized. In some audio systems, the target listening spot is also called the listening sweet spot. In most cases, the target listening spot is typically located in a specific region of the target space170, such as a center point, axis, tangent plane, or the equal loudness center of a plurality of speakers. InFIG.1, the first location171where the user180is located, represents the target listening spot of the target space170. When the user180moves from the first location171along the movement trajectory173to the second location172, the user180is far away from the first speaker110and gets close to the second speaker120, and therefore the listening effect received by the user180is biased. Conventionally, an audio system is unable to track the movement of the user180to correspondingly adjust the listening effect received at second location172. The embodiments of proposed solutions in the invention will be described in detail later.

On the other hand, the target space170typically contains some ambient object175, such as sofa, table, window, wall, ceiling, or floor. These ambient objects175produce different degrees of interference effects to the sound played by the first speaker110and the second speaker120depending on varied materials, sizes and positions. For example, a cloth sofa or window may absorb sound, and marble floors or walls may reflect sound. In other words, the presence of an ambient object175may affect the listening experience of the first channel audio signal112and the second channel audio signal122at the target listening spot. Conventionally, an audio system is incapable of identifying ambient objects175in the target space170, nor does it have the function to compensate the first channel audio signal112and the second channel audio signal122according to the size, material, position of the ambient objects175. The audio system100of the embodiment is arranged to calculate and eliminate interferences caused by the ambient objects175of the target space170by compensating the first channel audio signal112and the second channel audio signal122. For ease of illustration, only one ambient object175is depicted inFIG.1of the embodiment to illustrate how the audio system100operates. However, it must be understood that the target space170ofFIG.1is not used to limit only one ambient object175. Solutions to the interference of ambient object175will be described in greater detail later.

The host device130of the embodiment further comprises a storage circuit131. The storage circuit131may comprise a non-volatile memory device for storing an operating system, and application software or firmware required for operating the host device130. The storage circuit131may also contain a volatile memory device for use as an operational storage for the control circuit132. The host device130of the embodiment further comprises a control circuit132. The control circuit132may be a central processing unit, a digital signal processor, or a microcontroller. The control circuit132may read from the memory circuit131a pre-stored operating system, software, or firmware, to control the host device130, the first speaker110and the second speaker120to perform audio playback operations. Furthermore, the host device130of the embodiment uses the control circuit132to perform a series of sound field compensation operations to dynamically optimize the playback effect, to solve the shortcomings that the conventional audio system cannot overcome.

To dynamically optimize the playback effect at the target listening spot, the audio system100of the embodiment comprises a sensor circuit140, arranged to dynamically sense a target space170and respectively generate field context information. The sensor circuit140may be a component located outside the host device130and coupled to the host device130. The sensor circuit140may be a camera610, an infrared sensor620, a wireless detector630, or a combination of one or more thereof. The field context information captured by the sensor circuit140may comprise diverse types of information depending on different implementations of the sensor circuit140. For example, the field context information may include images of the user and ambient objects, pictures, thermal imaging, radio wave imaging, or a combination of one or more thereof. In one embodiment, the sensor circuit140is disposed around the target space170. To be noted, althoughFIG.1only depicts a sensor circuit140, the audio system100in practice, may comprise multiple sets of sensor circuits140, respectively allocated in various positions around the target space170to obtain sound field context information with better accuracy.

The host device130of the embodiment may further comprise a recognizer circuit134coupled to the sensor circuit140. The recognizer circuit134may identify from the field context information the key information affecting the sound field, allowing the control circuit132to dynamically adjust the first channel audio signal112and the second channel audio signal122played from the first speaker110and the second speaker120. For example, the recognizer circuit134may identify a user from the field context information and determine the user's position in the target space. Since the field context information provided by the sensor circuit140may comprise a variety of combinations in diverse types, the recognizer circuit134may be correspondingly implemented to adapt respective identification technology or solutions. For example, when the field context information is an image, the recognizer circuit134may employ artificial intelligence (AI) identification technology to distinguish the user in an image. Through the application of artificial intelligence, after identifying the user in the image134, the user head, face, and even ear position can be further located. If the sensor circuit140is capable of providing diversified information such as a three-dimensional image with a spatial depth, infrared thermal imaging, or wireless signals, the recognizer circuit134can be leveraged to render more accurate identification results.

To calculate the degree of interference caused by the ambient object175to the sound field environment, the host device130requires the spatial configuration information and acoustic attribute information of the ambient object175. The spatial configuration information may include the size of the ambient object175, position, shape, and various appearance characteristics. The acoustic attribute information can include material-related characteristics such as acoustic absorption rate, reflectivity of sound, and resonance frequency of sound. In one embodiment, when the recognizer circuit134identifies the field context information, the recognizer circuit134may further identify spatial configuration information of the ambient object175in the target space170out of the field context information, and accordingly lookup the acoustic attribute information. To identify ambient objects175, an object database is required. In one embodiment, the storage circuit131in the host device130may also be used to store an object database. The object database may contain information about various appearance characteristics used to identify the ambient object175, as well as various acoustic attribute information corresponding to each ambient object175. For example, when the host device130needs to calculate the degree of interferences to the sound field environment caused by an ambient object175, the recognizer circuit134first determines the object name of the ambient object175, and then the host device130reads the storage circuit131to find the acoustic absorption rate and reflectivity of sound corresponding to the ambient object175.

In practice, the recognizer circuit134may be a customized processor chip, performing the artificial intelligence identification function cooperated with the existing operating system, software or firmware stored in the storage circuit131. The recognizer circuit134may also be implemented as one of the cores or threads in the control circuit132, performing predetermined artificial intelligence software products in the storage circuit131to achieve the identification functions. The recognizer circuit134may also be a memory module storing a particular artificial intelligence software product, and the identification function is completed by execution of the product in the control circuit132.

The user interface circuit133in the host device130can be used by the user to control the operation of the host device130. The user interface circuit133may include a display screen, button, turntable, or touch screen, allowing the user to perform basic audio system100control functions, such as adjusting the volume, playback, fast-forward, and backwards. In one embodiment, the control circuit132may also perform a configuration program through the user interface circuit133for the user to set various sound field scenarios, or the target space170in the ambient object175spatial configuration information to the host device130. For example, in the configuration program, the control circuit132uses the user interface circuit133to receive object configuration data entered by the user, such as object name, type, size and location of one or more ambient objects175. After obtaining the spatial configuration information in the control circuit132, the corresponding acoustic absorption rate and reflectivity of sound are looked up from the object database stored in the storage circuit131, so that the subsequent sound field compensation operation is performed. In a further derived embodiment, the user interface circuit133may also be provided by the user device150. The user may operate the configuration program using the user device150, and finally the user device150transmits the configured settings to the control circuit132through the communication circuit136.

The host device130may also be connected to a remote database160via a communication circuit136. In a further embodiment, the object database implemented by the storage circuit131may also be implemented by a remote database160. When the host device130needs to calculate the degree of interference caused by an ambient object175to the sound field environment, the field context information may first be obtained by the recognizer circuit134to be analyzed, and thereafter, the communication circuit136is used to access the remote database160, allowing an ambient object175to be found that matches the characteristics of the ambient object175, and thus the sound field attribute information of the ambient object175is obtained. The remote database160may be a server located in the cloud or other system, coupled to the host device130via wired or wireless two-way network communication technology. In addition to providing a lookup function, the remote database160may also accept upload of updated data to continuously expand the database content. For example, the host device130may utilize a Structured Query Language (SQL) to communicate with the remote database160.

Based on the system architecture ofFIG.1, the audio system100proposed in the present application may achieve at least the following technical effects. Firstly, the audio system100allows dynamic tracking of the user's position to be a target listening spot. The audio system100may also dynamically obtain the spatial configuration information of the ambient object as a basis for optimizing the soundstage effect. Furthermore, the audio system100dynamically compensates the speaker output according to the user position and ambient object spatial configuration information, to eliminate object interferences and optimize the listening experience on the target listening spot. In an embodiment, the dynamic user position tracking may employ multiple technical solutions such as cameras, infrared sensors, or wireless positioning. The implementation of spatial configuration information acquisition of the ambient objects175may be automated or manual. For example, the audio system100may use a camera to capture images and perform artificial intelligence identification, or through a configuration program to manually input the environmental conditions of the scene. Embodiments of the speaker output compensation may be implemented based on several different algorithms. For example, this specification of the invention provides a channel-based algorithm and an object-based algorithm.

An embodiment about how the audio system100dynamically tracks the user position, obtains a sound field environment configuration using a camera, and compensates the speaker output with channel-based compensation operation, is illustrated inFIG.2as described below.

FIG.2is a flowchart of a dynamic sound optimization method according to an embodiment of the present invention.

In the flowchart ofFIG.2, the processes located in a column corresponding to a particular apparatus, described the processes executed by the particular apparatus. For example, the processes marked in the “sensor circuit” column, are processes performed by the sensor circuit140. The processes marked in the “host device” column are related to processes performed by the host device130. The processes marked in the “speaker” column, are processes conducted by the first speaker110and/or the second speaker120. The other parts of the figures are described analogously in the same planning, so are the other flowcharts below.

In operation202, the sensor circuit140dynamically senses the target space170to generate field context information. In the embodiment, the field context information may comprise optical, thermal, or electromagnetic wave information in the target space170. For example, the sensor circuit140may comprise a camera, continuously capturing a video of the target space170in a video recording manner, or periodically snapshotting the target space170to acquire still photos of the target space170. In another embodiment, the sensor circuit140may further comprise an infrared sensor, arranged to capture a thermal image in the target space. The thermal image generated by infrared sensors, may provide more than just depth information of the space. Since the infrared sensor is extremely sensitive to temperature changes, it is particularly suitable for tracking the user's position. In another embodiment, the sensor circuit140may further comprise a wireless detector, disposed in the target space170for detecting a wireless signal of an electronic device. When a user is holding an electronic device, the wireless detector can detect the beacon time difference of the electronic device or the strength of the wireless signal, as an auxiliary means of tracking the user's location. The electronic device may be the user's own mobile phone, a special beacon generator, a virtual reality headset, a game handle, or a remote control of the audio system100. It will be appreciated that the embodiment does not limit the number of sensor circuits140, nor does it limit the use of only one sensing scheme at a time. For example, the audio system100of the embodiment may employ a plurality of sensor circuits140to work together from various locations, or simultaneously employ one or more cameras, infrared sensors and wireless detectors. Thus, the host device130is allowed to obtain more comprehensive field context information to acquire subsequent identification results more accurately.

In operation204, the sensor circuit140transmits the acquired field context information to the host device130. The sensor circuit140may transmit data in a continuous manner, such as video streaming, or in a periodic manner for still image data. The duty cycle of the transmission action performed by the sensor circuit140may be adaptively adjusted according to the amount of information of the field context information, the tracking accuracy requirements, and the computing power of the host device130. The sensor circuit140and the host device130may be connected via a dedicated cable, or via a communication circuit136. In a further derived embodiment, the sensor circuit140may share the audio transmission circuit135with the speaker, whereby the field context information is transmitted to the host device130through the audio transmission circuit135.

In the operation206, the host device130determines the user position according to the field context information received from the sensor circuit140. The recognizer circuit134in the host device130may perform identification operations on the field context information, such as applications of artificial intelligence. As the sensor circuit140may implement different sensing technology, the recognizer circuit134may adapt corresponding identification algorithms. It is appreciated that the target space170and the user position may be represented in two-dimensional space or three-dimensional space. If the audio system100is implemented with only a single sensor circuit140, the position information of at least two-dimensional space can be perceived. If the audio system100is implemented with increased number of sensor circuits140or hybrid sensing schemes, the depth information in the three-dimensional space can be obtained for better recognition of the user position or the user's head position. In one embodiment, the recognizer circuit134may dynamically identify the user's head position, face orientation, or ear position according to the camera captures the sound ambient image. In another embodiment, the recognizer circuit134may analyze the movement trajectory of the thermal image generated by the infrared sensor to dynamically determine the position of the user180. For another example, the recognizer circuit134may dynamically locate a coordinate value of the electronic device in the target space170according to the characteristics of the wireless signal detected by the wireless detector. By virtue of the coordinate value, the control circuit132may further speculate the position of the user's ear.

In operation208, when the recognizer circuit134in the host device130analyzes the user position, the control circuit132in the host device130dynamically assigns the user position to be a target listening spot. To facilitate the description of subsequent embodiments, the target space170is described herein as a two-dimensional coordinate space or a three-dimensional coordinate space, and the target listening spot may be represented as a coordinate value in the target space170. With the layout arrangements of the speakers, the range of the target listening spot may not be limited to a single point, which can also be a surface, or a three-dimensional area with predetermined length, width, and height. For example, after the recognizer circuit134analyzes the user's head position or ear position, the control circuit132may use the head position or ear position to determine the target listening spot. The control circuit132therefore performs subsequent compensation operations so that the playback effect obtained by the target listening spot is not affected by the user's movement. In practice, the control circuit132compensates for the listening experience at the target listening spot by adjusting the first channel audio signal112and the second channel audio signal122. It is understandable that operation208may be dynamically executed when the user's location is changed. Thus, operation208is not limited to be performed in the order in whichFIG.2is plotted. That is, the target listening spot can be updated in real time as the user's position changes. Further adjustment operations will be described below.

In operation210, the recognizer circuit134in the host device130performs further recognition on the field context information provided by the sensor circuit140to obtain spatial configuration information of the ambient object in the target space170. In other words, the field context information acquired by the sensor circuit140, is not only used to determine the user position, but also useful for determining various ambient objects175presented in the target space170. In one embodiment, when the camera in the sensor circuit140captures an ambient image of the target space170, the recognizer circuit134analyzes the ambient image. One or more ambient objects175are thereby identified from the target space170, along with the spatial configuration information of these ambient objects175. The spatial configuration information comprises size, position, shape, appearance characteristics of the ambient object175. The recognizer circuit134may also determine the acoustic attribute information of each ambient object175through artificial intelligence calculation or database lookups, including the acoustic absorption rate and reflectivity of sound. In a further derived embodiment, the recognizer circuit134may also determine the application scenario of the target space170according to the ambient image. The application scenario can be theater, living room, bathroom, outdoor, and so on. If the host device130is provided with the application scenario information related to the target space170, the ambient object175in the target space170can be identified in shorter time with lower mis-judgement rate. Relevant embodiments will be illustrated inFIG.9.

In operation212, the control circuit132in the host device130may calculate how much the playback effect at the target listening spot is affected by the ambient object. The playback effect of a speaker at a target listening spot can be defined as the equal loudness or sound pressure level (SPL) received at the target listening spot. In the ISO226 standard, an equal response curve (Fletcher-Munson Curve) is defined, indicating that the equal loudness perceived by the user in different sub-bands actually corresponds to different sound pressure levels. In one embodiment, the control circuit132may employ an equal loudness contour as a standard reference for the playback effect, to calculate the sound pressure level received at the target listening spot under various scenarios. The control circuit132may use the spatial configuration information of the ambient object175and the acoustic attribute information to evaluate the interference caused by the ambient object175at the target listening spot, so as to further determine how to eliminate the interference. The spatial configuration information of the ambient object175and the influence of the attribute information may correspond to a variety of scenarios. For example, when the volume of the ambient object175gets larger, the interference coefficient respective to the target listening spot gets larger. Whether the ambient object175obstacles the user180and the speaker, is also a factor determining the playback effects. The ambient object175, depending on the material, may absorb sound or bounce sound. Therefore, the control circuit132is required to accordingly select different parameters or formulas based on the acoustic attribute information, so as to calculate the degree of influence on the playback effects.

In operation214, the control circuit132in the host device130performs a channel-based compensation operation to respectively calculate the output compensation value for the channel audio played in each speaker. The channel-based compensation operation is calculated separately on a per-channel audio basis when determining the playback effects at the target listening spot. Taking a first speaker110among a plurality of speakers for example, which plays a first channel audio signal112. When the first channel audio signal112is transmitted through the air to the target listening spot, it may be interfered with by an ambient object175and lose energy. The change of the position of the target listening spot also affects the reception of the sound pressure level generated by the first channel audio signal112. With the channel-based compensation operation, the control circuit132may calculate the amount of change in the sound pressure level the first channel audio signal112imposes at the target listening spot. The control circuit132of the embodiment adds an output compensation value for the first channel audio signal112to cancel the amount of change in the sound pressure level, so that the first channel audio signal112perceived at the target listening spot is restored to the state before being affected. In other words, the output compensation value has the same value as the amount of change in sound pressure level, but with opposite polarity.

In operation216, the control circuit132adjusts and outputs the channel audio signal to the speaker according to the output compensation value. Since the adjusted channel audio signal has cancelled the influence caused by the user180movement in the target space170and the interference caused by the ambient object175, the listening experience of the user180remains consistent. Taking the first speaker110and the second speaker120in the target space170as an example, the control circuit132calculates and adjusts the sound pressure levels of different sub-bands in the first channel audio signal112and the second channel audio signal122, thereby mitigating the equal loudness deviation experienced by the user180due to movement. On the other hand, the control circuit132compensates the first channel audio signal112and the second channel audio signal122respectively according to the amount of sound pressure level change imposed at the target listening spot calculated from the position, size, and acoustic attribute information of the ambient object175.

In operation218, each speaker receives channel audio signal from the host device130correspondingly through the audio transmission circuit135. Taking the first speaker110and the second speaker120in the target space170as an example, the control circuit132outputs the first channel audio signal112and the second channel audio signal122to the corresponding first speaker110and the second speaker120through the audio transmission circuit135, respectively. Thus, the first speaker110and the second speaker120correspondingly play the adjusted first channel audio signal112and the second channel audio signal122, so that the user180located at the target listening spot experiences an optimized listening effect. For ease of illustration, the embodiment of the target space170ofFIG.1only depicts two speakers and an ambient object175. However, it is understandable that in practice, the host device130may contain more than two speakers, and the number of ambient objects175is not limited to one. In a further derived embodiment, each speaker is specialized at playing different range of frequencies. For example, some speakers are mid-tweeters, and some speakers are subwoofers. When the control circuit132adjusts the channel audio signal, it can be arranged to further adjust corresponding outputs to the first channel audio signal112and the second channel audio signal122according to the characteristics of different speakers.

The following description employsFIG.3to illustrate an embodiment of an audio system100that dynamically tracks the user position, uses the camera to obtain a sound field environment configuration, and performs an object-based compensation operation on the speaker outputs.

FIG.3is a flowchart of a dynamic sound optimization method according to an embodiment of the present invention.

In the flowchart ofFIG.3, the processes located in a column corresponding to a particular apparatus, described the processes executed by the particular apparatus. For example, the processes marked in the “Sensor Circuit” column, are processes performed by the sensor circuit140. The processes marked in the “host device” column, are processes performed by the host device130. The processes marked in the “speaker” column, are processes conducted by the first speaker110and/or the second speaker120. The other parts of the figures are described analogously in the same planning, so are the other flowcharts below.

Processes202,204,206,208and210inFIG.3are the same as the previous embodiments, and therefore not repeated herein for simplism of the specification.

When the audio system100of the embodiment completes operation210, the control circuit132has tracked the position of the user180and assigned it as the target listening spot, wherein the configuration information of one or more ambient objects175in the target space170are also obtained. The object-based compensation operation is then explained in the subsequent processes, whereby channel audio signals for each of the speakers are adjusted.

The object-based acoustic system is originated from the audio mixing technology that creates virtual reality, wherein fancy audio object effects such as movement can be simulated by just a limited number of physical speakers. Some of the existing software products, such as Dolby Atmos, Spatial Audio Workstation, or DSpatial Reality, are in the category of object-based acoustic systems. A user can define the movement trajectory of a source object in a virtual space through a human-machine interface. The object-based system can use physical speakers to simulate the sound effect of the source object in this virtual space, allowing the user at the target listening spot to realistically feel the source object moving through space.

The object-based acoustic system is based on a large number of acoustic parameter matrix calculations. Each source object may comprise metadata that describes characteristics of the source object, including type, position, size (length, width, height), divergence, and so on. After the matrix calculations in the object-based operation, the sound of a source object will be assigned to one or more speakers and played together, and each speaker may relatively play a portion of the sound of the source object. In other words, the matrix calculations in the object-based operation can use multiple speakers to simulate spatial effects of a sole source object. The embodiment ofFIG.3proposes an object-based compensation operation based on the object-based acoustic system to solve the problem of conventional playback effect.

In operation312, the control circuit132in the host device130creates a compensatory audio object according to the ambient object175for use in the object-based acoustic system. In practice, the control circuit132first maps the target space170to an object-based space of virtual reality, and then creates a compensatory audio object in the object-based space according to the ambient object175, for generating a sound source effect that cancels the influences of the ambient object175. From the perspective of a user180located at the target listening spot, the presence of the ambient object175is analogous to a sound source object. In practical applications, the ambient object175may reflect the sound emitted by a speaker to the target listening spot. On the other hand, the ambient object175may also block or absorb a portion of the sound emitted by a speaker to the target listening spot, causing listen experience degradation. In other words, the control circuit132of the embodiment analogizes the ambient object as a sound source object, and creates a negative sound source object with the opposite sound source effect in the object-based space, as a means of counteracting the interference. The acoustic effect described in the embodiment may be referred to as sound pressure level, equal loudness, or gain value generated from the perspective of the target listening spot.

In operation314, the host device130substitutes the compensatory audio object into the object-based compensation operation to render the channel audio signals. The object-based compensation operation may employ an object-based matrix calculation module in the existing object-based acoustic products to perform a large number of matrix operations related to acoustic interaction according to the metadata of the sound source object. For example, the metadata of the compensatory audio object comprises coordinate, size, reflectivity of sound, and acoustic absorption rate of the ambient object175. The control circuit132performs an object-based compensation operation on the target listening spot according to the metadata to render the first channel audio signal112and the second channel audio signal122optimized for the target listening spot, such that the interference the ambient object175imposed on the target listening spot is cancelled.

In one embodiment, the object-based compensation operation is conducted on a plurality of sub-bands. Due to the nature of sound transmission, the sound pressure level in different sub-bands causes different equal loudness. Taking the impact from the first channel audio signal112generated by the first speaker110on the ambient object175as an example, the control circuit132of the embodiment may calculate based on the coordinate, the size, and the reflectivity of sound and acoustic absorption rate of the ambient object175, a plurality of sound source effects passively generated by the ambient object175respectively responsive to a plurality of sub-bands of the first channel audio signal112. Thereafter, the control circuit132creates a compensatory audio object according to the calculated sound source effects. In the embodiment, the compensatory audio object is correspondingly created according to the ambient object175, wherein the metadata thereof comprises the same coordinate, size, and reflectivity of sound and acoustic absorption rate as the ambient object175, which produces a sound source effect opposite to that of the ambient object175.

The known human ear audible range is between 20 Hz (Hz) to 20000 Hz. The embodiment may divide the human ear audible range into a plurality of sub-band intervals to compensate separately. The bandwidth of each sub-band can be an exponential range. For example, an exponential range based on 10 can divide the sound signal into multiple sub-band ranges such as 10 Hz to 100 Hz, 100 Hz to 1000 Hz, 1000 Hz to 10000 Hz, and etc. In other embodiments, the division of bands may also be based on the needs of the fineness of the playback quality, such as a 2-based or 4-based exponential index. In the field of audio processing, the concept of multi-band subdivision is known in the equalizer technology, which will not be explained in depth herein.

After the control circuit132obtains the negative acoustic effect of the compensatory audio object, the object-based compensation operation is performed. The negative acoustic effect is correspondingly mixed into the first channel audio signal112and the second channel audio signal122according to a predetermined proportion determined by a mix operation, and thereby cancelling the interference the ambient object175imposed on the target listening spot. Regarding the object-based compensation operation, further descriptions will be provided in detail in the embodiments ofFIGS.11to13.

In operation316, the host device130outputs the first channel audio signal112and the second channel audio signal122correspondingly to the first speaker110and the second speaker120according to the operation result of operation314. Process316ofFIG.3is different from operation216of the embodiment ofFIG.2. The embodiment ofFIG.2calculates a compensation value for the existing channel audio signal to adjust the existing channel audio signal. The control circuit132in the object-based compensation operation, on the other hand, calculates the corresponding channel audio signal of each speaker at a time directly according to all the metadata. The object-based compensation operation synthesizes the components required for compensation into the channel audio signal in a form of a compensatory audio object. In other words, because the channel audio signal contains a compensated audio signal emitted by the compensatory audio object, the user180does not feel the impact of the presence of the ambient object175at the target listening spot.

It can be seen from operation316, the object-based compensation operation converts the target listening spot and ambient object into the metadata of the object-based acoustic system, and then creates a compensatory audio object to simplify the operation processes for elimination of interferences and optimization of the playback effects. It should be understood that the audio system100of the embodiment may utilize the sensor circuit140to continuously or periodically track the position of the user180, and dynamically update the target listening spot. The object-based compensation operation performed by the control circuit132may also be able to simultaneously update metadata of the relative position of the target listening spot170as the target listening spot is moved.

InFIG.3, operation218is analogous to the previous embodiment, and therefore is not repeated herein for simplicity of the specification.

The following description employsFIG.4to illustrate an embodiment of an audio system100that dynamically tracks the user position, runs a configuration program to obtain the sound field environment configuration, and compensates the speaker output using the channel-based compensation operation.

FIG.4is a flowchart of a dynamic sound optimization method according to an embodiment of the present invention.

In the flowchart ofFIG.4, the processes located in a column corresponding to a particular apparatus, described the processes executed by the particular apparatus. For example, the processes marked in the “sensor circuit” column, are processes performed by the sensor circuit140. The processes marked in the “host device” column are related to processes performed by the host device130. The processes marked in the “speaker” column, are processes conducted by the first speaker110and/or the second speaker120. The other parts of the figures are described analogously in the same planning, so are the other flowcharts below.

Processes202,204,206, and208inFIG.4are the same as the previous embodiments, and therefore are not repeated herein for simplicity of the specification.

When the audio system100of the embodiment completes operation210, the control circuit132has tracked the position of the user180and assigned it as the target listening spot, wherein the configuration information of one or more ambient objects175in the target space170are also obtained. The object-based compensation operation is then explained in the subsequent processes, whereby channel audio signals for each of the speakers are adjusted.

In order to eliminate the interference in the sound field environment, the audio system100needs to obtain spatial configuration information related to various ambient objects175in the target space170.

In operation410, the control circuit132in the host device130may run a configuration program to obtain spatial configuration information of one or more ambient objects175in the target space170. In the previous embodiment, the host device130uses a sensor circuit140to capture the field context information to automatically identify the spatial configuration information of an ambient object175. When the configuration program is executed, the host device130may utilize a user interface circuit133to interact with the user, allowing the user to manually enter spatial configuration information of the ambient object175. The user interface circuit133may provide a screen and an input method, allowing the user to define the spatial configuration information of various objects in the target space170in a two-dimensional plane or a three-dimensional diagram. The spatial configuration information of the ambient object175, may include relative position, size, name, and material type of the ambient object175in the target space170. In a further derived embodiment, the user180may inform the host device130through the user interface circuit133an adequate application scenario for the current target space170. In different application scenarios, such as open outdoor spaces, theater spaces, or bathrooms, the types of frequently seen ambient objects175are different, and the soundstage atmosphere felt by users is also different. It is one of the essential functions for an audio system100to optimize the soundstage atmosphere in different application scenarios.

Different material types feature different acoustic properties. When the host device130runs the configuration program, the host device130further queries an object database according to the object name or material type input by users, to obtain acoustic attribute information of the ambient object175, such as the acoustic absorption rate or reflectivity of sound. Thus, in the subsequent operation212, the host device130is allowed to calculate the degree to which the playback effect of each speaker on the target listening spot is affected by the ambient object175, according to the aforementioned spatial configuration information and acoustic attribute information. In a further derived embodiment, the host device130may be able to prioritize the usage of one of a plurality of object databases based on the adequate application scenario of the target space170, so as to accelerate the identification of the ambient object175in the target space170. Relevant embodiments will be illustrated inFIG.9.

Processes212,214,216, and218inFIG.4are the same as the previous embodiment, and therefore are not repeated herein for simplicity of the specification.

The embodiment ofFIG.4illustrates that in addition to dynamically tracking the user position, the audio system100also allows the user180to setup the spatial configuration information of the ambient object175in the target space170through a configuration program. The configuration program provides a manual input interface to compensate for the lack of identification capabilities. In addition to assisting the host device130to make better decision with the auxiliary inputs, the user also has the opportunity to deliberately customize different application scenarios according to their own preferences, or deliberately set the imaginary virtual audio object to change the playback effect. The host device130performs the channel-based compensation operation to calculate corresponding output compensation values of each speaker according to the spatial configuration information of the ambient object175in the target space170.

The following embodiment usesFIG.5to illustrate an embodiment of the audio system100that dynamically tracks the user position, runs the configuration program to obtain the sound field environment configuration, and compensates the speaker output with the object-based compensation operation.

FIG.5is a flowchart of a dynamic sound optimization method according to an embodiment of the present invention.

In the flowchart ofFIG.5, the processes located in a column corresponding to a particular apparatus, described the processes executed by the particular apparatus. For example, the processes marked in the “sensor circuit” column, are processes performed by the sensor circuit140. The processes marked in the “host device” column are related to processes performed by the host device130. The processes marked in the “speaker” column, are processes conducted by the first speaker110and/or the second speaker120. The other parts of the figures are described analogously in the same planning, so are the other flowcharts below.

Processes202,204,206,208, and210inFIG.5are the same as the previous embodiments, and therefore not repeated herein for simplicity of the specification.

The embodiment ofFIG.5is analogous to the embodiment inFIG.4, wherein operation410is processed to eliminate interference in the sound field environment.

In operation410, the host device130may run a configuration program to obtain spatial configuration information of one or more ambient objects175in the target space170. In the embodiment ofFIG.4, it is illustrated that the host device130may receive the spatial configuration information of the ambient object175manually entered by the user through a user interface circuit133. In a further derivative embodiment, the host device130may also use a communication circuit136to receive spatial configuration information transmitted by the user device150or other devices. For example, the user device150may be a mobile phone, running an application to provide functions similar to the user interface circuit133. The application allows the user to define the scope and size of the target space170, the position of each speaker relative to the target space170, the position, size, name and type of various ambient objects175, even the location where the user180itself is located. The application may also communicate with the control circuit132through the communication circuit136to perform various playback operations, such as playback, pause, fast forwarding, adjusting the volume and the like. Furthermore, the user may configure the application scenario of the target space170through the user interface circuit133, allowing the host device130to produce a diversified playback effect on the target space170.

In a further derived embodiment, the user device150coupled by the host device130may be a virtual reality device or a game console. The user device150generates a source signal for the host device130to play. The source signal may contain virtual objects that move around in a virtual reality space, such as an airplane or a fire-breathing dragon. The user device150may transmit metadata of these virtual objects to the host device130, which then turn into a part of the spatial configuration information of the ambient objects in the target space170. In other words, the host device130may employ an object-based acoustic system to process virtual objects and physical objects without discrimination. Through an object-based compensation operation, the host device130may allow the user to feel that there is a virtual object in the target space170, and may also allow the user not to feel the interference of a physical object in the target space170. Regarding the implementation of the object-based compensation operation, further descriptions are provided in the embodiments ofFIGS.11to13.

When the audio system100of the embodiment completes operation410, the control circuit132has tracked the position of the user180to assign the target listening spot, and also obtained the spatial configuration information of one or more ambient objects175in the target space170. Then in processes312to316, the host device130uses an object-based algorithm to adjust the channel audio signal of each speaker. Since processes312to316, and processes218are the same as the previous embodiments, details are omitted herein for simplicity of the specification.

The embodiment ofFIG.5illustrates that in addition to dynamically tracking the user position, the audio system100also allows the user180to set spatial configuration information of ambient objects175in the target space170through a configuration program. The configuration program can be integrated with existing virtual reality technology to receive spatial configuration information for virtual objects. The audio system100converts the physical ambient objects175and virtual objects into metadata of a consistent format, and then applies all the metadata to the object-based matrix calculation module in the existing object-based acoustic system to perform the object-based compensation operation. Thus, the control circuit132does not need to develop additional computing modules for different objects, which can reduce costs and improve execution efficiency.

The following description usesFIG.6to illustrates several embodiments of the sensor circuit, and the channel-based compensation algorithm.

FIG.6is a schematic view of a target space600of the present invention, for illustrating an embodiment for calculating the amount of the audio adjustment according to the position of the optimal listening spot.

The audio system of the present application100uses a sensor circuit140to dynamically sense the target space600and generate the field context information. The field context information mainly comprises the location of the user180, and may also contain spatial configuration information of the ambient object175. The technical solutions for dynamic sensing can be available with diverse options. For example, the sensor circuit140may be a combination of one or more of the cameras610, the infrared sensor620, the wireless detector630, respectively, configured in different locations around the target space600, providing field context information comprising spatial depth information that helps the recognizer circuit134and control circuit132in the host device130to track the position of the user180more efficiently. Thus, the recognizer circuit134uses the field context information provided by the sensor circuit140to identify not only the user180position, but also the face facing direction, ear position, and even gestures or body posture. As a result, the control factors applicable for adjusting the sound field are enriched. For example, focus detection, sleep detection, gesture control, and etc.

In the target space600ofFIG.6, a first speaker110and a second speaker120are arranged. The channel-based compensation operation calculates the output compensation value for each speaker individually. In a default scenario, the target listening spot is located in the center of the target space600, i.e., the first location601inFIG.6. The distance from the first location601to the first speaker110and the second speaker120is identically R1. At this time, the first channel audio signal112and the second channel audio signal122played by the first speaker110and the first channel audio signal112are also in a default state wherein no compensation process is required for the position.

When the user180moves from the first location601along the movement trajectory173to the second location602, the sensor circuit140detects a new position of the user180, and the target listening spot of the audio system100is assigned to the second location602. Meanwhile, the distance between the user180and the first speaker110is changed to R2, and the distance between the user180and the second speaker120is changed to R2′. From the perspective of the user180, the first speaker110becomes farther away, so the first channel audio signal112received is attenuated by distance. Conversely, the second speaker120became closer, and the received second channel audio signal122is enhanced. In other words, the intensity of the first channel audio signal112and the second channel audio signal122received on the second location602are imbalanced. The embodiment utilizes the channel-based algorithm, so that the listening experience received at the second location602is reduced to the same default state as the first location601. In other words, the control circuit132compensates for the first channel audio signal112and the second channel audio signal122output from the first speaker110and the second speaker120, to cancel the listening effect deviation caused by the movement of the user180. The target space600shown inFIG.6is not limited to a multi-speaker environment that is only suitable for horizontal configuration. In a 3D soundstage environment with an upper and lower speaker, the problem of distance deviation may also occur. For example, if a user changes from a standing position to a seated position, the user actually moves away from the upper speaker and gets close to the lower speaker.

To obtain a better compensation effect, the embodiment uses equal loudness as a calculation standard. For example, the embodiment may employ the equal loudness curve defined by the ISO226; 2003 standard to calculate the sound pressure level that needs to be compensated at the target listening spot. Each audio channel is divided into multiple sub-bands for processing separately. In addition, different ranges of the distance between the user180and the speaker correspond to different sound formula. Since the equal loudness curve defines a linear relationship between the equal loudness and the sound pressure level, the equal loudness and the “gain value” in decibels have a linear correspondence. Thus, the measurement unit for performing the adjustment in the embodiment is not limited to equal loudness, sound pressure level, or gain value.

In the audio system100, the space where sound is transmitted by air vibration is called a sound field. In a closed room, due to the presence of reflections, the sound field can be distinguished into many types: (1) near field: when the user180is located relatively close to the sound source, the physical effects of the sound source (such as pressure, displacement, vibration) will enhance the sound; (2) reverberant field: sound is reflected by objects to cause wave superposition effect; (3) free field: a sound field that is not interfered with by the aforementioned near and reflected sound fields. The above reflected and free sound fields can be collectively referred to as a far field.

In many of today's audio systems, the near and far fields are defined differently. For example, if R is the distance (meter, m) between a speaker and a user, L is the width of the speaker (m), and λ is the representative wavelength (m) of a sub-band signal, then the desirable conditions for the far field include the following types:
R>>λ/2π  (1)
R>>L(2)
R>>πL2/2λ  (3)

Take the first speaker110ofFIG.1as an example. When the distance from the target listening spot to the first speaker110is greater than the wavelength of the sub-band signal or a specific proportion of the size of the first speaker110, the audio system100determines that the sound field type is a distant sound field. When the distance of the target listening spot to the first speaker110is less than the wavelength of the sub-band signal or the specific proportion of the size of the first speaker110, the sound field type is deemed to be a near field. In a simpler practice, the audio system100may take a twice value (2λ) of the wavelength corresponding to a central frequency of a sub-band signal to be a demarcation point between the far field and the near field.

In the far field, a relationship between the sound pressure level change and the distance change when the user180receives a sub-band signal from the speaker is as follows:
SPL2=SPL1−20 log10(R2/R1)  (4)

Wherein, SPL2is the sound pressure level of the sub-band signal received by the new position, SPL1is the sound pressure level of the sub-band signal received at the original position, R2is the distance between the new position and the speaker, R1is the distance between the original position and the speaker.

From Equation (4), it can be seen that the difference between SPL1and SPL2is the amount of sound pressure that the speaker that needs to be compensated.
SPL2′=SPL2+20 log10(R2/R1)=SPL1  (5)

Wherein, SPL2′ is the sound pressure level of the sub-band signal received by the new position after compensation. From formula (5) it can be seen that the embodiment is to compensate for the changed portion.

In the near field, the relationship between the distance change when the user180receives the sound pressure level of the sub-band signal received from the speaker and is as follows:
SPL2=SPL1−10 log10(R2/R1)  (6)
SPL2′=SPL2+20 log10(R2/R1)=SPL1  (7)

From formulas (6) and (7), it can be seen that the slope of sound attenuation in the near field is gentler than that of the far field, while the other calculation logics are analogous.

It is understandable that the above formula may have exceptions when encountering some exceptional circumstances. For example, when the user180moves from the first location601to the second location602and gets close to the second speaker120, the distance between the user180and the second speaker120becomes smaller from R1to R2′, which leads to a negative result in equation (7). However, the sub-band signal output of the second speaker120cannot be negative, wherein the minimum allowable value is the lowest audible value to the human ear. For example, the sound pressure level of the sub-band signal output by the second speaker120is zero. On the other hand, when the user180moves from the first location601to the second location602and away from the first speaker110, the distance between the user180and the first speaker110is increased from R1to R2. The maximum output limit of the first speaker110may not be able to satisfy formula (5). In that case, the audio system100may issue an overload alert to the user180.

The embodiment ofFIG.6highlights the following advantages. Through the channel-based compensation algorithm, the optimal listening spot for user is not affected by movement. The calculation of the channel-based is simple and efficient, and can be applied in most of the target space600.

FIG.6has illustrated a sound compensation method according to the user180movement. The following description usesFIG.7to illustrate a sound compensation method for the ambient object175. The acoustic attribute information of the ambient object175comprises reflectivity of sound and acoustic absorption rate. The embodiment adapts an appropriate calculation method to calculate the acoustic influence of the ambient object according to the spatial configuration information of the ambient object175.

FIG.7is a schematic view of a target space700of the present invention, for illustrating an embodiment of calculating the amount of audio adjustment according to the acoustic absorption rate of the ambient object.

FIG.7shows a target space700, wherein an ambient object175is located in the middle of a first speaker110and a user180. For example, the ambient object175may be a sofa or a pillar. In this case, the ambient object175may cause listening experience for the user180to decay due to occlusion. In other words, the sound pressure level received by the user180from the first speaker110will be obscured or absorbed. When the control circuit132interprets this layout condition through the spatial configuration information, the acoustic absorption rate of the ambient object175is used to calculate a degree of influence to which the playback effect the first speaker110imposes at the target listening spot (the position of the user180) is affected by the ambient object175, to determine the equal loudness, sound pressure level or gain value of the first channel audio signal112needs to be output.

In one embodiment, the sound pressure level received by the ambient object175from the first speaker110may be used to calculate the sound loss absorbed by the ambient object175:
At[n]=R[n]*SPLt(8)

wherein n represents the number of the sub-band. That is, the first channel audio signal112output from the first speaker110may be divided into a plurality of sub-bands to be separately calculated. At[n] represents the gain value of the nth sub-band detected at time point t. R[n] represents the acoustic absorption rate of the nth sub-band. SPLt represents the sound pressure level of the first speaker110received by the ambient object175at the time point t. The time point t may represent the time difference between the sound transmitted from the first speaker110to the ambient object175.

It can be seen from Equation (8) that At[n] represents a gain value on the n-th sub-band of a first channel audio signal112absorbed by the ambient object175, but also represents an output compensation value required to play the n-th sub-band of the first channel audio signal112. Thus, when the control circuit132generates the first channel audio signal112through the first speaker110, the nth sub-band gain value of the first channel audio signal112is increased by the gain value At[n].

The configuration of the ambient object175located in the middle of the first speaker110and the user180may have a variety of scenarios. The embodiment is based on whether the line of sight between the first speaker110and the user180is obscured, or even further, based on whether a line of sight between to the first speaker110and the user ear is obscured. It will be appreciated that the SPLt itself is a function related to the distance of the ambient object175and the first speaker110, and time. The calculated degree of influence of At[n] on the user180is a function related to the distance of the ambient object175and the user180, and time. When the arrangement of different angles and the relative placements are jointly considered, a variety of nonlinear correlations are involved. The present application is not intended to limit derivative changes of formula (8). Other weight coefficients, parameters, and offset correction values may be included depending on practical implementations. For example, there may be a sofa placed between the user180and the first speaker110. Although the sofa does not block the line of sight, it may still affect the sound pressure level transmitted from the first speaker110to the user180. The control circuit132may incorporate an interpolation method or other correction formula according to equation (8) to make the compensation result more in line with practical requirements.

FIG.8is a schematic diagram of a target space800of the present invention, for illustrating an embodiment for calculating the amount of audio adjustment according to the reflectivity of sound of the ambient object.

FIG.8shows a target space800, wherein a user180is located in the middle of a first speaker110and an ambient object175. The environmental object175may be a wall, ceiling or floor. In this case, the ambient object175will bounce the first speaker110output of the first channel audio signal112to the user180. In other words, the sound pressure level received by the user180from the first speaker110will be superimposed or interfered with. When the control circuit132interprets this layout condition through the spatial configuration information, the reflectivity of sound of the ambient object175is used to calculate a degree of influence on which the playback effect of the first speaker110at the target listening spot (the position of the user180) is affected by the ambient object175, so as to determine the equal loudness, sound pressure level or gain value that the first channel audio signal112needs to output.

In the embodiment, the effect caused by the ambient object175may also be calculated according to equation (8), but R[n] is changed to represent the reflectivity of sound of the ambient object175in the n-th sub-band.

A calculation result of Equation (8), At[n], may represent the amount of the first channel audio signal112reflected to the user180by the ambient object175on the nth sub-band. Thus, the control circuit132renders the first channel audio signal112through the first speaker110with appropriately reduced gain value, so that the total sound pressure level perceived by the user180from the first speaker110and the ambient object175is maintained at the predetermined level.

Similar to the embodiment ofFIG.7, the scenario that the user180inFIG.8is located in the middle of the first speaker110and the ambient object175may have a variety of changing scenarios. The embodiment may be dependent on whether a line of sight between the first speaker110and the ambient object175is obscured by the user180. However, in practice, walls, ceilings, and floors have a reflective effect at any angle. Therefore, the operation formula of the embodiment is not limited to formula (8), and other nonlinear compensation calculation methods may be further derived according to the arrangement and the relationship between near and/or far placements. For example, the target space800can be classified into different application scenarios, such as living room, study, bathroom, theater, or outdoor and the like, depending on the materials of the wall/ceiling/floor, the room size, the room shape, and other characteristics. The host device130may first classify an adequate application scenario best suitable for the target space800, and then adapt corresponding parameters or formulas for calculation, respectively.

The embodiments inFIG.7andFIG.8highlight the following advantages. Through the channel-based compensation operation, the impact on the listening experience of the user180caused by the ambient object175is eliminated. The channel-based compensation operation can flexibly apply different object acoustic properties according to the configuration of ambient objects, which can effectively cope with the optimization problems under a variety of complex environments.

In summary, the recognizer circuit134may receive information from the sensor circuit140to identify the position of the user180in the target space170, allowing the control circuit132to dynamically assign the position of the user180to be the target listening spot. The compensation for the movement of the target listening spot performed by the control circuit132, has been described in the embodiment ofFIG.6and equations (4) to (7). The compensation performed by the control circuit132for the interference of the ambient object175is described inFIGS.7to8and Equation (8). These two compensation operations can be performed separately and applied to the channel audio signal. In other words, the final output of the optimized channel audio signal comprises a compensation value for the movement of the target listening spot, and also comprises compensation for interference of the ambient object175.

The recognizer circuit134captures the field context information according to the sensor circuit140, so that the position of the user180is identified. The identification process may also include the identification of the application scenario to help accelerate the subsequent operation of the control circuit132. The following description usesFIG.9to illustrate how the host device130processes the object identification according to the application scenario.

FIG.9is a flowchart of the host device130identifying the object according to an embodiment of the present invention. The ambient objects175that appear in different application scenarios often have significant ethnic affinity for their acoustic properties, and the coefficient of sound field reflection caused by the surrounding environment material or room size is also different. Therefore, distinguishing the application scenario in advance, helps the audio system100to improve the efficiency of sound field optimization. It will be appreciated that each process inFIG.9performed by the host device130, is not limited to a single circuit or module in it, which may also be a coordinated operation of a plurality of circuits.

In operation902, the host device130acquires an adequate application scenario of the target space170. The host device130may obtain the adequate application scenario in several diverse ways. In one embodiment, the recognizer circuit134in the host device130may identify the field context information provided by the sensor circuit140, and determine the adequate application scenario according to the field context information. In another embodiment, the control circuit132of the host device130obtains spatial configuration information of the ambient object by running a configuration program through the user interface circuit133, and also obtains the application scenario defined by the user180through the configuration program. In a further derived embodiment, the control circuit132in the host device130may acquire relevant information of the adequate application scenario from a user device150by means of a communication circuit136.

In operation904, to accelerate the query of ambient objects and improve the accuracy, the host device130preferentially selects one of the relevant object databases according to the adequate application scenario. The object databases are typically pre-built collections of data that can be provided from a number of various sources. For example, the storage circuit131in the host device130may provisionally store one or more object databases corresponding to different application scenarios. In another embodiment, the host device130may be connected to a remote database160using a communication circuit136. The remote database160may contain a plurality of object databases corresponding to different application scenarios. Each object database contains information about the appearance characteristics of multiple ambient objects175, as well as acoustic attribute information.

When the host device130obtains an application scenario in the operation902, an object database may be preferentially selected from the storage circuit131or the remote database160best suitable for the application scenario to perform subsequent identification of ambient objects. In one embodiment, the recognizer circuit134analyzes the field context information provided by the sensor circuit140to obtain one or more object appearance characteristic information, and looks up the object database according to the object appearance characteristic information. An ambient object in line with the object appearance characteristic information may be identified, comprising the name, acoustic absorption rate, and reflectivity of sound. In another embodiment, the control circuit132executes a configuration program, using a user interface circuit133to obtain the name of an ambient object. The control circuit132looks up the object database according to the name of the ambient object175to obtain the corresponding acoustic absorption rate and reflectivity of sound corresponding to the ambient object.

In a further derived embodiment, the parameters used in the process of finding may be multi-variably combined. For example, when the recognizer circuit134analyzes the field context information, the ambient object175may obtain the material, size, shape and other appearance features. The recognizer circuit134transmits the appearance feature information to the object database for multi-condition cross-comparison, and obtains a list of candidates sorted according to the matching score. If the application scenario information is used together in the process of finding the object database, the search process can be effectively narrowed down to accelerate the recognition, and improve the correctness.

In operation906, the control circuit132looks up the acoustic absorption rate and reflectivity sound corresponding the ambient object from the object database selected in operation904. In practice, the acoustic attribute information of ambient objects stored in the object database is not limited to being stored in multiple independent object databases. An object database can be a correlational database that contains a variety of fields connected together in the form of correlation coefficients. For example, the fields of an object database can contain object names, application categories, materials, acoustic absorption rates, reflectivity of sound, and even appearance features such as shape, color, gloss, and so on. The field value corresponding to each ambient object175is not limited to a one-to-one relationship, but can be one-to-many, or many-to-one. The value stored in each field is not necessarily an absolute value, but a range value or probability value. In a further derived implementation, the object database may be an adaptive database that can be machine learned and constantly iteratively corrected. The user180may feedback the preference settings through the user interface circuit133and train the object database.

In operation908, the control circuit132individually adjusts each of the channel audio signals in a plurality of sub-bands according to search results and configuration conditions of the ambient object. The acoustic properties of the ambient object175, may be significantly varied in different sub-bands. For example, a sofa may absorb a large number of high-frequency signals, but does not affect the penetration of low-frequency signals. Therefore, the acoustic absorption rate or reflectivity of sound found from the object database can be an array value corresponding to multiple sub-bands, or a frequency response curve. Regarding the bandwidth or segmentation of each sub-band, it may vary with the design requirements, and is not limited in the embodiment. The adjustment performed by the control circuit132to adjust the gain value of the channel audio signal in a plurality of sub-bands, may be simulated as an equalizer or filter concept in practice. In other words, the control circuit132may be a first-class equalizer for each speaker in the audio system100, and the output compensation value calculated according to the foregoing embodiment is customized into the equalizer, allowing the corresponding channel audio signal to be adjusted. Further embodiments for calculating the output compensation value will be illustrated inFIG.10.

In operation910, the control circuit132transmits adjusted audio signals to corresponding speakers through the audio transmission circuit135. An embodiment of the audio transmission circuit135is described inFIG.1and will not be repeated herein.

The embodiment ofFIG.9highlights the following advantages. The object recognition can be performed with reference to the application scenario (automatic recognition or manual input) to increase identification efficiency. The object database adopts an extensible architecture to continuously enhance recognition capabilities in the long term under the lifecycle of big data services and machine learning in the cloud. The sound system100may apply the concept of equalizers to divide the channel audio signal into multiple sub-bands for separate process, so that the final synthesized sound quality is effectively improved.

The following description further illustrates inFIG.10how the control circuit132calculates the output compensation value of each channel according to the spatial configuration information of the ambient object175.

FIG.10is a flowchart of an audio processing method according to an embodiment of the invention, explaining how to calculate the output compensation value based on position relationships between ambient objects. The processes inFIG.10are mainly performed by the control circuit132in the host device130.

In operation1002, the control circuit132determines a relative position relationship between the ambient object, the target listening spot, and the speakers. A plurality of speakers and a plurality of ambient objects175in the target space170, may be arranged with the target listening spot to form a plurality of sets of position relationships. Each set of positional relationships corresponds to a speaker, an ambient object175, and a target listening spot. The control circuit132examines and evaluates each set of positional relationships in the target space170and calculates the corresponding output compensation value. The following description takes one of the position relationships in the audio system100as an example, to explain how the control circuit132compensates for to a speaker interference at the target listening spot caused by an ambient object175

In operation1004, the control circuit132determines whether the ambient object175is between the target listening spot and the speaker. The position of the ambient object175in the target space170may also be obtained by the recognizer circuit134, or by the user interface circuit133controlled by a configuration program. After the control circuit132synthesizes the above information, a relative position relationship composed of an ambient object175, a target listening spot, and a speaker is determined, and a corresponding compensation is conducted on each speaker output. Process1004shows the situation to be determined asFIG.7. If the situation is met, operation1008is conducted. If the situation is not met, operation1006is conducted.

In operation1006, the control circuit132determines whether the target listening spot is located in the middle of the ambient object175and the speaker. The situation to be determined by operation1006is shown asFIG.8. If the situation is met, operation1010is conducted. If the situation is not met, operation1012is conducted.

In operation1008, the control circuit132uses the acoustic absorption rate of the ambient object175to calculate the output compensation value of the channel audio signal. In a preferred embodiment, the output compensation value of the channel audio signal from a speaker is divided into a plurality of sub-bands for separated calculation. Detailed calculations may refer to the target space700ofFIG.7and the formula (8). The control circuit132may look up the acoustic absorption rate of the ambient object175from an object database, and substitute it into equation (8) to obtain the output compensation value.

In operation1010, the control circuit132uses the reflectivity of sound corresponding to the ambient object175to calculate the output compensation value of the channel audio signal. Referring to the target space800ofFIG.8and equation (8), the control circuit132may look up the reflectivity of sound corresponding to the ambient object175from the object database, and substitute it into equation (8) to obtain an output compensation value.

It is understandable that the output compensation value calculated according to the acoustic absorption rate of the ambient object175may amplify the gain value, the sound pressure level, or the equal loudness of the adjusted channel audio signal to compensate for the energy absorbed. Conversely, the output compensation value calculated according to the reflectivity of sound corresponding to the ambient object175may reduce the gain value, the sound pressure level, or the equal loudness of the adjusted channel audio signal to balance the energy reflected back. In other words, the output compensation values calculated based on the acoustic absorption rate are usually polarly opposite that calculated from the reflectivity of sound.

In operation1012, if the ambient object175does not meet the conditions of operation1004and operation1006, the control circuit132may determine the ambient object175in a position that will not affect the speaker to the target listening spot playback. In that case, the control circuit132may ignore the effect of the ambient object175imposed on the speaker and the target listening spot. However, it is to be understood that a target space170typically contains a plurality of speakers. The ambient object175does not affect one of the speakers to the target listening spot, but may still affect another speaker. In other words, the control circuit132needs to individually perform processes ofFIG.10for each set of the positional relationships in the target space170.

In some specific cases, the presence of the ambient object175may be directly ignored. For example, if the sound reflected or absorbed by the ambient object175is less than a particular threshold, the presence of the ambient object175in the target space170may be ignored. On the other hand, if the control circuit132determines that the volume of the ambient object175is less than a particular size, the presence of the ambient object175may also be ignored.

In a further derived embodiment, if more than one user is detected in the target space170, the target listening spot may be determined to be a position center point of the plurality of users, or selectively based on the location of one of the users. As for the other users not elected as the target listening spot, the host device130may take them as ambient objects and accordingly apply the embodiments ofFIGS.7to8.

The embodiment ofFIG.10highlights the following advantages. The embodiment ofFIG.10follows the processes inFIGS.7and8, to simplify the complex environmental problem into a plurality of linear relationship problems that can be solved separately. For some particular cases, the ambient object175may also be ignored to simplify the complexity of the calculation.

FIG.11is a schematic diagram of a target space of the present invention1100for illustrating an embodiment of optimizing the sound field by performing an object-based compensation.

In the target space1100, there are multiple speakers, such as a first speaker1110, a second speaker1120, a third speaker1130and a fourth speaker1140. In the case where the audio system100operates based on the object-based compensation operation, the control circuit132logically treats the target space1100as a coordinate system. The spatial coordinate can be a two-dimensional planar coordinate or a three-dimensional Cartesian coordinate. For ease of illustration,FIG.11illustrates the description in a way that comprises a two-dimensional plane coordinate of an X axis and a Y axis.

In the target space1100, the user180is located at the origin P0. The control circuit132assigns the location of the user180to be the target listening spot. As described in the embodiment ofFIG.3, the object-based acoustic system is based on a large number of acoustic parameters matrix calculation. Each source object has metadata for describing properties of the source object, such as type, position, size (length, width, height), divergence, and so on. After the object-based operation, the sound presented by a source object is assigned to one or more speakers to be jointly played. Each speaker may respectively play a proportion of the sound of the source object. In other words, the object-based acoustic system can utilize multiple speakers to simulate the physical presence of a sole source object. For example, through the object-based compensation operation, the user180at the target listening spot, can hear a virtual audio object1105along the movement trajectory1103from the first location P1to the new first location P1′.

The object-based compensation operation of the embodiment may optimize all speaker output channel audio signal experienced at the target listening spot. The Object-based compensation operation utilizes array arithmetic modules in existing object-based acoustic systems to parameterize various distance factors and sound field categories, and can perform operations similar to equations (4) to (7). For the audio system100, the host device130only needs to apply the position information of the user180to the object-based compensation operation, and thereby the channel audio signal output from all speakers can be optimized from the perspective of the target listening spot.

In one embodiment, the control circuit132may define a target listening spot as the origin of the entire spatial coordinate system. When the user180moves, the entire spatial coordinate system moves with the origin. In other words, the position of the virtual audio object1105relative to the origin remains unchanged. When the control circuit132plays the effect of the virtual audio object1105through the object-based compensation operation, the relative position of the virtual sound source object1105felt by the user180will not change with the movement of the user180.

In the target space of the embodiment1100, there may exist an ambient object175having a substantial impact on the listening experience for the user180. The control circuit132may obtain the spatial configuration information of the ambient object in the target space1100through the operation902ofFIG.9, wherein the location of the ambient object175is determined to be at the second location P2. When the user180moves, the origin of the entire spatial coordinate system changes with the user180. Although the ambient object175does not move, the relative position of the origin changes. It is therefore understandable that in the space coordinate system after the movement of the user180, the coordinate value of the ambient object175is moved in the opposite direction.

To cancel the interference generated by the ambient object175on the user180, the control circuit132of the embodiment creates an object-based compensatory audio object according to the ambient object175. The metadata of the compensatory audio object comprises: the coordinate of the ambient object175, the size, the reflectivity of sound, and the acoustic absorption rate. The reflectivity of sound and the acoustic absorption rate of the ambient object175may be obtained in operation906ofFIG.9. The compensatory audio object is regarded as a negative source object of the ambient object175and is applied to the object-based compensation operation to become a virtual sound source that can cancel the ambient object175.

It is understandable that the essence of the compensatory audio object is the negative source object corresponding to the ambient object175, the position of which overlaps with the ambient object175, so it is not otherwise illustrated inFIG.11. Furthermore, the four-speaker configuration of the target space1100is just an example. In practical applications of the audio system100, the number of speakers may have a lot more varieties, including the 3D case wherein upper speakers and lower speakers are deployed. The description is not intended to limit any other possible configurations.

The embodiment ofFIG.11illustrates the advantages of the object-based compensation operation. The control circuit132converts the information of the target space1100into the form of a spatial coordinate system, which may simplify the complex multi-object interaction calculation into a matrix calculation of metadata. The position of the user180in motion is set to the origin of the spatial coordinate system, so that the processing of the virtual object is completely unaffected by the movement of the user180, and the operation process is simplified. The embodiment also proposes the concept of compensating the sound source object, directly applying the object-based compensation operation to offset the interference of the ambient object, thereby eliminating the need for complex multi-channel interaction operations.

The following description explains inFIG.12the simplicity of the object-based compensation operation and possible derivative applications.

FIG.12is a schematic diagram of a target space of the present invention1200for illustrating an embodiment of an optimized sound field operating on an object-based compensation.

The target space1200may contain a plurality of speakers, such as the first speaker1210, the second speaker1220, the third speaker1230, the fourth speaker1240, the fifth speaker1250and the sixth speaker1260, arranged as a long bar sound field. Each speaker corresponds to an ID. When the user180is located in the first location P1, the data in a virtual audio object (not shown) is mapped to the ID of the first speaker1210and the second speaker1220. After the control circuit132performs the object-based compensation operation, the first speaker1210and the second speaker1220play the first channel output1212and the second channel output1222, so that the user180feels the existence of the virtual audio object. When the user180moves along the movement trajectory1203to the second location P2, the control circuit132is recalculated by the target listening spot, the data in the virtual audio object is mapped to the fifth speaker1250and the sixth speaker1260. After the control circuit132performs the object-based compensation operation, the fifth speaker1250and the fifth speaker1250play the fifth channel output1252and the sixth channel output1262, so that the user180feels that the virtual audio object still exists around the user180, and does not leave with the movement of the user180.

The embodiment mainly illustrates the flexible application and simplicity of the object-based compensation operation. In many exceptional cases, only a small amount of computation is required to optimize the sound field. For example, if the user180is located in a spherical sound field, the control circuit132only needs to perform the calculation of rotation coordinates, allowing the user180to experience consistent sound field effect when facing various directions.

The following description summarizes inFIG.13the fundamental logic of the control circuit132when performing the object-based compensation operation.

FIG.13is a flowchart of an object-based compensation operation according to an embodiment of the present invention, illustrating the concept of creating a compensatory audio object.

In operation1304, the control circuit132creates a corresponding compensatory audio object according to the ambient object175. For the user180located at the target listening spot, the presence of an ambient object175is a physical sound source. The ambient object175may reflect the sound emitted by a speaker to the target listening spot. The ambient object175may also block or absorb a portion of the sound, so that the sound emitted from the speaker to the target listening spot is attenuated. The compensatory audio object is a negative sound source object created for the ambient object175. When the host device130substitutes the compensatory audio object into the object-based compensation operation to produce channel audio signal, the presence of the ambient object175is cancelled. The specific details of the object-based operation itself can be extended to the calculation method of the existing object-based acoustic product, using the metadata of the sound source object to perform a large number of related matrix calculations. For example, one of the data of the compensatory audio object comprises: the coordinate of the ambient object175, the size, the reflectivity of sound, and the acoustic absorption rate.

In operation1306, the control circuit132calculates the acoustic effect of the compensatory audio object. In the embodiment, the compensatory audio object is correspondingly created according to the ambient object175, wherein the data has the same coordinate position as the ambient object175, the size, the reflectivity of sound, and the acoustic absorption rate, which renders inverted sound source effects that cancels the gain value of the ambient object175.

The embodiment ofFIG.13may also apply the calculations ofFIGS.7and8. Equation (8) may further be derived into Equation (9), calculating a passively generated gain value of the ambient object175according to the sound pressure level generated by the first speaker110and imposed on the ambient object175:
At[m][n]=R[n]*SPLt[m](9)

Wherein, m represents the speaker number, n represents the number of the sub-bands. At[m][n] represents the gain value produced by the influence of the n-th sub-band by the m-th speaker. R[n] represents the acoustic absorption rate of the n-th sub-band. SPLt[m] represents the sound pressure level of the ambient object175received at the time point t from the m-th speaker. The time point t may represent the time difference in which the sound is transmitted from the speaker to the ambient object175. If the time difference is greater than a non-negligible range, it means that there is an echo condition in the target space170.

From Equation (9) it can be seen that the calculation results of each ambient object, essentially comprise a gain value array corresponding to a plurality of speakers and a plurality of sub-bands at a time point. The acoustic effect of the compensatory audio object is the negative value of the gain value array. That is, the object-based compensation operation based on Equation (9) contains matrix calculations of interactively arranged parameters in multiple dimensions. For illustrative purposes, the gain values corresponding to one of the speakers and one of its sub-bands at a time point are illustrated below.

The embodiment ofFIG.13is similar to the embodiments ofFIGS.7and8, which may be based on the spatial configuration information of the ambient object175to correspondingly calculate the acoustic influence of the ambient object using an appropriate calculation method. For example, if the target listening spot is located between a speaker and the ambient object175of the line of sight, the control circuit132calculates the acoustic effect of the compensatory audio object according to the reflectivity of sound corresponding to the ambient object175. In contrast, if the ambient object175is located between the target listening spot and the speaker's line of sight, the control circuit132calculates the acoustic effect of the compensatory audio object according to the acoustic absorption rate of the ambient object175.

For example, when an ambient object175absorbs the sound emitted by a speaker, the loudness effect received by the target listening spot is reduced. Meanwhile, the control circuit132creates a virtual sound source object that will produce a corresponding loudness effect on the coordinate of the ambient object175as compensation. In contrast, if an ambient object175reflects the sound of a speaker, so that the target listening spot receives too much loudness. In that case, the control circuit132creates at the coordinate of the ambient object175a virtual audio object having a negative gain value.

It is understandable that the line of sight is defined as a straight line of two objects in space. Physical objects have a certain volume and area, wherein the volume can be so large that partially or totally occludes the line of sight. The embodiment may be based on Equation (9), and further weight coefficients or different offset corrections may be included in the equations depending on practical scenario.

In operation1308, the control circuit132mixes the acoustic effect of the compensatory audio object into the channel audio signal, to be played by corresponding speakers. The object-based compensation operation performed by the control circuit132may manage complex matrix calculations corresponding to objects, wherein each speaker is assigned to play a corresponding one-channel audio signal synthesized from a plurality of audio source signals. After applying the object-based compensation, the listening experience received at the target listening spot comprises the source effect produced by the compensatory audio object. Thus, the interference caused by the ambient object175may be effectively cancelled by the compensatory audio object.

In the operation1310, the control circuit132determines whether the target listening spot is moved to a new position. As described in operation208, the audio system100continuously tracks the movement of the user180and updates the target listening spot. If the target listening spot has moved, the operation1312is performed. Otherwise, the playback operation of operation1308continues.

In the operation1312, the control circuit132updates the metadata of the compensatory audio object. In the embodiment, the control circuit132will establish an object-based space with the target listening spot as an origin. If the target listening spot moves to a new position, the control circuit132assigns the new position to the new origin of the object-based space. The difference between the new origin and the origin coordinate can be expressed as an offset vector. The spatial coordinate value of the ambient object175relative to the target listening spot will also change inversely with the movement vector. The control circuit132then updates the metadata of the compensatory audio object corresponding to the ambient object175according to the offset vector. In a further embodiment, all speakers in the object-based space may also be regarded as objects, having a corresponding ID, metadata and coordinate values.

In another embodiment, the audio system100is not limited to using the target listening spot as the origin. The audio system100may also employ a fixed reference point as the origin of the object-based space. When the source object in the object-based space appears relative position changes, the control circuit132correspondingly updates the coordinate values in the source object's metadata.

When operation1312is completed, the control circuit132repeats operation1308.

The embodiment ofFIG.13illustrates the advantages of the object-based compensation operation. The control circuit132converts the information of the target space1100into the form of a spatial coordinate system, which may simplify the complex multi-object interaction calculation into a matrix calculation of metadata. The position of the user180in motion is set to the origin of the spatial coordinate system, so that the processing of the virtual object is completely unaffected by the movement of the user180, and the operation process is simplified. The embodiment also proposes the concept of compensating the sound source object, directly applying the object-based compensation operation to cancel the interference of the ambient object, eliminating the need for complex multi-channel interaction operations.

In a further derived embodiment, if the host device130is not capable of performing the object-based audio synthesizing, the control circuit132may provide a channel mapping function by executing the software, so that the operation result of the object-based can be correctly corresponded to each speaker.

In summary, the present application proposes an audio system100, which can dynamically track the user's position and optimize the sound field, and intelligently eliminate interference caused by ambient objects. The means for tracking the user's location may be implemented by a variety of separate applications or combinations of cameras, infrared, or wireless detectors. The spatial configuration information of the ambient object175in the target space170may be obtained by the image captured by the camera after identification, or it may be manually entered by the user. The sound field can be optimized by the channel-based compensation operation or the object-based compensation operation. When calculating the influence of the ambient object175on the target listening spot, the relative position relationship between the ambient object175and the speaker and the target listening spot may also be considered, so that different calculation methods can be adopted. When using the object-based compensation operation, the control circuit132creates a corresponding compensatory audio object for each ambient object175, so that the channel audio signal rendered by audio mixing cancels the interference caused by the ambient object175at the target listening spot.

Certain terms are used throughout the description and the claims to refer to particular components. One skilled in the art appreciates that a component may be referred to as different names. This disclosure does not intend to distinguish between components that differ in name but not in function. In the description and in the claims, the term “comprise” is used in an open-ended fashion, and thus should be interpreted to mean “include, but not limited to.” The term “couple” is intended to encompass any indirect or direct connection. Accordingly, if this disclosure mentioned that a first device is coupled with a second device, it means that the first device may be directly or indirectly connected to the second device through electrical connections, wireless communications, optical communications, or other signal connections with/without other intermediate devices or connection means.

The term “and/or” may comprise any and all combinations of one or more of the associated listed items. In addition, the singular forms “a,” “an,” and “the” herein are intended to comprise the plural forms as well, unless the context clearly indicates otherwise.

Other embodiments of the invention will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the invention indicated by the following claims.