Spatialized audio over headphones

A spatial element is added to communications, including over telephone conference calls heard through headphones or a stereo speaker setup. Functions are created to modify signals from different callers to create the illusion that the callers are speaking from different parts of the room.

BACKGROUND

This Background is intended to provide the basic context of this patent application and it is not intended to describe a specific problem to be solved.

Conference calls have been possible for many years. Callers from around the world can call in and discuss topics together. However, on a conference call, it is sometimes hard to tell who is talking. In some cases, voices are distinct and can be recognized. Conversation that occur in person have a spatial element such that if a person speaks from the left, the listener will know the sound is coming from the left. On conference calls, no such spatial element is present making it difficult to tell who is talking.

SUMMARY

A spatial element is added to communications, including over telephone conference calls heard through headphones or a stereo speaker setup. Functions are created to modify signals from different callers to create the illusion that the callers are speaking from different parts of the room. To create the function, a signal is communicated from a first location and is received in a left channel and a right channel at a listening point. The received signal at the left and right channel is compared to the communicated signal. A function is created to modify the signal to minimize the different between the communicated signal and the signal received in the left channel and the right channel. This function is then used to modify callers signals to add a spatial element to each caller's signal.

SPECIFICATION

FIG. 1illustrates an example of a suitable computing system environment100that may operate to execute the many embodiments of a method and system described by this specification. It should be noted that the computing system environment100is only one example of a suitable computing environment and is not intended to suggest any limitation as to the scope of use or functionality of the method and apparatus of the claims. Neither should the computing environment100be interpreted as having any dependency or requirement relating to any one component or combination of components illustrated in the exemplary operating environment100.

With reference toFIG. 1, an exemplary system for implementing the blocks of the claimed method and apparatus includes a general purpose computing device in the form of a computer110. Components of computer110may include, but are not limited to, a processing unit120, a system memory130, and a system bus121that couples various system components including the system memory to the processing unit120.

The computer110may operate in a networked environment using logical connections to one or more remote computers, such as a remote computer180, via a local area network (LAN)171and/or a wide area network (WAN)173via a modem172or other network interface170.

Computer110typically includes a variety of computer readable media that may be any available media that may be accessed by computer110and includes both volatile and nonvolatile media, removable and non-removable media. The system memory130includes computer storage media in the form of volatile and/or nonvolatile memory such as read only memory (ROM)131and random access memory (RAM)132. The ROM may include a basic input/output system133(BIOS). RAM132typically contains data and/or program modules that include operating system134, application programs135, other program modules136, and program data137. The computer110may also include other removable/non-removable, volatile/nonvolatile computer storage media such as a hard disk drive141a magnetic disk drive151that reads from or writes to a magnetic disk152, and an optical disk drive155that reads from or writes to an optical disk156. The hard disk drive141,151, and155may interface with system bus121via interfaces140,150.

A user may enter commands and information into the computer20through input devices such as a keyboard162and pointing device161, commonly referred to as a mouse, trackball or touch pad. Other input devices (not illustrated) may include a microphone, joystick, game pad, satellite dish, scanner, or the like. These and other input devices are often connected to the processing unit120through a user input interface160that is coupled to the system bus, but may be connected by other interface and bus structures, such as a parallel port, game port or a universal serial bus (USB). A monitor191or other type of display device may also be connected to the system bus121via an interface, such as a video interface190. In addition to the monitor, computers may also include other peripheral output devices such as speakers197and printer196, which may be connected through an output peripheral interface190.

FIG. 2is a flowchart of a method of providing directional hearing experience for a conference call. In real life, people can perceive direction with speech. For example, a person talking from the left side will be perceived as talking from the left side. Currently, when different people speak on a conference call, there is no directional component to the speech. In reality, the people in the conference call could be sitting around a table or could be in different parts of the world. It would be useful to have a directional component to conference calls to assist in determine who is speaking.

In most current designs of spatial audio systems aiming at real-time operation, externalization is typically achieved using artificial reverberation. Artificial reverberation is a well-studied topic and as a result, a rich collection of numerically motivated tools have been developed such as feedback delay networks. These tools, although computational efficient, do not have sufficient means to capture most of the subtitles of the environment.

In another extreme, sophisticated modeling techniques, notably wave-equation and ray-tracing based acoustic simulation methods, have emerged as possible candidates for real-time spatial audio synthesis. The cost of implementing these modeling methods on conferencing terminals is not acceptable, not to mention the challenges of building physical models in sufficient detail to be useful.

Instead, the method proposes to bypass any parametric modeling and use the room response directly measured from the actual physical space, i.e. a typical conference room in this case. Furthermore, as early reflections may be so closely coupled to the effect of Head-Related Transfer Function (HRTF), there is little benefit in trying to separately model the room and the head. Suppose a speaking person and a listening person are located in the same room, and assume a linear model from the speaking person's mouth to each of the listening person's two ears. If there are accurate estimates of the two linear responses and the linear responses are used to process the monophonic capture of the voice of the speaking person, a true binaural capture may result.

At block200, a first signal305may be broadcast from a first source310at a first location315. The first signal305may be virtually any signal that can be detected by a microphone320, such as a voice, a tone, music or a speech. In some embodiments, the method is directed to conference call and human voices may be be the logical choice for the first signal305. Studies on room acoustic measurement suggest a number of good candidates for reference signal r(t). Different choices have been compared and Maximum Length Sequence may be recommended for noisy rooms, and a form of chirp signal (logarithm sine sweep) is recommended for quiet rooms. As the noise level in the measurement environment may be controllable, a chirp signal may be selected due to its other advantages. Thus,

where f1is the starting frequency, f2is the ending frequency, T is the duration of the reference signal and t represents continuous time. Note that as all of processing steps are finished as digital time samples, the method may subsequently switch to a discrete time notation where r(n) denotes the appropriately sampled version of r(t), etc. Considering only the linear response, the captured signals may be
sil(n)=r(n)*hil(n)+u(n) andsir(n)=r(n)*hir(n)+v(n)

for any configuration i (0<i=I), where * denotes linear convolution and u(n) and v(n) are additive noise terms.

The source310may be a speaker as illustrated inFIG. 3or may be a person (voice)310as illustrated inFIG. 5. The first location315may be any location that is within a distance such that the first signal305may be received by the microphone320.

The details of the location315may be measured and stored in a variety of ways. In one embodiment, the location315may have a distance from the microphone320and a degree off from a centerline325(dashed) from the microphone320. For example, the first location315may be 0 degrees off the center line325and the second location330may be 30 degrees off the center line325. In some embodiments, the location may be stored in a360degree format, such that the first location315may be stored as 0 degree and the second location330may be stored as 330 degrees (360−30). In addition, the location may include some data about the environment, such as the size of the room or the distance from the first source315to the surrounding walls, etc. Other data may include the surface of the walls, whether there are windows in the location and if so, ambient noise in the room, how many, the type of ceiling, the ceiling height, the floor covering, etc.

At block205, the first signal305(r(t)) may be received at the hearing location323. The hearing location320may receive the first signal305as the received first left channel335and the received first right channel340. In one embodiment, the hearing location323is similar to a human head, possibly on a human body, and the received first left channel335hl(t) is received in a microphone close to the left ear of a human head and the received first right channel340hr(t)is received in a microphone close to the right ear of the human head. The using of both a received first left channel335and a received first right channel340may improve the ability to create a spatial component to the received sound. It may be assumed that all speaking persons lie on a plane with the same elevation. Each configuration may be indexed by i in hli(t) and hri(t), 0<i<=I.

At block210, the received first left channel335of the first signal305at the hearing location323may be stored in a memory as a first received left channel signal. The first signal305will be affected by a variety of factors before being received at the microphone320at the hearing location323and as the received first left channel335and the received first right channels340, such as the room and the shape of the hearing location323. Even the shape of the mock human head may affect the first signal305differently in each microphone placed near each mock ear. As a result, there will be difference between the communicated first signal305and the received first left channel335and received first right channel340.

At block215, the received first right channel340of the first signal305at the hearing location323may be stored in a memory as the received first right340signal. Again, the first signal305will be affected by a variety of factors before being received at the microphone320at the hearing location323and as the received first left channel335and the received first right channels340, such as the room and the shape of the hearing location323. Even the shape of the mock human head on the mock human body may affect the first signal305differently in each microphone placed near each mock ear. As a result, there will be difference between the communicated first signal305and the received first left channel335and the received first right channel340.

When noise is negligible, it is rather straightforward to recover the combined head and room impulse responses (CHRIRs) using inverse filter. In the frequency domain, the result may be

where R(.) etc denote the discrete-time Fourier transforms of their time domain counterparts. The simple solution is obviously inadequate in reality as the effect of noise will be ever present. Instead of strictly following the steps of constructing an inverse filter, the method may follow a slightly different procedure. First, the method may obtain the time reversed signal r(−n) and convolve with the response signal r(n). Equivalently, what happens in the frequency domain is, using the left-ear case as the example,
Gil(ω)=Sil(ω)R(ω)=Hil(ω)|R(ω)|2e−jωD+U(ω)R(−ω)

where D is an arbitrary constant delay depending on the length chosen for r(n).

Note that so far the method may not be concerned about the amplification of the high frequency noise as the method may have in the case of direct inverse filtering.

However, Gil(ω) may not be a good estimate of Hil(ω) due to the magnitude distortion caused by |R(ω)|2. To that end, the method may apply a linear phase equalization filter derived from psychoacoustics means. Using the exact same set up, the method may play a known speech signal x(n) through the loudspeaker310. Let the captured signal received by one of the microphones320(it doesn't matter which one) be y(n). The method may first define the initial equalization filter in the frequency domain to be
E(ω)=Y(ω)/Ĥil(ω)X(ω) and hence
Ĥil(ω)=Gil(ω)E(ω)

Under the ideal condition free of any noise, the method may have completely removed the effect of |R(ω)|2with the initial equalization filter. Such not being the case, the method may seek to find the filter E(ω) that minimizes the perceptual difference between the synthesized signal and captured signal:

where M(ω) is a frequency domain masking curve determined via any standard procedure for input X(ω), and k is the index to the critical band partition of choice. In other words, the method may obtain E(ω) by minimizing a metric based on a simplified model of the human perceptual system. Alternatively, the method may also obtain a reasonable approximation of E(ω) via subjective listening evaluation of the synthesized and captured signal. To keep the minimization manageable, it suffices to assume E(ω) is smooth and is a constant within each critical band. It should be pointed out as well that in a real implementation the above equation should be considered in a frame by frame fashion and averaged over all available frames. Within each frame, sufficient care should be taken so that linear convolution can be roughly approximated.

It is known that room response estimation routines often modify the timbre of the room. The proposed perceptual formulation gives a means to match the timbre close to that of true binaural recording while keeping the noise amplification under control simultaneously. As a minor detail, note that the delay between ĥiland ĥirfor the same i should be strictly maintained throughout the processing chain while the delays between ĥil(or ĥir) for different I does not matter too much and can be calibrated.

At block220, the first location315may be stored in a memory. The first location315may be a location in relation to the hearing location323. As explained previously, in one embodiment, the location315may have a distance from the microphone320and a degree off from a centerline325(dashed) from the microphone320. For example, the first location315may be 0 degrees off the center line325and the second location330may be approximately 30 degrees off the center line325. In some embodiments, the location may be stored in a 360 degree format, such that the first location315may be stored as 0 degree and the second location330may be stored as 330 degrees (360−30). In addition, the location may include some data about the environment, such as the size of the room or the distance from the first source315to the surrounding walls, etc. Other data may include the surface of the walls, ambient noise in the room, whether there are windows in the location and if so, how many, the type of ceiling, the ceiling height, the floor covering, etc.

FIG. 4may illustrate one embodiment of using the modeling and estimation ofFIG. 2to create a spatial audio signal. Multiple audio streams from all other remote participants may be commonly multiplexed into one before sending to a particular participant. In order to enable spatialized audio, the method may need a different architecture that resembles a full-mesh peer-to-peer network. Regardless of how the network topology is implemented, some embodiments of the method may assume that each participant has access to any other remote participant' voice as an individual stream. Furthermore, the method may assume each conferencing location may have only one voice which is captured with a monophonic close-range microphone. When such assumptions can not be met, techniques such as source separation and de-reverberation may be exploited so that a close enough approximation to our assumption can hold true.

When the number of participants is high in a meeting, it may not be practical to map each remote participant a distinctive location in which case strategies such as binning more than one remote participants to a shared virtual location can be considered. Without loss of generality, however, some embodiments may assume there is a one-to-one mapping between a remote participants and the rendering location. Under these assumptions, the task of the rendering spatial audio seems straightforward. For simplicity, suppose all CHRIRs, ĥil(n) and ĥir(n), have the same finite duration of N samples.

While on the surface this may appear similar to convolution reverberation, the described models entail a lot of more information than just reverberation and are estimated with unique means as discussed above. Nonetheless, the known difficulties with this approach still exist. Compared with the model-based approaches mentioned earlier, the CHRIRs are difficult to customize. Even with subjective tuning, the measured CHRIRs can not please every user. In particular, since human ears have varied tolerance to perceived reverberation, it may be beneficial to provide users with a means of adjusting to his own preference. Secondly, the method may be limited to render the speaker-listener configurations determined a prior at measurement time. It is rather difficult, for instance, to model a moving sound source. Thirdly, the computational cost is higher than the numerical model-based approach by any measure.

At block400, a first left channel function may be created to modify the first signal305to minimize the difference between the first signal305and the first received left channel signal335. In one embodiment, a Fourier transform is used to create the function to modify the first signal305. Of course, other method to create the first left channel function to modify the first signal305to minimize the difference between the first signal305and the first received left channel signal335are possible and are contemplated.

The adjusting acoustic ratio may also be adjusted. The acoustic ratio may refer to the ratio between the energies of the sound waves following the direct path and the reverber-ation. A higher acoustic ratio implies a drier sounding signal and vice versa. The method may use the following means to locate the peak in any CHRIR that corresponds to the direct path, based on the intuitive principle that the direct path sound has the highest energy:

From here, using left ear channel as the example, the method may modify the CHRIR as

where δ defines a small neighborhood and α>0 is a user controlled parameter which effectively changes the acoustic ratio of the synthesized audio.

In other applications of spatial audio such as games and movies, there are many occasions where the sound source undergoes significant motion while being rendered, in which case parametric 3D audio techniques that can explicitly model the motion trajectory are the most appropriate. In the pending method, there seems little need to model this type of source. Nonetheless, in the real world people do move slightly during talking and/or a listening person may sometimes want to move the virtual location of a remote participant. Following the method, it may be possible to include such small range motion in the synthesis system.

Upon inspection of a pair CHRIRs for the left and right ear channels from the same configuration, it may be seen that the most obvious contrast between them is the delay and level difference. Indeed, interaural time difference (ITD) and interaural intensity difference are the two prominent cues of directivity perception for the human hearing system. Though not sufficient to generate realistic spatial audio by themselves, experiences show that they suffice as tools to alter the perceived directivity from a pair of given CHRIRs. The ITD and IID of a pair of CHRIRs ĥil(n) and ĥir(n) are estimated as

Next, these discrete IID and ITD samples are interpolated to generate the corresponding parameters at any arbitrary configuration φ. Afterward, the method may construct the CHRIRs for any configuration φ as

During synthesis, the method may arbitrarily vary φ, at a small range around each i to simulate a slow, localized moving source i.e. the speaking person. In addition to ITD and IID, note that can be altered as well to simulate a change of range. The same mechanism also provides a means for users to control the virtual location of a given source.

The direct convolution approach may have an algorithm complexity of O(IN) where I is the total number of participant and N is the length of CHRIR. The issue is that both I and N can be fairly large. To tackle the dimensionality of N, fast convolution methods taking advantage of the fast Fourier transform are readily available, although they invariably introduce a delay as the processing is in a block to block fashion. Since additional delay is undesirable for real-time conferencing applications, the method may follow some alternative ideas on improving the computational efficiency with no delay penalty.

First, a CHRIR may receive contributions from a number of known factors: direct path propagation, reflection and diffraction due to the human body parts, early reflection and late reverberation of the room, etc. Fortunately, all of the location dependent effects take place in early part of the CHRIR while anything afterwards (e.g. 10 milliseconds) is generally considered reverberation. Reverberation due to its very nature is mostly location independent. Given these observations, the method may decompose CHRIRs into the early portion, namely a short filter, and the late portion (a longer filter). Furthermore, the long filter is shared among all locations:
ĥiSl(n)=ĥil(n), 0≦n<Mand
ĥL(n)=ĥil(n),M≦n<N

for any arbitrarily chosen i, where M is a threshold set to for instance10milliseconds, again using the left ear channel as the example. Thus, to synthesize spatial audio for the ith location, the method may simply follow

The right ear channel processing follows exactly the same routine. Note the new method has a complexity of O(IM+N). Since typically M<<N and N can be large, the saving is substantial.FIG. 7may illustrate one possible illustration of the process in a graphical form where an input signal305is transformed into an output signal350.

Secondly, the method may benefit from facts that voice activities come in segments and contain a lot of silences. In experience, the total span of voice activities in a multi-party conference is no longer than two times of the conference's duration. Thus each incoming remote participant's signal is monitored by a voice activity detector which typically has very low complexity. The spatial processing only takes place where actual speech activity is detected. Consequently, this further trims the algorithm complexity to0(2M+N). Note that synthesis now has bounded complexity independent of the total number of participants. The significance of this reduction is better appreciated in the context of real-world implementation where unbounded computational cost can not be tolerated. Once the first left channel function is created, at block230, it may be stored in a memory.

At block410, a first right channel function may be created to modify the first signal305to minimize the difference between the first signal305and the first right channel received signal240. In one embodiment, a Fourier transform is used to create the function to modify the first signal305. Of course, other method to create the first right channel function to modify the first signal305to minimize the difference between the first signal305and the first received right channel signal340are possible and are contemplated. Once the first right channel function is created, at block240, it may be stored in a memory.

At block420, a first modified conference signal may be created where the first modified conference signal comprises a modified first left channel and a modified first right channel by applying the first left channel function to a first conference call signal to create the modified first left channel and applying the first right channel function to the first conference call signal to create the modified first right channel.

At block430, the first modified conference call signal my be communicated to a user. On some situations, the user may have headphones or a telephone with stereo speakers which may make the directional effect even more pronounced. The communication may occur using traditional POTS (plain old telephone service) or VoIP (voice over Internet Protocol) or any appropriate communication medium or scheme. In some embodiments, as a two channel (left right) signal may be communicated which may require some additional processing by the telephone systems.

In some embodiments, the will be more than one caller on a conference call. The second call may be treated in a similar way as the first. A possible difference is that the second source330will likely be at a different location345than the first source310. More specifically, a second signal350from a second source330at a second location345wherein the second location345is different than the first location315. The second signal350may be received at the hearing location323where the second signal350is received in a left channel335and a right channel340located at the hearing location323. The received left channel335at the hearing location of the second signal350may be stored as a left received signal335of the second signal350in a memory. The right channel340of the second received signal350at the hearing location323maybe stored as a right received signal340of the second signal350in a memory. The second location345may be stored in a memory where the second location345may include a location in relation to the hearing location323. A second left channel function may be created to modify the second signal350to minimize the difference between the second signal350and the left channel received signal335of the second signal350. The second left channel function may be stored in a memory. Similarly, a second right channel function may be created to modify the second signal350to minimize the difference between the second signal350and the right channel received signal340of the second signal350. The second right channel function may be stored in a memory.

A second modified conference call may be created where the second modified conference call may include a modified second left channel and a modified second right channel by applying the second left channel function to a second conference call signal350to create the modified second left channel and applying the second right channel function to the conference call signal350to create the modified second right channel. The first modified conference signal and the second modified conference signal may be combined to create a modified conference signal and the modified conference signal may be communicated to the user.

Combining the first modified conference signal and the second modified conference signal may occur in any logical sounding combining methodology. Logically, the modified first left channel and the modified second left channel may be combined into a combined modified left channel and the modified first right channel and the modified second right channel may be combined into a combined modified right channel.

In another embodiment, first location315of the first signal305may be varied to be different degrees off center from the hearing location323in order to create a variety of functions to reflect signals coming from a variety of angles. In application, the variety of location may be used to mimic people sitting around a table at a conference such as illustrated inFIG. 5, with each location505-525having a different function to modify the left335and right channels340. In order to make the functions, the specific location505-525may be stored, an embodiment of the method such as the one described inFIG. 3may be started, the resulting first left channel function may be stored in a memory available to be searched and the resulting first right channel function may be in a memory available to be searched.

The various functions may be used in a variety of ways. If there are two callers, one may be at 90 degrees off center and the second may be at −90 degrees (or 270 degrees) to enhance the spatial effect of the embodiments of the method. If there are four callers, one may be at −90 degrees (270 degrees), a second at −30 degrees (330 degrees), a third at 30 degrees and a fourth at 90 degrees from a center line to further enhance the spatial effects. As can be imagined, the more locations that are sampled and related functions that are created, the more options are available to increase the spatial effects and provide a more spatially enhanced telephone experience.

As with any conference call, there is no requirement that all the callers sit around a round table as is illustrated inFIG. 5. For example, caller505may be in Bangalore, India, caller510may be in Paris, France, caller515may be in London, England, caller520may be in New York and caller525may be in San Francisco, Calif. and the listener323may be in Chicago, Ill. However, in the listener's ear, the illusion may be created, by applying the various modification functions in a logical manner, that each caller505-525is sitting around a round table. Of course, the functions may be created to provide the illusion that the callers are sitting around a square table, a rectangular table, up in balconies, in a concert hall, in a stadium, etc. The variety of environments that can be analyzed and mimicked using the functions is virtually limitless.

In some embodiments, the method may interpolate between sampled locations505-525to determine left channel functions and right channel functions at locations between sampled locations505-525. Various methods may be used to interpolated such as a weighting scheme or a least squares difference scheme. Of course, other schemes are possible and are contemplated.

In some embodiments, the method may be able to tell if a user turns their head, such as to face the person that is talking. In one embodiment, the user wears headphones and the headphones have motion sensors. Referring toFIG. 5, the centerline325originally pointed toward source515, with source520being 30 degrees off the centerline325and source525being 60 degrees off the centerline325. InFIG. 6, the listener has turned toward source520. The centerline325then adjusts to have source520at 0 degrees and source525is now at 30 degrees off the centerline325and source515is −30 degrees (330 degrees) off the centerline325. Similar to real life, as the listener turns their head to face a speaker505-525, the centerline may adjust and the relative locations of the sources505-525may also adjust accordingly. Once the relative position of the sources505-525is established in relation to the listener, an appropriate the right and left function may be selected that best match the degrees in relation to the new centerline325.

In conclusion, the detailed description is to be construed as exemplary only and does not describe every possible embodiment since describing every possible embodiment would be impractical, if not impossible. Numerous alternative embodiments could be implemented, using either current technology or technology developed after the filing date of this patent, which would still fall within the scope of the claims.