VOIP device, VOIP conferencing system, and related method

Voice over internet protocol (VoIP) devices and conferencing systems may include a spatial encoder associated with a first endpoint and a spatial renderer associated with a second endpoint. The spatial renderer may configured to receive audio data. The audio data may be rendered among a plurality of speakers based on a first set of spatial information for a plurality of microphones associated with the first endpoint, and a second set of spatial information for the plurality of speakers associated with the second endpoint. A method for generating a sound field may include determining spatial information for a plurality of microphones in a local room, determining spatial information for a plurality of speakers in a remote room, mapping the spatial information for the plurality of microphones and the spatial information for the plurality of speakers, and generating a sound field in the remote room based on the mapping.

TECHNICAL FIELD

The present disclosure generally relates to a voice over internet protocol (VoIP) device. More particularly, embodiments of the present disclosure relate to spatially rendering audio data from a local room to a remote room.

BACKGROUND

Voice over internet protocol (VoIP) is a technique for delivering voice information using an internet protocol (IP) network, such as the Internet. Generally, VoIP is not a traditional protocol based on a link, as in a public switched telephone network (PSTN), but rather is a protocol that transmits voice information in a digital form within discrete packets. In conventional VoIP devices, audio data may be mixed, encoded, and transmitted from one VoIP device to another VoIP device, such as in a VoIP conferencing application. For situations in which the microphone elements and the speaker elements to not align on a point-to-point basis, the sound field produced in a remote room may lose the effect of the location of the sound source within the local room.

SUMMARY

Embodiments of the present disclosure include a voice over internet protocol (VoIP) device. The VoIP device comprises a spatial renderer associated with a second endpoint, the spatial renderer configured to receive audio data from a first endpoint. The spatial renderer is further configured to render the audio data among a plurality of speakers based, at least in part, on a first set of spatial information for a plurality of microphones associated with the first endpoint, and a second set of spatial information for the plurality of speakers associated with the second endpoint.

Another embodiment of the present disclosure includes a voice over internet protocol (VoIP) conferencing system. The VoIP conferencing system comprises a local endpoint comprising a spatial encoder configured to send a first set of spatial information to a remote endpoint, and receive a second set of spatial information from a spatial renderer of the remote endpoint. The first set of spatial information includes relative position information for a plurality of microphones. The second set of spatial information includes relative position information for a plurality of speakers.

A particular embodiment includes a method of generating a sound field in a remote room from captured audio signals in a local room. The method comprises determining spatial information for a plurality of microphones in the local room, determining spatial information for a plurality of speakers in the remote room, mapping the spatial information for the plurality of microphones and the spatial information for the plurality of speakers, and generating a sound field in the remote room based on the mapping.

DETAILED DESCRIPTION

In the following description, reference is made to the accompanying drawings in which is shown, by way of illustration, specific embodiments of the present disclosure. Other embodiments may be utilized and changes may be made without departing from the scope of the disclosure. The following detailed description is not to be taken in a limiting sense, and the scope of the present invention is defined only by the appended claims.

Furthermore, specific implementations shown and described are only examples and should not be construed as the only way to implement or partition the present disclosure into functional elements unless specified otherwise herein. It will be readily apparent to one of ordinary skill in the art that the various embodiments of the present disclosure may be practiced by numerous other partitioning solutions.

In the following description, elements, circuits, and functions may be shown in block diagram form in order not to obscure the present disclosure in unnecessary detail. Additionally, block definitions and partitioning of logic between various blocks is exemplary of a specific implementation. It will be readily apparent to one of ordinary skill in the art that the present disclosure may be practiced by numerous other partitioning solutions. Those of ordinary skill in the art would understand that information and signals may be represented using any of a variety of different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be referenced throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof. Some drawings may illustrate signals as a single signal for clarity of presentation and description. It will be understood by a person of ordinary skill in the art that the signal may represent a bus of signals, wherein the bus may have a variety of bit widths and the present disclosure may be implemented on any number of data signals including a single data signal.

The various illustrative logical blocks, modules, and circuits described in connection with the embodiments disclosed herein may be implemented or performed with a general-purpose processor, a special-purpose processor, a Digital Signal Processor (DSP), an Application-Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general-purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A general-purpose processor may be considered a special-purpose processor while the general-purpose processor executes instructions (e.g., software code) stored on a computer-readable medium. A processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

Also, it is noted that the embodiments may be described in terms of a process that may be depicted as a flowchart, a flow diagram, a structure diagram, or a block diagram. Although a process may describe operational acts as a sequential process, many of these acts can be performed in another sequence, in parallel, or substantially concurrently. In addition, the order of the acts may be re-arranged. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. Furthermore, the methods disclosed herein may be implemented in hardware, software, or both. If implemented in software, the functions may be stored or transmitted as one or more instructions or code on computer readable media. Computer-readable media includes both computer storage media and communication media, including any medium that facilitates transfer of a computer program from one place to another.

A “stream” and a “channel” may be used interchangeably herein. A “microphone” and a “microphone element” may be used interchangeably herein. A “speaker” and a “speaker element” may be used interchangeably herein.

FIG. 1is a schematic block diagram of a VoIP engine100according to an embodiment of the present disclosure. The VoIP engine100may be configured to provide a connection between an audio conferencing system and a VoIP-based private branch exchange (PBX) phone system so that a user may transport audio signals across an IP network. The VoIP engine100may deliver wideband audio signals having transport layer security (TLS), advanced encryption standard (AES), secure real-time transport protocol (SRTP), or other similar encryption. In addition, the VoIP engine100may be further configured to provide a session initiated protocol (SIP) transport method for connecting with a far-end device, such as in a far-end conference room.

The VoIP engine100includes an application processor110, a distribution processor120, and a processor farm130. The application processor110may be coupled to the distribution processor120, which may be coupled to the processor farm130.

The application processor110may be configured as a main processor running the application code for the VoIP engine100, as well as being configured to function as the voice engine for the VoIP engine100. The application processor110may be configured to perform a variety of different audio processing operations, such as data compression, handling the communication protocol (e.g., session initiation protocol (SIP)), etc. As an example, the application processor110may be a microprocessor having an integrated silicon platform (e.g., system on a chip) configured for VoIP and broadband applications, such as the TNETV1050 microprocessor available from Texas Instruments, Inc. of Dallas, Tex. Other similar microprocessors may also be employed to implement embodiments of the disclosure.

The distribution processor120may be configured to distribute commands between the application processor120to the processor farm130. For example, in one operation (e.g., the VoIP engine100receiving data through Ethernet, USB, or other external port), the application processor110may transmit signals (e.g., commands, data, etc.) to the processor farm130. The signals may be transmitted through an address bus111(e.g., parallel address bus) and a first communication bus113(e.g., multi-channel buffered serial port (McBSP)) to the distribution processor120, and the distribution processor120may distribute the signals to the processor farm130via one or more communication buses121,123. The one or more communication busses121,123may include a serial peripheral interface (SPI) bus121or a time-division multiplexing (TDM) bus123. In the reverse operation (e.g., the VoIP engine100transmitting data through Ethernet, USB, or other external port), the distribution processor120may distribute the signals from the processing farm130to the application processor110.

In addition, the distribution processor120may be further configured to perform other operations, such as format conversion of the signals, in addition to simply distributing the signals. The distribution processor120may be further configured to serve as the master time source and router for the audio slots associated with the various audio buses. The audio signals received by the processor farm130may be sampled and processed at a sampling rate and resolution. The application processor110may also be operated at a certain rate and resolution. For example, audio samples may be processed using 24 bit resolution at 48 KHz in the processor farm130, and using 16 bit resolution at 16 KHz in the application processor110. In embodiments using different sampling rates, the distribution processor120may include a sample rate converter.

The processor farm130may include a plurality of processing resources132configured for conferencing, transcoding, and media termination point (MTP) services. For example, the processor farm130may be configured to handle operations, such as echo cancellation, gain control, noise reduction, etc. In some embodiments, the processor farm130may be a DSP farm including a plurality of DSP resources.

Data and commands may be sent over the McBSP bus113between the application processor110and the distribution processor120. Between the distribution processor120and the processor farm130, data may be sent over the TDM bus123and commands may be sent over the SPI bus121. In some embodiments, the distribution processor120may be configured to act as a memory bank to temporarily store information during operation of the VoIP engine100. The address bus111may be used to provide addresses to the distribution processor120to access such information.

Thus, to receive audio streams, the VoIP engine100may receive the audio streams at the application processor110(e.g., via Ethernet, USB, etc), which may decompress the audio streams, decode, and perform other VoIP processing. The audio streams may be further transmitted to the processor farm130for other processing, such as noise cancellation. In transmit mode for transmitting audio streams, sound may be captured by microphone elements coupled with the processing farm130, which may sample the audio signals and send the audio signals to the application processor110. The application processor may perform compression of the audio signals, encoding, packetizing, and other VoIP processing. The compressed audio stream may be transmitted to a remote endpoint (e.g., via Ethernet, USB, etc.).

The VoIP engine100may be further configured to determine spatial information for the microphone elements and/or speaker elements associated therewith. As a result, in transmit mode as a local endpoint, the VoIP engine100may be configured to determine how to encode audio data for maintaining an appropriate spatial effect for the audio data having knowledge of the spatial information of the speaker elements of the remote endpoint. In receive mode, the VoIP engine100may be configured to determine how to render the audio data to produce a sound field that maintains at least some of the spatial effects. Additional detail regarding such spatial rendering is described below with reference toFIGS. 8 through 14.

FIG. 2is a schematic block diagram of a VoIP engine200according to an embodiment of the present disclosure. The VoIP engine200includes the application processor110, the distribution processor120, and the processor farm130(FIG. 1), which may be configured generally as described above. The application processor110further includes a session initiation protocol (SIP) user agent212, an internal processor214, and a McBSP port216.

The SIP user agent212may include a processor configured to manage a SIP session. The SIP user agent212may receive data from or transmit data to a plurality of different callers202over an external link (e.g., Ethernet, USB, etc.). The data streams between the callers202and the SIP user agent may include real-time transport protocol (RTP) packets (i.e., RTP streams).

The internal processor214may be a DSP core (e.g., C55 processor) processor for the application processor110. The internal processor214may include a plurality of codecs215used to process the data. In particular, the codecs215may be configured to encode and decode the data input and output from the application processor110. As an example, the codecs215may configured for compression and/or decompression of data depending on the flow of data. The codecs215may also be configured for various communications standards, including G.722, G.711, and G.729AB.

The codecs215may be further configured for supporting other features, such as audio coding-low delay (AAC-LD). Such a feature may be configured as a single instance in a point-to-point call, which may result in an increase of the resolution and sampling rate at the application processor110. For example, the resolution may be increased to 24 bit resolution at a sampling rate of 48 kHz. As a result, the resolution and sampling rate of the application processor110and the processor farm130(FIG. 1) may be the same, such that a full sample from the processor farm130may be transmitted from the VoIP engine200rather than a down-converted sample. As a result, the full sample may provide uncompressed RTP data transmitted out of the VoIP engine200to another device as a full band audio stream covering the full range of human hearing.

With data being received from a plurality of callers202, the VoIP engine200may be configured for supporting a bridge call. For example, the plurality of data streams from each caller202may be received as independent audio samples on dedicated audio slots. The mixing of the plurality of data streams may be performed within the processor farm130(FIG. 1).

FIG. 3is a schematic block diagram VoIP engine300according to another embodiment of the present disclosure. The VoIP engine300includes an application processor110, a distribution processor120, and a processor farm130configured generally as described before. The VoIP engine300may further include another application processor310operably coupled with the distribution processor120. The application processor310may be a dedicated processor configured for mixing data received by the first application processor110and the processor farm130.

The data from received by the first application processor110(e.g., via Ethernet, USB, etc.) may remain uncompressed for mixing with the data from the processor farm130. For example, an audio signal may be received from either the USB port or the Ethernet port to the first application processor110. The audio signal may be transmitted to the second application processor310via the distribution processor120and the McBSP busses113,313. The processor farm130may also have microphone inputs such that the processor farm130may also receive an audio signal that is transmitted to the second application processor310via the distribution processor120, the TDM bus123and the McBSP bus313.

Embodiments of the present disclosure may further include VoIP devices that are configured to generate a spatial audio representation from a local room to a far end room. To enable such a spatial audio representation, the devices may include a VoIP transport layer configured to send multiple audio streams to the far end site.

FIG. 4is a schematic representation of a point-to-point surround sound call400between two rooms410,420according to an embodiment of the present disclosure. The first room410includes a plurality of microphones412for transmitting RTP streams of audio data to a plurality of speakers422in the second room420. Each of the plurality of microphones412and the plurality of speakers422may be coupled to, or incorporated within, a VoIP engine that includes other processing components as described above. As shown inFIG. 4, each audio channel corresponds to an individual microphone412and speaker422in a point-to-point manner. The four audio streams may be implemented in a surround mode with independent audio streams, while in some embodiments, the one or more of the audio streams may be implemented in a stereo mode that includes a left channel and right channel in the encoded packet. In addition to audio data, additional data may be transmitted, such as metering data used for room acoustical signal processing. Such additional data may be transmitted over a separate real-time control protocol (RTCP) channel.

FIG. 5is a schematic representation of a point-to-point surround sound call500between two rooms510,520according to an embodiment of the present disclosure. The first room510includes a plurality of microphones512for transmitting RTP streams of audio data to a plurality of speakers522in the second room520. Each of the plurality of microphones512and the plurality of speakers522may be coupled to, or incorporated within, a VoIP engine that includes other processing components as described above. As shown inFIG. 5, each audio channel corresponds to an individual microphone512and speaker522in a point-to-point manner. The two audio streams may be implemented in a stereo mode with having a left channel and a right channel in the encoded RTP packets of the RTP streams.

FIG. 6is a schematic representation of a point-to-point mono sound bridge call600between four near rooms610A,610B,610C,610D and a far room620according to an embodiment of the present disclosure. Each of the near rooms610A,610B,610C,610D include a microphone612A,612B,612C,612D, respectively, for transmitting RTP streams of audio data to a plurality of speakers622in the far room620. Each of the plurality of microphones612A,612B,612C,612D and the plurality of speakers622may be coupled to, or incorporated within, a VoIP engine that includes other processing components as described above. As shown inFIG. 6, each audio channel corresponds to an individual microphone612A,612B,612C,612D and speaker622in a point-to-point manner.

FIG. 7is a schematic representation of a point-to-point stereo sound bridge call700between four near rooms710A,710B,710C,710D and a far room720according to an embodiment of the present disclosure. Each of the near rooms710A,710B,710C,710D include a microphone712A,712B,712C,712D, respectively, for transmitting RTP streams of audio data to a plurality of speaker pairs722in the far room720. Each of the plurality of microphones712A,712B,712C,712D and the plurality of speaker pairs722may be coupled to, or incorporated within, a VoIP engine that includes other processing components as described above. As shown inFIG. 7, each audio channel corresponds to an individual microphone712A,712B,712C,712D and a speaker pair722in a point-to-point manner. Each audio stream may include two channels (e.g., a right channel and a left channel) corresponding to the two speakers of the speaker pair722for the encoded RTP packets of the RTP streams as a stereo bridge call.

FIG. 8is a schematic block diagram representing a call setup800involving a plurality of endpoints810,820according to an embodiment of the present disclosure. Although two endpoints810,820are shown, embodiments are contemplated that may include any number of endpoints. The endpoints810,820may include one or more microphone elements and/or speaker elements and other components that are included within a VoIP engine. The microphone elements of one endpoint (e.g.,810) may be located at different relative positions than the speaker elements of the other endpoint (e.g.,820). In addition, the number of microphone elements of one endpoint (e.g.,810) may not equal the number of speaker elements of the other endpoint (e.g.,820). In other words, the microphone elements and the speaker elements may not correspond to a point-to-point basis with each other either in number or in relative locations.

In some of the embodiments, a conferencing situation is described in which the first endpoint810is described as having microphones, while the second endpoint820is described as having speakers. In addition, the first endpoint810having microphones may be described as a local endpoint, while the second endpoint820having speakers may be described as a remote endpoint. Such a description should not be construed to imply that the first endpoint810does not have speakers, or that the second endpoint820does not have microphones. In many conferencing situations, each endpoint810,820may both send and receive audio signals, and may act as a local endpoint and a remote endpoint during the same call.

The plurality of endpoints810,820may establish a call (e.g., a conference call) therebetween. For example, the first endpoint810may initiate a call with the second endpoint820. During call set up800, the endpoints810,820may pass call setup messages therebetween. Call setup messages may include information regarding the media capabilities of the endpoints810,820. For example, media capabilities may include the type of media (e.g., audio, video) supported by the endpoints810,820, as well as other information, such as formats supported, number of channels supported, which ports may be used for communication, etc. Media capabilities may be transmitted using Media Descriptions and SDP Attributes in SIP, Capability Exchange in H.323 protocol, or using other similar methods based on the media transport protocol.

A call set up message may further include spatial information regarding how the room for each endpoint810,820is set up. Spatial information may include information regarding the relative positions and orientations of the speaker elements and microphone elements relative to a coordinate system within the rooms, as well as other information, such as directionality of the microphone elements and speaker elements. Such information may be defined by an installer of the audio system. In some embodiments, at least some of the spatial information may be dynamic, and may change over time during use of the audio system. In some embodiments, additional information regarding how sound may be reflected or absorbed within the room may be shared by the endpoints810,820. Such information may include, for example, dimensions and materials that are present in the room and may enable modeling of reflections within the room. Such information may assist in the spatial rendering to more accurately reconstruct the audio signals and maintain the spatial sound effects.

FIG. 9is a media path900for a plurality of endpoints902,904according to an embodiment of the present disclosure. The plurality of endpoints920,904may be coupled together through an IP network905. The first endpoint902may include a spatial encoder910, an audio encoder920, and an RTP packetizer930. The second endpoint904may include an RTP depacketizer940, an audio encoder950, and a spatial renderer960. As discussed above, the first endpoint902may include a plurality of microphone elements, and the second endpoint904may include a plurality of speaker elements. As discussed above, the first endpoint902may be referred to as the local endpoint, and the second endpoint904may be referred to as the remote endpoint904. The components of each of the endpoints902,904may be incorporated within, a VoIP engine that may use the processing components described above to perform one or more of the functions described herein.

The spatial encoder910may capture audio data from the plurality of microphone elements and encode the audio data into separate audio streams. The input audio data may be mixed and encoded into a number of channels (i.e., streams) supported by the remote endpoint904, and based on the spatial rendering capabilities and output locations of the speaker elements for the remote endpoint904.

The audio encoder920may receive the individual audio streams from the spatial encoder910and compress the audio streams into different channels in a known audio transport protocol (e.g., AAC-LD, G.722, etc.).

The RTP packetizer930may receive the compressed individual audio streams from the audio encoder and form the compressed individual audio streams into RTP packet streams to be sent over the IP network905to the remote endpoint904, such as by using RTP/RTCP protocol for such transmission.

Referring to the second endpoint904, the RTP depacketizer940may receive the RTP packet streams from the first endpoint902. The RTP depacketizer940may further handle jitter buffering between RTP packets for each RTP packet stream, and time align the outputs handed off to the audio decoder950.

The audio decoder950may receive each RTP packet stream from the RTP depacketizer940, and decode each RTP packet stream from a known audio transport protocol format to a supported uncompressed format that may be used by the spatial renderer960.

The spatial renderer960may receive the individual uncompressed data streams from the audio decoder950, and reconstruct the individual uncompressed data streams to be played out on speaker elements of the second endpoint904. The spatial renderer may render the audio signals based on the spatial information for the microphone elements and the speaker elements in their respective rooms. As a result, the audio outputs played by the speaker elements of the second endpoint904may sound like it is being replicated spatially in the remote room as it was captured in the local room.

In summary, the spatial encoder910may define how each audio stream is created (e.g., including mixing the raw audio data from various individual microphones). The spatial encoder910may also map each audio stream to a particular location in the remote room to which the audio stream would be targeted. In other words, the local endpoint902may be configured to recommend placement of the audio stream within the sound field of the remote endpoint904. The spatial renderer960may receive each audio stream and know which part of the remote room the audio stream is targeted for. The spatial renderer960may segment the remote room and determine which audio stream is played by which speaker element, as well as perform other controls such as adjusting volume and/or direction for the speaker elements located in the different targeted areas. Such a determination may be assisted by geometrically mapping coordinate systems for each set of spatial information. Defining coordinate systems and determining the spatial information will be discussed in further detail below.

In some embodiments, the raw audio data for each microphone element may be sent to the remote endpoint904without spatial encoding. In such an embodiment, the remote endpoint904may perform all processing with regard to spatial rendering (including mixing) having each set of spatial information as well as the raw audio data. As a result, the spatial encoder910may not be needed for such an embodiment. There may be certain advantages, however, to performing spatial encoding prior to transmitting audio data to the remote endpoint904. For example, the remote endpoint904may have limited processing capabilities to perform such mixing of audio data. In addition, the remote endpoint904may have a limited number of channels available for receiving audio data. As a result, spatial encoding prior to sending audio data may keep more of the audio signal intact.

In some embodiments, the local room associated with the first endpoint902may have more microphone elements than speaker elements in the remote room associated with the second endpoint904. For example, the local room may have ten microphones spaced in a grid configuration. The remote room may have two speakers that are configured to provide two stereo (L/R) channels to the remote room. The first endpoint902and the second endpoint904may share spatial information regarding their respective elements during call setup and negotiation. The spatial encoder904may determine how to mix the audio source data to generate the appropriate number of channels and in the appropriate manner to maintain spatial effects of the audio sources. For example, the spatial encoder may mix the audio signals from the five left-most microphone elements into a left channel, and mix the audio signals from the five right-most microphone elements into a right channel. The spatial renderer960may receive the mixed audio signals in the left channel and right channel, and, using the spatial information, the spatial renderer960may determine the appropriate speaker elements for playback as well as other playback characteristics.

In some embodiments, the local room may include fewer microphone elements than speaker elements in the remote room. Using the spatial information, the spatial encoder910may determine how to mix the audio source data to generate the appropriate number of channels and in the appropriate manner to maintain spatial effects of the audio sources. The spatial renderer960may receive the mixed audio signals, and, using the spatial information, the spatial renderer960may determine the appropriate speaker elements for playback as well as other playback characteristics.

In some embodiments, the local room and the remote room may have the same number of microphone elements and speaker elements. While it may be possible to have a one-to-one correspondence of channels between the two endpoints902,904(see, e.g.,FIGS. 4 and 5), the relative locations of the microphone elements and speaker elements may be not match from one room to the other room. Thus, without using spatial information for spatial rendering the audio signals, at least some (if not most) spatial effects of the sound may be lost. As a result, using the spatial information, the spatial encoder910may determine an appropriate mix for the audio signals based on the spatial information of both the microphone elements and the speaker elements. Likewise, the spatial renderer920may use the spatial information to determine the appropriate rendering on the remote endpoint904.

While certain numbers and configurations are described and shown in the various figures herein, any number or configuration is contemplated as an embodiment of the present disclosure. Such numbers and configurations should not be limited unless specifically described as being so limited. In addition, the different coordinate systems chosen for each room may be roughly the same. As a result, the coordinate systems may be superimposed on each other when mapping the coordinate systems and the spatial information during spatial encoding and spatial decoding. In some embodiments, the coordinate systems may be roughly the same, but that the mapping may be performed by creating a mirror image of the other coordinate system during spatial encoding and spatial decoding. In some embodiments, the coordinate systems may be substantially different (e.g., the rooms have substantially different dimensions). As a result, a more complex mapping (or a simplified estimate of one or more of the rooms) may be performed during spatial encoding and spatial decoding. In some embodiments, room sizes may be substantially different, and a scaling factor may be applied such that the sound may be reproduced in the remote room to sound either closer or further away (as the case may be) than may actually be the case in the local room.

FIG. 10is an illustration of a room1000including a plurality of speakers1020A-1020H according to an embodiment of the present disclosure. Each of the plurality of speakers1020A-1020H may be coupled to, or incorporated within, a VoIP engine that includes other processing components as described above. The plurality of speakers1020A-1020H, in conjunction with a VoIP engine, may be configured to output spatially encoded audio signals received from an endpoint having a plurality of microphones. In particular,FIG. 10shows an example of how the spatial information regarding the speakers may be defined relative to a coordinate system. The coordinate system may be defined as an XY axis. As an example, the relative locations of the speakers may be defined in polar coordinates. For example, the first speaker1020A may have spatial information associated therewith, such as a radius (R1) and an angle relative to the coordinate system. The radius (R1) may be measured from the origin of the coordinate system, and the angle (θ1) may be measured from the positive X axis of the coordinate system. Each of the other speakers1020B-1020H may have spatial information associated therewith, which may be defined in a similar manner. Of course, other coordinate systems and coordinate systems may be employed for determining positions of the speakers relative to each other using a coordinate system. For example, Cartesian coordinates may be employed, such that an (X,Y) coordinate for each speaker1020A-1020H may be determined.

In some embodiments, the speakers1020A-1020H may be non-directional (in other words, they are omni-directional) and generate sound in all directions rather than favoring a specific direction. For example, the speakers1020A-1020H may be configured as ceiling speakers in the room1000. In some embodiments, speakers may be directional and generate sound to favor a specific direction. For example, speakers may be wall-mounted speakers, mounted to a video screen, etc., and may point in a direction that is different than toward the horizontal plane. In some embodiments, speakers may include a speaker array configured to generate a beam pattern such that the beam pattern is directional. Such speakers may further be configured to steer the beam pattern to be directed and favor a particular direction. In such embodiments including directional speakers, the spatial data may further include directional data in addition to location and orientation of the speakers. In addition, for embodiments where the beam pattern may be steered, the directional data may be combined with the other spatial data during rendering of the incoming audio signals such that the rendering may include steering the beam patterns of one or more of the speakers.

FIG. 11is an illustration of a room1100including a plurality of microphones1110A-1110E according to an embodiment of the present disclosure. Each of the plurality of microphones1110A-1110E may be coupled to, or incorporated within, a VoIP engine that includes other processing components as described above. The plurality of microphones1110A-1110E, in conjunction with a VoIP engine, may be configured to capture and send outgoing audio signals to a remote endpoint. In particular,FIG. 11shows an example of how the spatial information regarding the microphones1110A-1110E may be defined relative to a coordinate system. The coordinate system may be defined as an XY axis. As an example, the relative locations of the microphones1110A-1110E may be defined in polar coordinates. For example, the fourth microphone1110D may have spatial information associated therewith, such as a radius (R4) and an angle relative to the coordinate system. The radius (R4) may be measured from the origin of the coordinate system, and the angle may be measured from the positive X axis of the coordinate system. In the example ofFIG. 11, because the fourth microphone1110D is located on the positive X axis, the angle is 0, and therefore, not specifically shown. Each of the other microphones1110A,1110B,1110C,1110E may have spatial information associated therewith, which may be defined in a similar manner. Of course, other coordinate systems and coordinate systems may be employed for determining positions of the speakers relative to each other using a coordinate system. For example, Cartesian coordinates may be employed, such that an (X,Y) coordinate for each microphone1110A-1110E may be determined.

In some embodiments, the microphones1110A-1110E may be non-directional (in other words, they are omni-directional) and generate sound in all directions rather than favoring a specific direction. For example, the microphones1110A-1110E may be located along a table1102in the room1100, such that sound may be captured from participants on each side of the table1102.

FIG. 12is an illustration of a room1200including a microphone1210according to an embodiment of the present disclosure. The microphone1210may be coupled to, or incorporated within, a VoIP engine that includes other processing components as described above. The microphone1210, in conjunction with a VoIP engine, may be configured to output outgoing audio signals to a remote endpoint. The microphone1210may be configured as a beam forming microphone array. Thus, the microphone1210will also be referred to as a microphone array1210. In particular, the microphone1210may include a plurality of microphones that are directional and capture sound within a field illustrated by a lobe1212.FIG. 12shows an example of how the spatial information regarding the individual directional microphones of the microphone array1210may be defined relative to a coordinate system. The microphone array1210may be located along a table1202in the room1200, such that sound may be captured from participants on each side of the table1202.

As in the previous examples, the coordinate system may be defined as an XY axis, and the relative locations of the individual microphones of the microphone array1210may be defined in polar coordinates. For example, each individual microphone of the microphone array1210may have spatial information associated therewith, such as a radius and an angle relative to the coordinate system. The radius may be measured from the origin of the coordinate system, and the angle may be measured from the positive X axis of the coordinate system. In addition to the radius and angle representing the location of the individual microphone, the spatial information may further include additional angles (ζ, ±Φ) representing directional characteristics of the beam. For example, the directional angle (ζ) may define the angle between the positive X axis and the general direction1215of the beam represented by the lobe1212. The width angles (±Φ) represent the width of the beam between lines1211,1213. These additional angles (ζ, ±Φ) may be transmitted to the remote endpoint with the spatial information in order to further assist in spatial rendering of the audio signals.

FIG. 13is an illustration of a room1300including a plurality of microphones1310A-1310E according to an embodiment of the present disclosure. Each of the plurality of microphones1310A-1310E may be coupled to, or incorporated within, a VoIP engine that includes other processing components as described above. The plurality of microphones1310A-1310E, in conjunction with a VoIP engine, may be configured to capture and send outgoing audio signals to a remote endpoint. As discussed in the previous examples, the spatial information regarding the microphones1310A-1310E may be defined relative to a coordinate system (e.g., XY axis). In the example shown inFIG. 13, the origin for the coordinate system may be at a location different than near the center of the room1300. For example, the room1300may be a conference room set up for video conferencing having a video screen1304near one end of the room1300. The origin for the coordinate system inFIG. 13may be proximate the front of the room1300near the video screen1304.

The room1300also shows a plurality of speakers1320A,1320B positioned proximate the video screen1304. Each of the plurality of speakers1320A,1320B may be coupled to, or incorporated within, a VoIP engine that includes other processing components as described above. The audio signals captured by the plurality of microphones1310A-1310E may be spatially rendered and output to the plurality of speakers1320A,1320B of a remote endpoint. The plurality of speakers1320A,1320B may be directional speakers, which information may be sent to the microphones1310A-1310E along with the spatial information. It should be understood that the speakers1320A,1320B may be part of a remote room that is different than the room1300including the plurality of microphones1310A-1310E. In other words, the room1300is shown to act as both the local endpoint as well as the remote endpoint, for convenience in illustration to not include a figures for both. Of course, embodiments may include a single room that both captures audio signals through a plurality of microphones as well as outputs the spatially rendered audio signals within the same room.

In some embodiments, when the audio signals are spatially rendered, the coordinate system associated with the microphones1310A-1310E may be constructed as a mirror image of the reference from for the speakers1320A,1320B. As a result, the sound projected from the speakers1320A,1320B may provide sound to the remote room such that the location of the person speaking in the local room may be apparent to the listener in the remote room. As a result, such an embodiment may provide an effect to the listeners in the remote room as if they were watching a video conference through a window rather than a two dimensional screen.

In some embodiments, a video conference may be configured to include automatic camera tracking based on which microphone is gated on. For example, the camera (not shown) that captures the video for a video conference may be focused to pan and zoom to the person speaking into a microphone (e.g., a microphone may be gated on or off when sound is detected). Having spatial rendering of the audio signals may also enable increasing the gain to an appropriate audio channel during spatial rendering, which may have the effect of the person sounding closer than they actually are.

FIG. 14is an illustration of a room1400including a plurality of microphones1410A-1410P according to an embodiment of the present disclosure. Each of the plurality of microphones1410A-1410P may be coupled to, or incorporated within, a VoIP engine that includes other processing components as described above. The plurality of microphones1410A-1410P, in conjunction with a VoIP engine, may be configured to capture and send outgoing audio signals to a remote endpoint. As discussed in the previous examples, the spatial information regarding the microphones1410A-1410P may be defined relative to a coordinate system (e.g., XY axis). In the example shown inFIG. 14, the origin for the coordinate system may be at a location different than near the center of the room1400. The origin for the coordinate system inFIG. 14may be proximate the front of the room1400. For example, the room1400may be an auditorium or other room (e.g., lecture hall) set up for the primary sound source may be near one end of the room1400. The microphones1410A-1410P may be positioned at locations associated with a plurality of tables1402A-1402H. The room1400also shows a plurality of speakers1420A-1420H distributed along the ceiling of the room1400. Each of the plurality of speakers1420A-1420H may be coupled to, or incorporated within, a VoIP engine that includes other processing components as described above.

The audio signals captured by the plurality of microphones1410A-1410H may be spatially rendered and output to the plurality of speakers1420A-1420H of a remote endpoint. The plurality of speakers1420A-1420H may be omni-directional speakers. As withFIG. 13, a single room is shown having both the plurality of microphones1410A-1410H and speakers1420A-1420H. A local endpoint and a remote endpoint for spatially rendering audio signals may be located within the same rooms or different rooms.

In some embodiments, when the audio signals are spatially rendered, the coordinate system associated with the microphones1410A-1410may be constructed as being superimposed with the coordinate system for the speakers1420A-1420H. As a result, with the plurality of speakers1420A-1420H may provide the effect of being spatially accurate. For example, a person speaking near the front of the room1400may translate to the speaker1420A being louder, with the other speakers1420B,1420C,1420D fading out going down the line.

Various embodiments have been described including the locations of the speakers and the microphones being fixed relative to the coordinate system defined for the rooms. In some embodiments, the speakers and/or the microphones may be movable throughout the room, such that the spatial information may be dynamically changing and sent from endpoint to endpoint with the audio streams.

The angular and radial information for the microphone may be determined based on determining a position of the microphone within the room. For example, various sensors, transmitters, etc. may be used to determine positional data from which the radial and angular information may be determined. Direction and orientation information may also change dynamically and may be re-sent as such information changes. For example, a person may be wearing a lapel microphone, and may be walking around the room. When streaming the spatially rendered audio to a remote room, the sound may also reflect this movement in the speakers.

In another embodiment, sound sources may be part of a virtual environment rather than solely in a physical environment. For example, in a gaming application, a set of players may be gaming on-line with an established conference call within rooms. As virtual players move around a screen in a virtual world, an endpoint for a remote room may be configured to spatially render audio signals speakers in a remote room based on a changing location of the virtual player in the virtual world or of the players in the physical world.

Although the foregoing description contains many specifics, these are not to be construed as limiting the scope of the present disclosure, but merely as providing certain exemplary embodiments. Similarly, other embodiments of the disclosure may be devised which do not depart from the scope of the present disclosure. For example, features described herein with reference to one embodiment also may be provided in others of the embodiments described herein. The scope of the invention is, therefore, defined only by the appended claims and their legal equivalents, rather than by the foregoing description.