Sound sharing apparatus and method

Disclosed are a sound sharing apparatus and a sound sharing method. The sound sharing apparatus according to one embodiment of the present disclosure includes at least one processor configured to implement: a modifier configured to change a default audio render driver of a local machine from a first audio render driver to a second audio render driver; a capturer configured to capture audio data transmitted to the second audio render driver; and a mixer configured to mix the captured audio data: i) with first voice data to output first mixed data, wherein the first voice data is received from a remote machine connected to the local machine through a network, or ii) with second voice data to output second mixed data, wherein the second voice data is received input through a microphone of the local machine.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority to and the benefit of Korean Patent Application No. 10-2017-0016305, filed on Feb. 6, 2017, the disclosure of which is incorporated herein by reference in its entirety.

BACKGROUND

Embodiments of the present disclosure relate to a technique for sharing a sound in a voice communication system which provides services such as web conferencing and the like.

2. Discussion of Related Art

Web conferencing is an online service capable of hosting real-time meetings, conferences, presentations, and trainings through the Internet. Generally, sharing of voice content, image content, and the like in such web conferencing may greatly help a conference proceed, and various content sharing efforts are being made.

However, when a moving picture experts group 4 (MPEG-4) video prepared in advance is shared, there is a troublesome problem in that a device for streaming the video and a player for reproducing audio data being streamed need to be separately implemented at both the sender's end and the receiver's end in addition to an existing voice channel. Furthermore, in such a situation, there is a problem in that sharing of the video and the like, which is already being streamed through a web browser and the like, is impossible.

Furthermore, as another method of content sharing, there is a method of capturing data transmitted from an operating system to an audio render driver and transmitting the captured data to the other party, but in this case, since the captured data includes voice data transmitted from the other party, there is a problem in that the other party has to rehear what they spoke. In order to resolve the above-described problem, there has been proposed a method of removing the voice data of the other party from the captured data using an acoustic echo canceller (AEC), but in this case, distortion may occur in sound (i.e., the captured data) to be shared.

SUMMARY

Embodiments of the present disclosure provide a means for efficiently sharing sound in an environment in which a local machine and a remote machine are connected via a network.

According to an exemplary embodiment of the present disclosure, there is provided a sound sharing apparatus including at least one processor configured to implement: a modifier configured to change a default audio render driver of a local machine from a first audio render driver to a second audio render driver; a capturer configured to capture audio data transmitted to the second audio render driver; and a mixer configured to mix the captured audio data: i) with first voice data to output first mixed data, wherein the first voice data is received from a remote machine connected to the local machine through a network, or ii) with second voice data to output second mixed data, wherein the second voice data is received through a microphone of the local machine.

The mixer may output the first mixed data to the first audio render driver.

The first audio render driver may be configured to drive a speaker of the local machine.

The sound sharing apparatus may be configured to transmit the second mixed data to the remote machine through the network.

The at least one processor may further implement a resampler configured to change a sampling rate of the captured audio data to that of the first audio render driver or to that of the second voice data.

According to another exemplary embodiment of the present disclosure, there is provided a computing device including one or more processors; memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for: changing a default audio render driver of a local machine from a first audio render driver to a second audio render driver; capturing audio data transmitted to the second audio render driver; and mixing the captured audio data: i) with first voice data to output first mixed data, wherein the first voice data is received from a remote machine connected to the local machine through a network, or ii) with second voice data to output second mixed data, wherein the second voice data is received through a microphone of the local machine.

According to still another exemplary embodiment of the present disclosure, there is provided a sound sharing method which is executed in a computing device including one or more processors and including a memory storing one or more programs executed by the one or more processors, the method including changing a default audio render driver of a local machine from a first audio render driver to a second audio render driver; capturing audio data transmitted to the second audio render driver; and mixing the captured audio data: i) with first voice data to output first mixed data, wherein the first voice data is received from a remote machine connected to the local machine through a network, or ii) with second voice data to output second mixed data, wherein the second voice data is received through a microphone of the local machine.

The sound sharing method may further include, outputting the first mixed data to the first audio render driver.

The first audio render driver may be configured to drive a speaker of the local machine.

The sound sharing method may further include transmitting the second mixed data to the remote machine through the network.

The sound sharing method may further include, before the mixing the captured audio data with the first voice data, changing, by a resampler a sampling rate of the captured audio data to that of the first audio render driver, before the mixing the captured audio data with the second voice data, changing, by the resampler, the sampling rate of the captured audio data to that of the second voice data.

The captured audio data may comprise audio data generated in the computing device, and the captured audio data excludes the first voice data and the second voice data.

The captured audio data, by excluding the first voice data and the second voice data, may reduce an acoustic echo phenomenon at the remote machine.

DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS

Hereinafter, specific embodiments of the present disclosure will be described with reference to the accompanying drawings. The following detailed description is provided to help a comprehensive understanding of methods, apparatuses, and/or systems described herein. However, these are merely illustrative embodiments, and the present disclosure is not limited thereto.

In the following description of embodiments of the present disclosure, if a detailed description of the known related art is determined to obscure the gist of the present disclosure, the detailed description thereof will be omitted. Further, all terms used hereinafter are defined by considering functions in the present disclosure, and meanings thereof may be different according to a user, the intent of an operator, or custom. Therefore, the definitions of the terms used herein should follow contexts disclosed herein. The terms used herein are used to describe the embodiments and are not intended to restrict and/or limit the present disclosure. Unless the context clearly dictates otherwise, the singular form includes the plural form. In this description, the terms “comprising,” “having,” or the like are used to specify that a feature, a number, a step, an operation, a component, an element, or a combination thereof described herein exists, and they do not preclude the presence or addition of one or more other features, numbers, steps, operations, components, elements, or combinations thereof.

FIG. 1is an exemplary diagram for describing a loopback capture interface used in a voice communication system. In the embodiments of the present disclosure, the voice communication system is used to collectively refer to various types of network-based audio-based communication systems such as voice calling, multi-party voice conferencing, and the like. Further, the voice communication system is not limited to a communication system using only audio, but may also include a communication system such as a two-party video calling, multi-party video conferencing, and the like in which audio is included as a part of a communication means. That is, it is noted that the embodiments of the present disclosure are not limited to a communication system of a specific type or method.

Referring toFIG. 1, various applications or hardware devices related to sound reproduction may be present in a local machine used by a participant in voice communication. As one example, the local machine may include a sound sharing apparatus102, a media player104, a web browser106, and the like. The sound sharing apparatus102may be a hardware device having a dedicated application for multi-party voice communication or be a computer readable recording medium for executing the applications, and may transmit a playback request for audio data of the other party, which is received from a remote machine, to an operating system108. Further, the media player104may transmit a playback request for first voice data in the terminal to the operating system108, and the web browser106may transmit a playback request for second voice data online to the operating system108. For example, the first voice data may be a music file stored in the terminal, and the second voice data may be sound content which is reproducible online.

The operating system108may mix the audio data, the first voice data, and the second voice data to transmit the mixed data to a default audio render driver110, and the default audio render driver110may transmit the mixed data, in which the audio data, the first voice data, and the second voice data have been mixed, to a default speaker112. Thereafter, the default speaker112may output the mixed data. Here, the default audio render driver110refers to an audio render driver which is set to be used by default by a local machine among one or more audio render drivers in the local machine, and the default speaker112refers to a speaker which is set to be used by default by the local machine among speakers in the local machine.

At this point, the operating system108may provide a loopback capture interface. An application developer may capture a sound transmitted to the default audio render driver110through the loopback capture interface provided by operating system108.

However, when the loopback capture interface is used, the first voice data and the second voice data and also the audio data of the other party, which is transmitted through the sound sharing apparatus102, are mixed and captured. When such captured mixed data is shared with the other party, the other party has to rehear what they spoke. That is, in this case, an acoustic echo phenomenon occurs.

FIG. 2is an exemplary diagram for describing a process of capturing audio data at a terminal200(i.e., the local machine) according to one embodiment of the present disclosure. As shown inFIG. 2, various applications or hardware devices related to sound reproduction may be present in the terminal200according to one embodiment of the present disclosure. As one example, the terminal200may include a sound sharing apparatus202, a media player204, a web browser206, and the like. As described above, the sound sharing apparatus202may be a hardware device having a dedicated application for a multi-party voice communication or be a computer readable recording medium for executing the dedicated application. Further, the media player204and the web browser206may transmit a playback request for various audio data to an operating system208.

Furthermore, a first audio render driver210and a second audio render driver212may be installed at the terminal200according to one embodiment of the present disclosure. Here, the first audio render driver210may be an actual audio render driver configured to drive a speaker214(a hardware device) of the terminal200, and the second audio render driver212may be a virtual audio render driver configured to drive a virtual speaker. For example, the second audio render driver212may be distributed from a server (not shown) together with the dedicated application for multi-party voice communication to be installed in the terminal200. Meanwhile, before the second audio render driver212is installed, the first audio render driver210may be set as a default audio render driver of the terminal200.

When a sound is shared, the sound sharing apparatus202may change the default audio render driver of the operating system208in the terminal200from the first audio render driver210to the second audio render driver212. As one example, the sound sharing apparatus202may initiate a sound sharing service by executing the dedicated application for multi-party voice communication according to a request of a user, and when initiating the sound sharing service, the sound sharing apparatus202may change the default audio render driver from the first audio render driver210to the second audio render driver212. In this case, the remaining applications, i.e., the media player204, the web browser206, and the like but not the dedicated application in the sound sharing apparatus202, may transmit audio data to be reproduced, i.e., the first voice data, the second audio date, and the like to the second audio render driver212which is the default audio render driver. At this point, the sound sharing apparatus202may originally output voice data of the other party, which is received from another terminal (not shown), that is, the remote machine, to the first audio render driver210.

Thereafter, the sound sharing apparatus202may capture the audio data transmitted to the second audio render driver212. As one example, the sound sharing apparatus202may capture the audio data (e.g., the first voice data and the second voice data) transmitted to the second audio render driver212using the above-described loopback capture interface. In this case, all the audio data generated in the terminal200may be fundamentally separated from the voice data transmitted through the sound sharing apparatus202. Consequently, the audio data of the other party may not be included in the captured audio data, and the sound sharing apparatus202may transmit the captured audio data to the remote machine through the network. The remote machine may be connected to the terminal200(i.e., the local machine) through the network. Here, the network may include any type of communication networks capable of performing packet communications including a mobile communication network such as a third-generation (3G) or long-term evolution (LTE) network, wired or wireless Internet networks, and the like.

As described above, when the captured audio data is shared with the other party, the acoustic echo phenomenon does not occur. Further, when the terminal200receives voice data from a user through a microphone (not shown), the sound sharing apparatus202may mix the voice data with the captured audio data to generate mixed data and transmit the mixed data to the remote machine.

Furthermore, the sound sharing apparatus202may mix the captured audio data with the audio data of the other party received from the remote machine and may output mixed data to the first audio render driver210. As described above, since the second audio render driver212is the virtual audio render driver, the second audio render driver212is not actually connected to the speaker214. Accordingly, the sound sharing apparatus202may mix the captured audio data with the audio data of the other party, which is received from the remote machine, to output the mixed data to the first audio render driver210, and the first audio render driver210may transmit the mixed data in which the captured audio data and the audio data of the other party have been mixed to the speaker214. Thereafter, the speaker214may output the mixed data, and the user may hear the mixed data.

FIG. 3is an exemplary diagram for describing a process of processing the captured audio data at the terminal200according to one embodiment of the present disclosure. Here, the first voice data is the voice data of the other party, which is received from the remote machine connected to the local machine through the network, and the mixed data shown in a portion A ofFIG. 3is data provided to the user of the local machine. Further, the second voice data is the audio data of the user, which is input through the microphone of the local machine, and the mixed data shown in a portion B ofFIG. 3is data provided to the other party (i.e., a user of the remote machine). At this point, it is assumed that the audio data generated at the terminal200is transmitted to the second audio render driver212, and the audio data transmitted to the second audio render driver212is captured through the loopback capture interface.

Referring to the portion A ofFIG. 3, a decoder222may decode the first voice data received from the remote machine. Further, since a sampling rate of the first audio render driver210may be different from that of the second audio render driver212, a resampler306may change a sampling rate of the captured audio data to the sampling rate of the first audio render driver210. Thereafter, a mixer308may mix the first voice data passed through the decoder222with the audio data passed through the resampler306and generate mixed data to output the mixed data to the first audio render driver210. The first audio render driver210may transmit the mixed data to the speaker214. The speaker214may output the mixed data, and the user may hear the mixed data.

Further, referring to the portion B ofFIG. 3, a microphone216may receive second voice data from the user of the local machine. Furthermore, since the sampling rate of the second audio render driver212may be different from that of the second voice data input from the microphone216, the resampler306may change the sampling rate of the captured audio data to that of the second voice data. Thereafter, the mixer308may mix the second voice data input from the microphone216with the audio data passed through the resampler306and generate mixed data to output the mixed data to an encoder218. The encoder218may encode the mixed data and transmit the encoded data to a packetizer220. The packetizer220may packetize the encoded mixed data. Thereafter, each packet may be transmitted to the remote machine via a network (e.g., an existing voice channel) to which the terminal200(i.e., the local machine) and the remote machine are connected. Here, the voice channel may be the same as a voice channel through which the first voice data is transmitted.

FIG. 4is a block diagram illustrating a detailed configuration of the sound sharing apparatus202according to one embodiment of the present disclosure. As shown inFIG. 4, the sound sharing apparatus202according to one embodiment of the present disclosure includes a modifier302, a capturer304, a resampler306, a mixer308, and a transmitter310.

The modifier302changes the default audio render driver of the terminal200(i.e., the local machine) from the first audio render driver210to the second audio render driver212. As described above, the first audio render driver210may be the actual audio render driver configured to drive the speaker214of the terminal200, and the second audio render driver212may be the virtual audio render configured to actually drive the speaker214. In this case, all the audio data, except for the audio data transmitted through the sound sharing apparatus202, may be transmitted to the second audio render driver212which is the default audio render driver.

The capturer304captures the audio data transmitted to the second audio render driver212. As one example, the capturer304may capture the audio data transmitted to the second audio render driver212using the loopback capture interface. In this case, all the audio data generated in the terminal200may be fundamentally separated from the voice data transmitted through the sound sharing apparatus202.

The resampler306adjusts a sampling rate of the captured audio data. As one example, the resampler306may change the sampling rate of the captured audio data to that of the first audio render driver210. As another example, the resampler306may change the sampling rate of the captured audio data to that of the second voice data.

The mixer308mixes the captured audio data with the audio data to generate mixed data. As one example, the mixer308may mix the first voice data received from the remote machine with the captured audio data and generate mixed data to output the mixed data to the first audio render driver210. As another example, the mixer308may mix the captured audio data with the second voice data input through the microphone of the local machine and generate mixed data to output the mixed data to the encoder218.

The transmitter310transmits each packet of the mixed data passed through the encoder218and the packetizer220to the remote machine. At this point, the transmitter310may transmit each packet to the remote machine through a server providing a dedicated application for multi-party voice communication. Here, each packet may be transmitted to the remote machine through a network to which the terminal200(i.e., the local machine) and the remote machine are connected.

FIG. 5is an exemplary flowchart for describing a sound sharing method according to one embodiment of the present disclosure. In the illustrated flowchart, the method is described as being divided into a plurality of operations, but at least some of the operations may be performed in an altered order, may be integrally performed by being combined with other operations, may be omitted, may be performed by being divided into sub-operations, or may be performed with one or more additional operations which are not illustrated.

Further, although not shown in the drawing, the sound sharing apparatus202may initiate a sound sharing service by executing a dedicated application for a multi-party voice communication provided from a server according to a request of a user (e.g., a user A), and, when the sound sharing service is initiated, the sound sharing apparatus202may change the default audio render driver of the local machine200to the second audio render driver212. InFIG. 5, it is assumed that the default audio render driver of the local machine200has already been changed from the first audio render driver210to the second audio render driver212. Hereinafter, a detailed flow of the sound sharing method according to one embodiment of the present disclosure is as follows.

In an operation S502, a sound sharing apparatus402of a remote machine400receives first voice data from a user B.

In an operation S504, a sound sharing apparatus402of the remote machine400transmits the first sound data to the sound sharing apparatus202of the local machine200. At this point, the sound sharing apparatus402of the remote machine400may transmit the first sound data to the sound sharing apparatus202of the local machine200through the server.

In an operation S506, the media player204transmits audio data to the second audio render driver212which is a default audio render driver. For convenience of a description, although the media player204is shown to generate the audio data inFIG. 5, the audio data may be generated by a module instead of the media player204, in which case the operation S506may be identically performed.

In an operation S508, the sound sharing apparatus202captures the audio data transmitted to the second audio render driver212. As one example, the sound sharing apparatus202may capture the audio data transmitted to the second audio render driver212using the loopback capture interface.

In an operation S510, the sound sharing apparatus202mixes the first voice data with the captured audio data and generates mixed data to output the mixed data to the first audio render driver210.

In an operation S512, the first audio render driver210transmits the mixed data to the speaker214.

In an operation S514, the speaker214outputs the mixed data. Accordingly, the user A may hear the mixed data.

In an operation S516, the microphone216receives second voice data from the user A.

In an operation S518, the sound sharing apparatus202mixes the second voice data with the captured audio data to generate mixed data.

In an operation S520, the sound sharing apparatus202transmits the mixed data to the sound sharing apparatus402of the remote machine400. At this point, the sound sharing apparatus202may transmit the mixed data to the sound sharing apparatus402of the remote machine400through the server. Alternatively, the mixed data may be transmitted to the sound sharing apparatus402of the remote machine400through a network (for example, an existing voice channel) to which the local machine200and the remote machine400are connected.

In an operation S522, the sound sharing apparatus402of the remote machine400may transmit the mixed data to a speaker (not shown) of the remote machine400, and the speaker of the remote machine400may output the mixed data. Consequently, the user B may hear the mixed data.

FIG. 6is a block diagram for describing an example of a computing environment10including a computing device suitable for use in exemplary embodiments. In the illustrated embodiment, each component may have different functions and capabilities in addition to those described below, and additional components may be included in addition to components described below.

The illustrated computing environment10includes a computing device12. In one example, the computing device12may be the terminal200, the sound sharing apparatus202, or one or more components included in the sound sharing apparatus202.

The computing device12includes at least one processor14, a computer readable storage medium16, and a communication bus18. The processor14may control the computing device12so that the computing device12operates according to the above-described exemplary embodiments. For example, the processor14may execute one or more programs stored in the computer readable storage medium16. The one or more programs may include one or more computer-executable commands, and, when the one or more computer-executable commands are executed by the processor14, the one or more computer-executable commands may be configured to cause the computing device12to perform operations according to the exemplary embodiments.

The computer readable storage medium16is configured to store computer executable commands, program codes, program data, and/or other suitable forms of information. A program20stored in the computer readable storage medium16includes a set of commands executable by the processor14. In one example, the computer readable storage medium16may be a memory (a volatile memory such as a random access memory, a non-volatile memory, or a suitable combination thereof), one or more magnetic disk storage devices, optical storage devices, flash memory devices, any other type of storage medium capable of being accessed by the computing device12and storing desired information, other types of storage media, or any suitable combination thereof.

The communication bus18interconnects various other components of the computing device12, which include the processor14and the computer readable storage medium16.

The computing device12may also include one or more input/output interfaces22and one or more network communication interfaces26which provide interfaces for one or more input/output devices24. The Input/output interface22and the network communication interface26are connected to the communication bus18. The input/output device24may be connected to other components of the computing device12through the input/output interface22. The exemplary input/output device24may include an input device such as a pointing device (such as a mouse or a trackpad), a keyboard, a touch input device (such as a touch pad, a touch screen, or the like), a voice or sound input device, and various types of sensor devices and/or imaging devices, and/or may include an output device such as a display device, a printer, a speaker, and/or a network card. The exemplary input/output device24may be included inside the computing device12as a component constituting the computing device12, or may be connected to the computing device12as a separate device which is distinguished from the computing device12.

In accordance with the embodiments of the present disclosure, the audio data generated in the terminal and the voice data transmitted through the sound sharing apparatus can be fundamentally separated from each other using the virtual audio render driver such that sound can be easily shared without generation of an acoustic echo and sound distortion. Further, in this case, the sound can be shared through the existing voice channel, and thus an additional channel for sound sharing is not required. Accordingly, a network bandwidth for sound sharing can be saved, and a load on the server can be reduced by minimizing the number of packets transmitted to the server.

Although the present disclosure has been described by way of representative embodiments thereof, it should be understood that numerous modifications can be devised by those skilled in the art that fall within the spirit and scope of this disclosure with respect to the described embodiments. Therefore, the scope of the present disclosure should not be limited to the described embodiments, and it should be determined by not only the appended claims but also equivalents to which such claims are entitled.