Patent Description:
Audio communication sessions, such as voice over Internet protocol (VoIP) sessions, can involve two or more users providing audio inputs to their computing devices and the devices exchanging encoded audio packets indicative of the audio inputs via a network. Upon receipt, the audio packets are decoded to obtain an audio signal, which can be output by the receiving computing device via a speaker. In some cases, the playback of received audio can be captured by a microphone of the listening computing device, such as during a period when the listening user is not actively speaking. This captured playback can then be transmitted and output at the other computing device, which is also known as echo.

Document <CIT> refers to preventing a receiving buffer from becoming empty by: storing received packets in the receiving buffer; detecting the largest arrival delay jitter of the packets and the buffer level of the receiving buffer by a state detecting part; obtaining an optimum buffer level for the largest delay jitter using a predetermined table by a control part; determining, based on the detected buffer level and the optimum buffer level, the level of urgency about the need to adjust the buffer level; expanding or reducing the waveform of a decoded audio data stream of the current frame decoded from a packet read out of the receiving buffer by a consumption adjusting part to adjust the consumption of reproduction frames on the basis of the urgency level, the detected buffer level, and the optimum buffer level.

Document <CIT> refers to speeding up speech at the beginning of a talkspurt in a discontinuous transmission (DTX) packet telephony system to help make up for an access delay incurred during channel allocation. Incoming speech frames are buffered, a pitch period for a current portion of the signal is estimated, and then a pitch periods worth of the signal is cut from that portion. This is continued until the original access delay, as estimated from the time lag between the commencement of voice input for the talkspurt, and notification that a channel is available, is eliminated. The remainder of the talkspurt is then transmitted without such compression.

Document <CIT> refers to a "speech onset detector" which provides a variable length frame buffer in combination with either variable transmission rate or temporal speech compression for buffered signal frames. The variable length buffer buffers frames that are not clearly identified as either speech or non-speech frames during an initial analysis. Buffering of signal frames continues until a current frame is identified as either speech or non-speech. If the current frame is identified as non-speech, buffered frames are encoded as non-speech frames. However, if the current frame is identified as a speech frame, buffered frames are searched for the actual onset point of the speech. Once that onset point is identified, the signal is either transmitted in a burst, or a time-scale modification of the buffered signal is applied for compressing buffered frames beginning with the frame in which onset point is detected. The compressed frames are then encoded as one or more speech frames.

A computer-implemented method, a first computing device, and a computer-readable medium are presented. The first computing device can include one or more processors and a non-transitory memory storing a set of instructions that, when executed by the one or more processors, causes the first computing device to perform operations. The computer-readable medium can also have the set of instructions stored thereon that, when executed by the one.

or more processors of the first computing device, causes the first computing device to perform the operations.

The method and the operations can include obtaining, by the first computing device, an audio input signal for an audio communication session with a second computing device using audio data captured by a microphone of the first computing device; analyzing, by the first computing device, the audio input signal to detect a speech input by a first user associated with the first computing device; determining, by the first computing device, a duration of a detection period from when the audio input signal was obtained until the analyzing has completed; transmitting, from the first computing device and to the second computing device, (i) a portion of the audio input signal beginning at a start of the speech input and (ii) the detection period duration, wherein receipt of the portion of the audio input signal and the detection period duration causes the second computing device to accelerate playback of the portion of the audio input signal to compensate for the detection period duration; analyzing, by the first computing device, the audio input signal to detect an end of the speech input by the first user; and terminating transmission, from the first computing device to the second computing device, of the portion of the audio input signal at a point corresponding to the detected end of the speech input by the first user.

In some embodiments, the method and the operations can further include encoding, by the first computing device, the portion of the audio input signal to obtain a set of audio packets, wherein the transmitting includes transmitting, to the second computing device, (i) the set of audio packets and (ii) the detection period duration.

In the present invention multiple pitch periods are removed, in particular multiple pitch periods having a length of less than <NUM> milliseconds, in particular less than <NUM> milliseconds. This limiting can prevent buffering problems.

In the present invention receipt of the set of audio packets and the detection period duration causes the second computing device to: decode the set of audio packets to obtain an audio output signal; remove a redundant portion of the audio output signal corresponding to one or more pitch periods to obtain the modified audio output signal, wherein the modified output signal has a shorter length than the audio output signal; and output, by a speaker of the second computing device, the modified audio output signal. In some embodiments, a quantity of the one or more removed pitch periods corresponds to the detection period duration. In some embodiments, receipt of the set of audio packets and the detection period duration causes the second computing device to remove the redundant portion of the audio output signal by: cross-correlating the audio output signal with itself to obtain an autocorrelation signal; and detecting one or more peaks of the autocorrelation signal that exceed a threshold indicative of the one or more pitch periods of the audio output signal.

In a further embodiment the threshold is in the range of <NUM> to <NUM>, in particular in the range of <NUM> and <NUM>, more particular <NUM>, wherein lower thresholds can yield increased speed.

In some examples not being part of the invention, analyzing the audio input signal to detect the speech input includes applying a voice activity detection (VAD) technique to the audio input signal, the VAD technique having an aggressiveness or accuracy that corresponds to the detection period duration. In some embodiments, applying the voice detection technique to the audio input signal includes distinguishing the speech input by the first user from speech by the second user within the audio input signal.

Further areas of applicability of the present disclosure will become apparent from the detailed description provided hereinafter. It should be understood that the detailed description and specific examples are intended for purposes of illustration only and are not intended to limit the scope of the disclosure.

During audio communication sessions, such as voice over Internet protocol (VoIP) sessions, audio packets are received and decoded to obtain an audio signal, which is output by the receiving computing device via a speaker. In some cases, the playback of received audio can be captured by a microphone of the computing device, such as during a period when the listening user is not actively speaking. This captured audio playback can then be transmitted to and output by the other computing device, which is also known as echo. To minimize echo, echo suppression or echo cancellation techniques can be used. Echo cancellation, for example, can involve identifying the output audio signal that is output by the speaker and then detecting and removing the output audio signal from the input audio signal captured by the microphone. These techniques, however, do not work in certain environments, e.g., noisy areas. As a result, users may have to manually mute the microphones of their computing devices while they are not talking. Other techniques aim to detect local speech and only transmit audio when the user is actively speaking, but this can result in clipping of the user's speech and/or the audio becoming out-of-sync with corresponding video, e.g., for a video chat session.

Accordingly, improved techniques are not being part of the invention presented for decreasing echo and transmission periods for audio communication sessions. The techniques begin by detecting when a user is speaking (e.g., using a voice activity detection, or VAD technique). For accuracy purposes, there may be a slight lag (e.g., one hundred milliseconds or more) associated with the VAD technique. This analysis of this audio input can also involve distinguishing between the local user's speech and filtering out or ignoring speech by the other user (e.g., captured echo). Once speech is detected, the computing device can begin transmission. This transmission can include a portion of the speech input back to the point when speech was initially detected. The other computing device can receive the transmitted audio, but there will be a synchronization gap, e.g., due to the VAD technique.

The transmitting computing device according to the invention, therefore, can also calculate and transmit information indicative of a duration of the delay period. This can be used by the receiving device to rapidly regain sync without clipping any of the audio. Instead of speeding up the audio playback, which is done by conventional techniques and can undesirably affect the pitch, the receiving computing device can detect and remove a redundant portion of the audio output signal corresponding to one or more pitch periods before playback. Removing one or more pitch periods results in faster playback, because the length of the audio signal is shortened, but without any undesirable pitch modification.

One technical problem being discussed is echo prevention. As mentioned above, this echo can occur due to audio playback being captured by the listening computing device and transmitted back to the originating computing device. The technical advantages of these techniques include not requiring the user to actively control the microphone/speakers to avoid echo. Another technical problem being solved by the present invention is audio synchronization after a delay without affecting the audio pitch. As mentioned above, conventional techniques accelerate audio playback, which affects the pitch and is undesirable to the listening user. The technical advantages of these techniques, therefore, include fast audio playback synchronization after a delay without affecting the audio pitch.

Referring now to <FIG>, a diagram of an example computing network <NUM> is illustrated. The computing network <NUM> can include a first computing device <NUM> that can communicate with a second computing device <NUM> via a network <NUM>. While mobile phone configurations of the computing devices <NUM>, <NUM> are illustrated, it will be appreciated that the first and second computing devices <NUM>, <NUM> can be any suitable computing devices configured for communication via the network <NUM> (desktop computers, laptop computers, tablet computers, etc.). The network <NUM> can be a cellular network (<NUM>, <NUM>, <NUM> long term evolution (LTE), etc.), a computing network (local area network, the Internet, etc.), or some combination thereof. A server computing device <NUM> can also communicate via the network <NUM>. For example, the server computing device <NUM> could coordinate the audio communication session (e.g., a voice over Internet protocol (VoIP) session) between the first and second computing devices <NUM>, <NUM>.

This audio communication session could be established, for example, in response to inputs from users <NUM>, <NUM> at one or both of the first and second computing devices <NUM>, <NUM>. For example, the second user <NUM> may provide an input at the second computing device <NUM> to call the first user <NUM> (an audio communication session request), which could then be accepted by the first user <NUM> via another input at the first computing device <NUM>, thereby establishing the audio communication session. During the audio communication session, audio packets corresponding to audio inputs (e.g., from users <NUM>, <NUM>) can be exchanged via the server computing device <NUM> between the first and second computing devices <NUM>, <NUM>. While the first computing device <NUM> is described as receiving audio data packets from the second computing device <NUM>, it will be appreciated that the first computing device <NUM> can also transmit audio packets to the second computing device <NUM>.

The term "audio communication session" as used herein can refer to either an audio-only communication session or an audio/video communication session. Further, while the techniques herein are described as being implemented at one of the first and second computing devices <NUM>, <NUM> that is receiving the audio packets (the receiving device), it will be appreciated that at least a portion of these techniques could be implemented at the server computing device <NUM>. More particularly, when the server computing device <NUM> is coordinating the audio communication session, the audio packets can flow through the server computing device <NUM>. For example, the server computing device <NUM> could have a queue of audio packets and could perform at least a portion of these techniques, such as decoding, compressing, and then re-encoding for transmission to the receiving device, which could then merely decode and playback upon receipt.

Referring now to <FIG>, a functional block diagram of an example computing device <NUM> is illustrated. The computing device <NUM> can represent the configurations of the first and second computing devices <NUM>, <NUM>. It will be appreciated that the server computing device <NUM> could also have the same or similar configuration as the computing device <NUM>. The computing device <NUM> can include a communication device <NUM> (e.g., a wireless transceiver) configured for communication via the network <NUM>. A processor <NUM> can be configured to control operation of the computing device <NUM>. The term "processor" as used herein can refer to both a single processor and two or more processors operating in a parallel or distributed architecture. A memory <NUM> can be any suitable storage medium (flash, hard disk, etc.) configured to store information at the computing device <NUM>. In one implementation, the memory <NUM> can store instructions executable by the processor <NUM> to cause the computing device <NUM> to perform at least a portion of the disclosed techniques.

The computing device <NUM> includes a microphone <NUM> configured to capture audio input and a speaker <NUM> configured to generate audio output. The microphone <NUM> can be any suitable acoustic-to electric transducer or sensor that converts sound into an electrical signal. This can include speech (e.g., by users <NUM>, <NUM>) as well as other noise, such as background noise. The captured audio data (e.g., an analog signal) is then digitized and converted to an audio input signal (e.g., a digital signal). This audio input signal can be encoded into audio packets for transmission via the network <NUM>. Received audio packets can be decoded into an audio output signal. The audio output signal can be provided to the speaker <NUM>, which in turn can produce audible sound corresponding to the audio output signal. The speaker <NUM> can include a set of electroacoustic transducers that convert an electrical signal into a corresponding sound. While not shown, it will be appreciated that the computing device <NUM> can include other suitable components, such as a display (a touch display), physical buttons, a camera, and the like.

Once the audio communication session is established between the first and second computing devices <NUM>, <NUM>, audio information can be exchanged. The first computing device <NUM> captures audio information using its microphone <NUM> to obtain an audio input signal. The first computing device <NUM> then analyzes the audio input signal to detect a speech input by the first user <NUM>, such as by applying speech detection (e.g., a VAD technique) on the audio input signal. To achieve a desired accuracy, the VAD technique may have a slight delay associated therewith (e.g., a few hundred milliseconds). This delay period, also referred to herein as a detection period, can be described as having a duration that corresponds to an aggressiveness or accuracy of the VAD technique. In other words, this period represents a delay from when the audio input signal is obtained to a point where the speech input is detected.

Once the speech input is detected in the audio input signal, the first computing device <NUM> identifies a portion of the audio input signal beginning at the point of the detected speech. The first computing device <NUM> then encodes audio data packets corresponding to this identified portion of the audio input signal. The first computing device <NUM> transmits these encoded audio data packets to the second computing device <NUM>, along with information indicative of the detection period duration. This information relating to the detection period duration could also be included in encoded data packets. No audio information, however, is transmitted prior to these encoded audio data packets. By transmitting only the portion of the audio input signal beginning with the speech input, echo can be decreased or eliminated without using an echo canceler or suppresser.

The first computing device <NUM> also analyzes the audio input signal to determine an end of the speech input by the first user <NUM>. Once the end of this speech input has been detected, the first computing device <NUM> terminates transmission of the portion of the audio input signal to the second computing device <NUM>. The transmission termination point is a particular point in the audio input signal that corresponds to the detected end of the speech input. The first computing device <NUM> continues analyzing the audio input signal to detect a next occurrence of a speech input by the first user <NUM>, after which transmission to the second computing device <NUM> is resumed according to the techniques herein.

The second computing device <NUM> receives the encoded audio packets and can decode the encoded audio packets to obtain an audio output signal. The second computing device <NUM> also receives the information indicative of the detection period duration and process it accordingly to obtain the detection period duration. The second computing device <NUM> then accelerates playback of the audio output signal to compensate for the determined detection period. This acceleration of the audio playback includescompressing (e.g., removing a redundant portion of) the audio output signal and then outputting the modified audio output signal. A quantity of the one or more removed pitch periods corresponds to the detection period duration. After the pitch period(s) are removed, the second computing device <NUM> has a modified audio output signal having a shorter duration than the original audio output signal, which results in accelerated playback.

The second computing device <NUM> utilizes signal correlation to identify one or more pitch periods for removal. More particularly, the second computing device <NUM> can cross-correlate the audio output signal with itself to obtain an autocorrelation signal. Autocorrelation, cross-autocorrelation, and serial correlation all refer to the process of cross-correlating a signal with itself at different temporal points. The autocorrelation signal represents a similarity of samples as a function of a time gap between them and it can be used for identifying the presence of a periodic signal obscured by noise. Specifically, the second computing device <NUM> can identify a peak in the autocorrelation signal, which represents a strong periodicity in the audio output signal. This identification can be performed using a threshold. For example only, a threshold of approximately <NUM> can be used. In contrast, a straightforward accelerated playback technique might use a threshold of approximately <NUM>. It will be appreciated that any suitable threshold may be used, but lower thresholds will generally provide for increased speed.

Specifically, the lower threshold of approximately <NUM> increases speed (e.g., up to -<NUM>%) while making little if any difference on the quality of the modified audio output signal. The location of this peak can also represent a pitch period of the audio input signal (i.e., a pitch period of the speech). The second computing device <NUM> can then remove at least one of the pitch periods from the audio output signal to obtain a modified audio output signal. In some implementations, multiple pitch periods could be removed, but the length of the multiple pitch periods could be limited to a certain size (e.g., less than <NUM> milliseconds) to avoid potential buffering problems. Various combinations of the above could also be implemented: lower correlation threshold only, removal of multiple pitch periods, or both. The results can include up to <NUM>% increased speed compared to straightforward playback acceleration techniques, while not having a negative effect on audio output pitch. The effective accelerate rate is increased to between <NUM>% and <NUM>%, depending on the audio input signal, which translated to reducing buffer delay by <NUM> to <NUM>.

Referring now to <FIG>, a flow diagram of an example technique <NUM> not being part of the invention for decreasing echo and transmission periods for audio communication sessions is shown. At <NUM>, an audio communication session (VoIP, video chat, etc.) can be established (e.g., by the server computing device <NUM>) between the first computing device <NUM> and the second computing device <NUM>. At <NUM>, the first computing device <NUM> can obtain an audio input signal for the audio communication session based on audio data captured by its microphone <NUM>. At <NUM>, the first computing device <NUM> can analyze the audio input signal to detect a speech input by the first user <NUM>. At <NUM>, the first computing device <NUM> can determine a duration of a detection period from when the audio input signal is obtained to when the analyzing has completed. At <NUM>, the first computing device <NUM> can transmit, to the second computing device <NUM>, the portion of the audio input signal (e.g., encoded audio packets) and the detection period duration.

At <NUM>, the first computing device <NUM> can analyze the audio input signal to detect an end of the speech input by the first user <NUM>. If the end is not detected, the technique <NUM> can return to <NUM>. If the end is detected, however, the technique <NUM> can proceed to <NUM> where the first computing device <NUM> can terminate transmission of the portion of the audio input signal at an appropriate point. The technique <NUM> can then end or return to <NUM>. As previously discussed herein, receipt of the portion of the audio input signal and the detection period duration causes the second computing device <NUM> to accelerate playback of the portion of the audio input signal to compensate for the detection period duration, e.g., by removing a redundant portion of the audio output signal corresponding to one or more pitch periods to obtain a modified audio output signal for output by its speaker <NUM>.

One or more systems and methods discussed herein do not require collection or usage of user personal information. In situations in which certain implementations discussed herein may collect or use personal information about users (e.g., user data, information about a user's social network, user's location and time, user's biometric information, user's activities and demographic information), users are provided with one or more opportunities to control whether the personal information is collected, whether the personal information is stored, whether the personal information is used, and how the information is collected about the user, stored and used. In addition, certain data may be treated in one or more ways before it is stored or used so that personally identifiable information is removed. As one example, a user's identity may be treated so that no personally identifiable information can be determined. As another example, a user's geographic location may be generalized to a larger region so that the user's particular location cannot be determined.

In some example embodiments, well-known procedures, well-known device structures, and well-known technologies are not described in detail.

As used herein, the term module may refer to, be part of, or include: an Application Specific Integrated Circuit (ASIC); an electronic circuit; a combinational logic circuit; a field programmable gate array (FPGA); a processor or a distributed network of processors (shared, dedicated, or grouped) and storage in networked clusters or datacenters that executes code or a process; other suitable components that provide the described functionality; or a combination of some or all of the above, such as in a system-on-chip. The term module may also include memory (shared, dedicated, or grouped) that stores code executed by the one or more processors.

The term code, as used above, may include software, firmware, byte-code and/or microcode, and may refer to programs, routines, functions, classes, and/or objects. The term shared, as used above, means that some or all code from multiple modules may be executed using a single (shared) processor. In addition, some or all code from multiple modules may be stored by a single (shared) memory. The term group, as used above, means that some or all code from a single module may be executed using a group of processors. In addition, some or all code from a single module may be stored using a group of memories.

The techniques described herein may be implemented by one or more computer programs executed by one or more processors. The computer programs include processor-executable instructions that are stored on a non-transitory tangible computer readable medium. The computer programs may also include stored data. Non-limiting examples of the non-transitory tangible computer readable medium are nonvolatile memory, magnetic storage, and optical storage.

Certain aspects of the described techniques include process steps and instructions described herein in the form of an algorithm. It should be noted that the described process steps and instructions could be embodied in software, firmware or hardware, and when embodied in software, could be downloaded to reside on and be operated from different platforms used by real time network operating systems.

The present disclosure also relates to an apparatus for performing the operations herein. This apparatus may be specially constructed for the required purposes, or it may comprise a general-purpose computer selectively activated or reconfigured by a computer program stored on a computer readable medium that can be accessed by the computer. Such a computer program may be stored in a tangible computer readable storage medium, such as, but is not limited to, any type of disk including floppy disks, optical disks, CD-ROMs, magnetic-optical disks, read-only memories (ROMs), random access memories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, application specific integrated circuits (ASICs), or any type of media suitable for storing electronic instructions, and each coupled to a computer system bus. Furthermore, the computers referred to in the specification may include a single processor or may be architectures employing multiple processor designs for increased computing capability.

The algorithms and operations presented herein are not inherently related to any particular computer or other apparatus. Various general-purpose systems may also be used with programs in accordance with the teachings herein, or it may prove convenient to construct more specialized apparatuses to perform the required method steps. The required structure for a variety of these systems will be apparent to those of skill in the art, along with equivalent variations. In addition, the present disclosure is not described with reference to any particular programming language. It is appreciated that a variety of programming languages may be used to implement the teachings of the present disclosure as described herein, and any references to specific languages are provided for disclosure of enablement and best mode of the present invention.

The present disclosure is well suited to a wide variety of computer network systems over numerous topologies. Within this field, the configuration and management of large networks comprise storage devices and computers that are communicatively coupled to dissimilar computers and storage devices over a network, such as the Internet.

Claim 1:
A computer-implemented method, comprising:
establishing an audio session between a first and a second computing device;
capturing audio information by the first computing device using a microphone of the first computing device to obtain an audio input signal;
analyzing, by the first computing device, the audio input signal to detect a speech input by a first user associated with the first computing device by applying speech detection on the audio input signal, wherein the speech detection has a detection period associated therewith that represents a delay from when the audio input signal is obtained to a point where the speech input is detected;
identifying, by the first computing device, a portion of the audio input signal beginning at a point of the detected speech;
encoding, by the first computing device, audio data packets corresponding to the identified portion of the audio input signal;
transmitting, by the first computing device, the encoded audio data packets to the second computing device along with information indicative of the detection period duration;
analyzing, by the first computing device, the audio input signal to determine an end of the speech input;
in response to determining the end of the speech input, terminating transmission of the encoded audio data packets, wherein a transmission termination point is a particular point in the audio input signal that corresponds to the detected end of the speech input;
decoding, by the second computing device, the received encoded audio data packets to obtain an audio output signal;
processing, by the second computing device, the received information indicative of the detection period duration to obtain the detection period duration;
identifying, by the second computing device, one or more pitch periods for removal using signal correlation;
removing, by the second computing device, the one or more pitch periods for removal, wherein a quantity of the one or more removed pitch periods corresponds to the detection period duration;
obtaining, by the second computing device, a modified portion of the audio input signal after removing the pitch periods;
outputting, by the second computing device, the modified portion of the audio input signal; and
continuing with analyzing, by the first computing device, the audio input signal to detect a next occurrence of a speech input by the first user, after which transmission to the second computing device is resumed.