Systems and methods for time-scale modification of audio signals

System and methods are provided for modifying audio signals. A waveform representing an audio signal changing over time is received. A first time length is selected. A first starting point in the waveform is selected. A first pair of adjacent segments of the waveform are determined based at least in part on the first starting point and the first time length. The first pair of adjacent segments each correspond to the first time length. A first difference measure associated with the first pair of adjacent segments is calculated. In response to the first difference measure being smaller than a threshold, compression or expansion of the waveform is performed based at least in part on the first time length and the first starting point.

FIELD

The technology described in this patent document relates generally to signal processing and more particularly to audio signal processing.

BACKGROUND

An audio signal (e.g., music or speech) usually includes many components, such as pitch, volume, timbre and time. The modification of the time aspect of an audio signal, which is generally referred to as time-scale modification of the audio signal, is very useful for certain applications, such as voice-mail, dictation-tape playback or post synchronization of film and video.FIG. 1(A)-FIG. 1(C)depict example diagrams showing a basic principle of time-scale modifications of an audio signal. As shown inFIG. 1(A), an original audio recording100includes segments102,104and106of a same time length L0. Time-scale modifications can be performed on the original audio recording100to expand or compress the segments. As shown inFIG. 1(B), the segments102,104and106are expanded to different extents to have time lengths longer than the original time length L0. On the other hand, as shown inFIG. 1(C), the segments102,104and106are compressed to different extents to have time lengths shorter than the original time length L0. Usually, time-scale modifications of an audio signal speed up or slow down the audio signal without changing the pitch of the audio signal which corresponds to a fundamental period of the audio signal.

SUMMARY

In accordance with the teachings described herein, system and methods are provided for modifying audio signals. A waveform representing an audio signal changing over time is received. A first time length is selected. A first starting point in the waveform is selected. A first pair of adjacent segments of the waveform are determined based at least in part on the first starting point and the first time length. The first pair of adjacent segments each correspond to the first time length. A first difference measure associated with the first pair of adjacent segments is calculated. In response to the first difference measure being smaller than a threshold, compression or expansion of the waveform is performed based at least in part on the first time length and the first starting point.

In one embodiment, a system for modifying audio signals includes: one or more data processors and a computer-readable storage medium encoded with instructions for commanding the data processors to execute certain operations. A waveform representing an audio signal changing over time is received. A first time length is selected. A first starting point in the waveform is selected. A first pair of adjacent segments of the waveform are determined based at least in part on the first starting point and the first time length. The first pair of adjacent segments each correspond to the first time length. A first difference measure associated with the first pair of adjacent segments is calculated. In response to the first difference measure being smaller than a threshold, compression or expansion of the waveform is performed based at least in part on the first time length and the first starting point.

In another embodiment, a non-transitory computer readable storage medium includes programming instructions for modifying audio signals. The programming instructions are configured to cause one or more data processors to execute certain operations. A waveform representing an audio signal changing over time is received. A first time length is selected. A first starting point in the waveform is selected. A first pair of adjacent segments of the waveform are determined based at least in part on the first starting point and the first time length. The first pair of adjacent segments each correspond to the first time length. A first difference measure associated with the first pair of adjacent segments is calculated. In response to the first difference measure being smaller than a threshold, compression or expansion of the waveform is performed based at least in part on the first time length and the first starting point.

DETAILED DESCRIPTION

A Pointer-Interval-Controlled-Overlap-Add (PICOLA) algorithm is frequently used to perform time-scale modifications of an audio signal.FIG. 2(A)-FIG. 2(C)depict example diagrams showing a process of compressing a waveform using PICOLA. A waveform202is compressed by replacing segments204and206with a newly generated segment208. Specifically, as shown inFIG. 2(A), the waveform202represents an audio signal changing over time. The first two segments204and206of the waveform202relative to the initial position210are selected, and each of the segments204and206has a same time length Tp which corresponds to a fundamental period (e.g., pitch) of the audio signal. A new segment208having the time length Tp is generated (e.g., overlap-added) based at least in part on the two segments204and206, as shown inFIG. 2(B). Then, the new segment208is used to replace the segments204and206. The newly formed waveform212is shorter than the waveform202, which indicates that the audio signal associated with the waveform202is sped up.FIG. 3(A)-FIG. 3(C)depict example diagrams showing a process of expanding a waveform using PICOLA. A waveform302is expanded by inserting a newly generated segment308between segments304and306of the waveform302. Specifically, as shown inFIG. 3(A), the first two segments304ad306of the waveform302relative to an initial position310are selected, and each of the segments304and306has a same time length Tp′ which corresponds to a fundamental period (e.g., pitch) of the audio signal. A new segment308having the time length Tp′ is generated based at least in part on the two segments304and306, as shown inFIG. 3(B). Then, the new segment308is inserted between the segments304and306. The newly formed waveform312is longer than the waveform302, which indicates that the audio signal associated with the waveform302is slowed down. A basic assumption of PICOLA is that the waveform of an audio signal is periodic, and thus the first two segments of the waveform relative to an initial position are selected for pitch detection, as shown inFIG. 2(A)andFIG. 3(A). However, the basic assumption of PICOLA is often not true in reality. For example, a starting point may not be accurately determined. Such deficiencies of PICOLA may cause inaccuracy in results of time-scale modifications under some circumstances.

FIG. 4(A)-FIG. 4(C)depict example diagrams showing a process of compressing a waveform. As shown inFIG. 4(A)-FIG. 4(C), the waveform402is compressed by replacing segments404and406with a newly generated segment408. Specifically, the waveform402represents an audio signal changing with time. Different time lengths and different starting points can be selected and examined to reduce a difference between two adjacent segments that are next to a starting point. A proper time length TBand a proper starting point410(e.g., different from an initial position412) are determined so that a difference between the segments404and406that are next to the starting point410is smaller than a threshold. Each of the segments404and406has the same time length THwhich corresponds to a fundamental period (e.g., pitch) of the audio signal. A new segment408having the time length TBis generated (e.g., overlap-added) based at least in part on the two segments404and406, as shown inFIG. 4(B). For example, triangle window functions are used to add the segments404and406to form the new segment408. Then, the new segment408is used to replace the segments404and406to form a new waveform414, as shown inFIG. 4(C). For example, the waveform402corresponds to an original sampling length L, and the waveform414corresponds to a length L-T5which is shorter than the original sampling length L.

FIG. 5(A)-FIG. 5(C)depict example diagrams showing a process of expanding a waveform. A waveform502is expanded by inserting a newly generated segment508between segments504and506of the waveform502. As shown inFIG. 5(A), a proper time length TBand a proper starting point510(e.g., different from an initial position512) are determined so that a difference between the segments504and506that are next to the starting point510is smaller than a threshold. Each of the segments504and506has the same time length TBwhich corresponds to a fundamental period (e.g., pitch) of the audio signal. A new segment508having the time length TBis generated (e.g., overlap-added) based at least in part on the two segments504and506, as shown inFIG. 5(B). Then, the new segment508is inserted between the segments504and506to form a new waveform514. For example, the waveform502corresponds to an original sampling length L, and the waveform514corresponds to a length L+TBwhich is longer than the original sampling length L.

FIG. 6depicts an example diagram showing a system for performing time-scale modifications of an audio signal. As shown inFIG. 6, a waveform-extraction component602extracts a waveform from an audio signal604, and a waveform-processing component606searches for a proper starting point and a proper time length that corresponds to a fundamental period of the audio signal604. Once the proper starting point and the proper time length are determined, an overlap-adding component608generates a new segment, and a waveform-synthesis component610replaces a pair of original segments that are next to the determined starting point with the new segment for compression of the waveform, or inserts the new segment between the pair of original segments for expansion of the waveform.

Specifically, the waveform-processing component606selects a time length within a time range. For example, the time range has a lower limit Lminand an upper limit Lmaxthat are determined as follows:

A sampling length L is calculated as follows:

L={Pl×γγ-1⁢γ>1Pl×γ1-γ⁢1>γ>0.5(2)
where Pl represents the selected time length, and γ represents a speed control factor. The waveform-processing component606selects a starting point, shiftPos, within a position range, for example, [0, L−2×Pl]. Then, the waveform-processing component606calculates a difference measure, EshiftPos, associated with two adjacent segments that are next to the selected starting point. The difference measure, EshiftPos, is determined as follows:

If the difference measure is smaller than a threshold value, the waveform-processing component606outputs the two adjacent segments that are next to the selected starting point to the overlap-adding component608that generates a new segment based on the two adjacent segments. In addition, the waveform-processing component606outputs the selected starting point shiftPos and the selected time length Pl to the waveform-synthesis component610which outputs a newly generated waveform. For example, the waveform-synthesis component610generates the new waveform by replacing the two adjacent segments that are next to the selected starting point with the new segment or inserting the new segment between the two adjacent segments.

If the difference measure is no smaller than the threshold value but is smaller than a difference value stored in a storage unit (e.g., a register) that is no smaller than the threshold value, the waveform-processing component606replaces the temporary difference value with the difference measure in the storage unit. In addition, the waveform-processing component606saves the selected starting point and the selected time length (e.g., in one or more storage units). Furthermore, the waveform-processing component606selects another starting point (e.g., based on performance demands) within the position range and provides the selected starting point to the buffer614for another cycle of processing. If the difference measure is no smaller than the stored difference value, the waveform-processing component606directly selects another starting point within the position range for another cycle of processing without replacing the difference value.

If there is no other starting point that can be selected and the difference measure is no smaller than the threshold value, the waveform-processing component606selects another time length within the time range, and another sampling length is calculated. Then, the waveform-processing component606selects another starting point based on the newly selected time length and the newly calculated sampling length for another cycle of processing.

If no other starting point and no other time length can be selected and the difference measure is no smaller than the threshold value, the waveform-processing component606selects a particular starting point and a particular time length that are stored in the storage unit and are related to a smallest difference measure.

FIG. 7depicts an example diagram showing a process for modifying audio signals. At702, a waveform representing an audio signal changing over time is received. At704, a first time length is selected. At706, a first starting point in the waveform is selected. At708, a first pair of adjacent segments of the waveform are determined using the first starting point. The first pair of adjacent segments each correspond to the first time length. At710, a first difference measure associated with the first pair of adjacent segments is calculated. At712, in response to the first difference measure being smaller than a threshold, compression or expansion of the waveform is performed based at least in part on the first time length and the first starting point.

This written description uses examples to disclose the invention, include the best mode, and also to enable a person skilled in the art to make and use the invention. The patentable scope of the invention may include other examples that occur to those skilled in the art. Other implementations may also be used, however, such as firmware or appropriately designed hardware configured to carry out the methods and systems described herein. For example, the systems and methods described herein may be implemented in an independent processing engine, as a co-processor, or as a hardware accelerator. In yet another example, the systems and methods described herein may be provided on many different types of computer-readable media including computer storage mechanisms (e.g., CD-ROM, diskette, RAM, flash memory, computer's hard drive, etc.) that contain instructions (e.g., software) for use in execution by one or more processors to perform the methods' operations and implement the systems described herein.