Abstract:
This invention locally controls the pitch of speech and audio signals. The invention is based on a seamless time scale modification (S-TSM) scheme connected to a synchronized sampling rate converter that switches between different time scale factors in a seamless manner and controls pitch during playback in a nearly continuous way.

Description:
TECHNICAL FIELD OF THE INVENTION 
       [0001]    The technical field of this invention is recording and transmitting digital audio data. 
       BACKGROUND OF THE INVENTION 
       [0002]    The prior art includes a variety of techniques and algorithms for improving the quality of digitally recorded and transmitted audio data. These techniques include altering audio pitch. 
         [0003]    One prior art technique achieves pitch shifting by seamless time-scale modification (TSM) and restoration of the original time scale through sampling rate conversion. Pitch shifters embedded in karaoke systems use this principle permitting adjustment of the key of a song accompaniment to the singer&#39;s voice. Previous approaches to pitch conversion generally employ either: constant pitch shift of the entire signal as seen in common key-shifting algorithms; or complex algorithms that rely on manually labeled databases, speech production models and/or frequency domain processing. 
       SUMMARY OF THE INVENTION 
       [0004]    The present invention locally controls the pitch of speech and audio signals. The invention uses time scale modification (S-TSM) and a synchronized sampling rate converter that seamlessly switches between different time scale factors. Since the time scale can be adjusted in small steps and transitions between time scales occur seamlessly, this invention provides nearly continuous playback pitch control. The invention is useful in key shifting function in recording studios or karaoke equipment and it can control intonation or fundamental frequency in speech and music synthesis without requiring a speech production model or manual pitch marking. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0005]    These and other aspects of this invention are illustrated in the drawings, in which: 
           [0006]      FIG. 1  illustrates the seamless time scale modification (S-TSM) of this invention continuously receiving input frames containing Sa samples and generating output frames containing Ss samples without changing the original pitch; 
           [0007]      FIG. 2  illustrates an overview of S-TSM processing; 
           [0008]      FIG. 3  illustrates the addition of overlapped frames with fade-in/fade-out windows; 
           [0009]      FIG. 4  illustrates the fine-tuning of the separation Ss between output frames; 
           [0010]      FIG. 5  illustrates the principle of determining optimal offset k; 
           [0011]      FIG. 6  illustrates a system based on Pythagorean tuning using small integer ratios; and 
           [0012]      FIG. 7  illustrates a block diagram of the present invention. 
       
    
    
     DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS 
       [0013]    There are two common approaches to changing the fundamental frequency contour in speech synthesis systems. The first approach uses a speech production model. Voiced speech is approximated as the output of a vocal tract filter fed by an impulse train or another excitation signal source. Controlling the fundamental frequency is relatively straightforward, since it is dictated by the fundamental frequency of the source. However, such systems only work satisfactorily for signals containing pure speech that can be approximated by the model. The second approach is known as PSOLA (pitch-synchronous overlap-add). This approach first marks a speech database containing natural speech utterances. These marks indicate positions in the speech waveform corresponding to fundamental periods. Speech is synthesized by concatenating segments of speech extracted from the database. In order to change the fundamental frequency, distances between marks are changed and the waveform between the marks is warped accordingly. This method usually results in high quality, but pitch marking is a laborious process that cannot be executed automatically. 
         [0014]      FIG. 1  illustrates seamless time scale modification (S-TSM) system  100 . S-TSM  100  continuously receives input frames containing a continuous audio stream of Sa samples  101  and generates output frames containing a continuous audio stream of Ss samples  102  without changing the original pitch. These continuous audio streams include frames that are segments of Sa and Ss and can vary from frame to frame to cope with dynamic time scale changes during playback. If the input consists of a continuous audio stream, the output frames can be concatenated successively without audible artifacts at frame transitions. 
         [0015]      FIG. 2  illustrates the two basic steps involved in audio stream processing. In the analysis step  201 , the input signal is subdivided into overlapping frames (f 1 , f 2 , f 3  . . . ) separated by Sa samples. Note that the larger the value of Sa, the smaller the amount of overlap between successive frames. In the synthesis step  202  the frames resulting from the analysis step are added using a different separation Ss to obtain the output signal. Time scale is reduced when Ss&lt;Sa or increased when Ss&gt;Sa. 
         [0016]    The frame addition operation in synthesis step  202  requires prior multiplication of the frames by fade-in and fade-out window functions.  FIG. 3  illustrates an example window function. The window function is valid in different forms but must assume the value 0 at the beginning of the overlapping region  301  and the value 1 at its end  302 , and the sum of the fade-in and fade-out window values must always equal 1.  FIG. 3  shows simple ramp functions that satisfy these properties. 
         [0017]    In general, parameters Sa and Ss are set arbitrarily within certain limits in order to achieve the desired time scale modification. Referring back to  FIG. 2 , selecting Sa=1024 samples and Ss=512 samples reduces the time scale by half. This results in double speed for a sampled audio signal. In practice the value of Ss must be fine-tuned in order to maximize phase coherence between the frames to be added. 
         [0018]      FIG. 4  illustrates this fine-tuning. An offset value k  401  is added to Ss  402 , resulting in the actual separation Ss+k  403  between output frames. An important part of the algorithm finds the optimal value of offset k that results in maximum coherence between the signal frames to be added. 
         [0019]      FIG. 5  illustrates the process of optimizing k. Consider the regions where the two signal frames to be added overlap, indicated as x  501  and y  502 . The optimal value of offset k is the one that results in maximum coherence between signals x  501  and y  502  by maximizing their similarity. For the example waveforms shown in the  FIG. 5 , it is clear that the particular value of k shown results indeed in maximum similarity. Mathematically, similarity can be approximated by a cross-correlation function. In this case, cross-correlation is evaluated for values of k from −k max  to k max  and the value that results in maximum cross-correlation is selected. Using cross-correlation or other functions as measures of signal similarity has been thoroughly studied in the literature. 
         [0020]    The S-TSM algorithm of the present invention has the additional property that the desired parameters Sa and Ss can be changed in real-time without introducing audible artifacts. There is no discontinuity from frame to frame even when time scales Sa and Ss are changed. A buffering mechanism stores a past history of data and keeps track of the last selected value of k. The deviation from the desired value of Ss by the amount k is always compensated in the following frame and an internal buffer exists as part of the S-TSM processing to absorb such deviations. As a consequence, the S-TSM algorithm always takes exactly the desired numbers of input and output samples regardless of the value of k. 
         [0021]    In principle, Sa and Ss can assume any integer values within a certain range but it is convenient to predefine a set of values relating to desired time scale modification factors. Table 1 defines possible values of Sa and Ss that allow time scale modification factors of 4/8 (0.5×) to 16/8 (2.0×) based upon a sampling frequency of 48 kHz. 
         [0022]    For musical applications a good choice appears to use time scales based on the musical scale covering 1 or 2 octaves of range. Other applications such as speech synthesis do not require such a wide range but finer gradation. 
         [0023]    Note that in Table 1 the number of input samples Sa is the same value of 1024 for all modes. The number of output sample Ss varies from 512 to 2048 and is eventually restored to 1024 by the synchronized sampling rate converter, resulting in the desired pitch modification factor. 
         [0000]                                          TABLE 1               Time Scale               Modification   Input Buffer   Output Buffer       Factor   Size (S a )   Size (S s )                                 4/8   1024   2048        5/8   1024   1638        6/8   1024   1365        7/8   1024   1170        8/8   1024   1024        9/8   1024   910       10/8   1024   820       11/8   1024   744       12/8   1024   682       13/8   1024   630       14/8   1024   586       15/8   1024   546       16/8   1024   512                    
The input and output buffer sizes of the S-TSM algorithm shown in Table 1 were conveniently selected to simplify the switching of the sampling rate conversion filter between different modification factors.
 
         [0024]      FIG. 6  illustrates the general case of sampling rate conversion by a rational factor Z/D, where Z is the up-sampling factor and D is the down-sampling (decimation) factor. Input  601  is up-sampled by up-sampler  603 . Low pass filter  604  filters the output of up-sampler  603 . Down-sampler  605  down-samples the filtered signal producing output signal  602 . Conversion factor table  607  determines the up-sampling factor Z and the down-sampling factor D dependent on the desired time-scale modification. Controller  606  controls the cut-off frequency of low pass filter  604  based on the factors selected by conversion factor table  607 . 
         [0025]    Sampling rate conversion must provide for seamless processing producing no audible artifacts from frame to frame due to transitions between different conversion factors. Use of an FIR (finite impulse response) filter easily satisfies this requirement as the low-pass filter with a delay line that encompasses the longest filter. 
         [0026]    In the preferred embodiment the up-sampling factor varies from 4 to 16 while the down-sampling factor is always 8 as shown in Table 1. The cut-off frequency fc of low-pass filter  604  must correspond in the digital domain to the smallest value out of π/8 or π/n, where n ranges from 4 to 16. Care must be taken to maintain signal continuity upon filter switching by means of shared filter delay lines and filter gain compensation. 
         [0027]    For a karaoke system, a larger number of sampling rate conversions based on a musical scale is desirable. Pythagorean tuning is based on similar small integer ratios. The system illustrated in  FIG. 6  may used in this case. Most modern systems use an equal temperament musical scale based on the (irrational) twelfth root of two. In this case a direct interpolation method may be more advantageous than the equivalent up-sampling/down-sampling conversion based on a rational approximation. In either approach using a 1024 sample buffer for Sa and an integer size for Ss allows the pitch to be accurately shifted to within two cents ( 1/100th of a musical half-step) of any equal tempered musical interval within one octave up or down. If further accuracy is desired, a different value of Sa can be used with the corresponding best value of Ss. 
         [0028]      FIG. 7  illustrates the block diagram of the pitch control system. The input audio stream  701  is split into frames numbered i=1, i=2 and so forth. Sa(i) is the input frame size. In the preferred embodiment the frame size is set to the constant value of 1024 samples. F 0 ( i ) is the original value of the fundamental frequency and k(i)  707  is the pitch change factor that can be set for each frame. Pitch change factor k  707  is selected according to method illustrated in  FIG. 5 . S-TSM  703  outputs Ss(i) samples, where Ss(i)=k(i)*Sa(i). Sampling rate converter SRC  705  is synchronized with k(i)  707  and restores the original number of samples Sa(i) by changing the fundamental frequency to k(i)Fo(i). Note that a particular pitch change factor will remain constant for 1024 samples or 21 ms at a 48 kHz sampling rate. This is sufficiently short to be considered instantaneous for most applications.