Abstract:
A system and method are disclosed for modifying an audio signal. A pitch associated with the audio signal is detected. A portion of the audio signal that is associated with the detected pitch is modified. Controlling the modification of a primary audio signal is disclosed. The level of a secondary audio signal is monitored. Modification of the primary audio signal is enabled if the level of the secondary audio signal rises above a first prescribed threshold at a time when the primary audio signal is not being modified. Modification of the primary audio signal is disabled if the level of the secondary audio signal drops below a second prescribed threshold at a time when the primary audio signal is being modified.

Description:
INCORPORATION BY REFERENCE 
     U.S. Pat. No. 6,405,163, entitled PROCESS FOR REMOVING VOICE FROM STEREO RECORDINGS, issued Jun. 11, 2002, is incorporated herein by reference for all purposes. 
     FIELD OF THE INVENTION 
     The present invention relates generally to digital signal processing. More specifically, pitch-based frequency domain voice removal is disclosed. 
     BACKGROUND OF THE INVENTION 
     This disclosure relates to voice removal techniques. Such techniques may be useful in a variety of applications, including the now very popular field of karaoke entertaining. In karaoke a (usually amateur) singer performs live in front of an audience with background music. One of the challenges of this activity is to come up with the background music (i.e. get rid of the original singer&#39;s voice to retain only the instruments so the amateur singer&#39;s voice can replace that of the original singer). 
     One way in which this can be achieved consists of using a stereo recording and making the assumption (usually true) that the voice is panned in the center (i.e., that the voice was recorded in mono and added to the left and right channels with equal level). In that case the voice can be significantly reduced by subtracting the right channel from the left channel (referred to herein as the “left minus right” technique), resulting in a mono recording from which the voice is nearly absent. Using this approach, a faint reverberated version of the voice typically is left in the difference signal because stereo reverberation is usually added after the mix. There are several drawbacks to this technique. First, the output signal is always monophonic. In other words it is not possible using this technique to recover a stereo signal from which the voice has been removed. Second, more often than not, other instruments are also panned in the center (bass guitar, bass drum, horns and so on), and the left minus right technique will also remove them, which is undesirable. 
     U.S. Pat. No. 6,405,163 (the &#39;163 Patent), incorporated by reference above, describes another method which comprises applying a gain to the left and right channels in the short time frequency domain to attenuate center-panned signals. The frequency domain processing method improves on the left minus right technique in that it outputs a stereo signal. While the techniques described in the &#39;163 Patent provide better results than the left minus right approach, the techniques taught by the &#39;163 Patent may result in center-panned signals other than the voice being removed. For example, as noted above percussion, bass, and other instruments are sometimes panned to the center. The &#39;163 Patent teaches restricting the attenuation to voice frequencies in an effort to avoid removing non-voice components. However, the voice spectrum overlaps with the frequency spectra of many instruments that might also be center panned (for example, guitars and snare drums), and components associated with such instruments may also be removed under the approaches taught in the &#39;163 Patent. Therefore, there is a need for a way to remove a center-panned voice component without also removing components associated with other center-panned signals, including signals in the voice frequency range. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       The present invention will be readily understood by the following detailed description in conjunction with the accompanying drawings, wherein like reference numerals designate like structural elements, and in which: 
         FIG. 1  is a block diagram illustrating a system used in one embodiment to remove or amplify one or more components from a stereo recording. 
         FIG. 2  is a flowchart illustrating a method used in one embodiment to remove or amplify one or more components of an audio signal. 
         FIG. 3  is a flowchart illustrating a method used in one embodiment that uses frequency domain combs to perform pitch detection of an audio signal (step  205 ). 
         FIG. 4A  is a block diagram illustrating a system used in one embodiment to perform step  302  and step  304  in  FIG. 3 . 
         FIG. 4B  is a block diagram illustrating a system used in one embodiment to perform step  304  in  FIG. 3 . 
         FIG. 5A  is a block diagram illustrating a system used in one embodiment to perform step  306  and step  308  in  FIG. 3 . 
         FIG. 5B  is a plot illustrating the cross correlation values C m  as a function of frequency for an audio signal. 
         FIG. 6A  is a flowchart illustrating a method used in one embodiment to modify portions of frequency domain spectra believed to be voice-related based on a detected pitch or pitches (step  210 ). 
         FIG. 6B  is the plot of  FIG. 5B  with harmonic regions each of length Δ labeled. 
         FIG. 7  is a block diagram illustrating one embodiment of voice removal block  110  in  FIG. 1 . 
         FIG. 8  is a flowchart illustrating a duophonic technique used in one embodiment to perform pitch detection of an audio signal. 
         FIG. 9  is a state diagram illustrating a duck technique. 
         FIG. 10  is a block diagram illustrating a system used in one embodiment to remove or amplify one or more components from a stereo recording incorporating an embodiment of a duck technique. 
     
    
    
     DETAILED DESCRIPTION 
     It should be appreciated that the present invention can be implemented in numerous ways, including as a process, an apparatus, a system, or a computer readable medium such as a computer readable storage medium or a computer network wherein program instructions are sent over optical or electronic communication links. It should be noted that the order of the steps of disclosed processes may be altered within the scope of the invention. 
     A detailed description of one or more preferred embodiments of the invention is provided below along with accompanying figures that illustrate by way of example the principles of the invention. While the invention is described in connection with such embodiments, it should be understood that the invention is not limited to any embodiment. On the contrary, the scope of the invention is limited only by the appended claims and the invention encompasses numerous alternatives, modifications and equivalents. For the purpose of example, numerous specific details are set forth in the following description in order to provide a thorough understanding of the present invention. The present invention may be practiced according to the claims without some or all of these specific details. For the purpose of clarity, technical material that is known in the technical fields related to the invention has not been described in detail so that the present invention is not unnecessarily obscured. 
     Removing or amplifying one or more components from a stereo recording is disclosed. In one embodiment, short time frequency domain techniques are used to selectively apply a gain to one or more frequency bins associated with one or more pitches. This approach allows center-panned signal components other than voice, including those at voice frequencies, to be preserved in the final output. In one embodiment, pitch estimation is used to selectively modify only the harmonics of the voice component. 
     The idea behind pitch-based processing is that measuring the pitch (or the pitches) of the signal can help discriminate the voice component from other audio components. For example, drum hits are not pitched, and bass-guitar notes might have a pitch much lower than a singer&#39;s pitch. If the voice pitch can be identified, harmonics of the fundamental frequency of the voice component can be modified and the remaining components of the audio signal can be preserved. 
       FIG. 1  is a block diagram illustrating a system used in one embodiment to remove or amplify one or more components from a stereo recording. As used herein, the term “component” refers to a portion of an audio signal that is associated with an identifiable audio source, such as the voice of a particular singer or the output of an instrument. The term “audio signal” as used herein refers to any set of audio data stored or transmitted in any form, including without limitation a sound recording.  FIG. 1  shows a left stereo channel s L (t) and a right stereo channel s R (t) being provided as inputs to short time Fourier transform (STFT) blocks  102 L and  102 R, respectively. In one embodiment, the left and right stereo channels may be in the form of digital signals. For analog stereo channels, the channels can be digitized using techniques well known in the art. The outputs of STFT blocks  102 L and  102 R are the frequency domain spectra of the left and right stereo channels, labeled in  FIG. 1  as S L (u,k) and S R (u,k), respectively. In one embodiment, u and k are discrete time and frequency indexes, respectively (e.g., u is the frame index and ω=2πk/N where ω is the frequency and N is the size of the STFT). In one embodiment, STFT blocks  102 L and  102 R have frame sizes that include several periods of the voice signal (for example, 30-60 ms). In one embodiment, each frame overlaps the previous frame. In general, the STFT blocks may be implemented or replaced by other blocks that perform a similar function. In one embodiment, STFT blocks  102 L and  102 R may comprise subband filter banks. In one embodiment, STFT blocks  102 L and  102 R may perform wavelet transforms. 
     The outputs of STFT blocks  102 L and  102 R are provided as input to a pitch detection block  106 . Pitch detection block  106  detects the pitch of the voice information in the signal using one of many methods well known in the art. In one embodiment, it is assumed that the voice component typically will be associated with the most prominent pitch associated with the audio signal, and in one such embodiment pitch detection block  106  is configured to detect the most prominent pitch in each portion of the audio signal. Pitch detection block  106  provides as output for each frame u a most-prominent pitch value P(u). The outputs of pitch detection block  106  and STFT blocks  102 L and  102 R are provided as inputs to voice removal block  110 . 
     As used herein, the term “remove” refers to any degree of attenuation of the affected component, including without limitation either full or partial attenuation of the affected component. While voice removal block  110  is labeled “voice removal”, those of skill in the art will recognize that the component removed may be other than voice. In one alternative embodiment, voice removal block  110  may be configured to amplify instead of remove the affected component(s). 
     Voice removal block  110  selectively modifies portions believed to be voice-related based on the output of pitch detection block  106 . In one embodiment, portions are identified as potentially voice-related if they are associated with a most-prominent pitch detected by pitch detection block  106 . In one embodiment, selectively modifying comprises calculating a gain and selectively applying the gain. In one such embodiment, the gain is zero for center-panned portions identified as potentially voice-related based on the output of pitch detection block  106  and nonzero (e.g., one) for other portions (voice removal). Alternatively, the gain may be greater than one for center-panned portions identified as potentially voice-related based on the output of pitch detection block  106  and one for other portions (voice amplification). In one embodiment, as described more fully below, the gain may vary based on the degree of similarity between the left and right channels and/or other factors. 
     Voice removal block  110  provides as output modified frequency domain spectra Ŝ L (u,k) and Ŝ(u,k) for the left and right channels, respectively. In one embodiment, the modified spectra comprise the original spectra as modified by applying the gains described above. The modified frequency domain spectra are provided as input to inverse short time Fourier transform (ISTFT) blocks  114 L and  114 R, respectively. ISTFT blocks  114 L and  114 R are configured to synthesize modified time-domain signals ŝ L (t) and ŝ R (t), respectively. If voice removal block  110  attenuated components of the signal panned to the center at the pitch believed to be voice, the modified stereo channels output by ISTFT blocks  114 L and  114 R will have voice removed. However, the instruments and other sounds not panned to the center and/or not at the pitch believed to be voice will be preserved. 
     In one embodiment, voice removal block  110  is implemented on a processor configured to perform the functions described above. In one embodiment, pitch detection block  106  is implemented on a processor configured to perform the functions described above. In one embodiment, system  100  is implemented on a processor configured to perform the functions described above. 
       FIG. 2  is a flowchart illustrating a method used in one embodiment to remove or amplify one or more components of an audio signal. In step  200 , the audio signal is transformed into a short time frequency domain. In one embodiment, the audio signal may comprise a stereo signal comprising a left stereo channel and a right stereo channel, and in one such embodiment step  200  comprises transforming the left stereo channel and the right stereo channel separately into left and right channel short time frequency domain spectra, as shown in  FIG. 1 . In step  205 , the pitch of the voice information in the audio signal is detected using one of many methods well known in the art. In one embodiment, the frequency domain spectra from step  200  are used to detect the pitch of the voice information in the signal. In one embodiment, it is assumed that the voice component typically will be associated with the most prominent pitch associated with the audio signal, and in one such embodiment the most prominent pitch in the audio signal is detected (monophonic approach). In another embodiment, it is assumed that the voice component typically will be associated with the first or second most prominent pitch associated with the audio signal, and in one such embodiment the first and second most prominent pitches in the audio signal are detected (duophonic approach). 
     Based at least in part on the pitch or pitches detected in step  205 , portions of the frequency domain spectra believed to be voice-related are modified in step  210  to produce modified frequency domain spectra. In one embodiment, portions of the frequency domain spectra believed to be voice related comprise a range of frequency bins located around each harmonic of the detected pitch or pitches. In one embodiment, the portions believed to be voice-related are amplified. In one embodiment, the portions believed to be voice-related are removed. In one embodiment, a gain is used to modify the frequency domain spectra. In one such embodiment, as described more fully below, the gain may vary based at least in part on the degree of similarity between the left and right channels and/or other factors. 
     In step  215 , a modified time domain signal is synthesized from the modified short time frequency domain spectra. In one embodiment, step  210  comprises generating modified spectra for the left and right stereo channels, as in  FIG. 1 , and step  215  comprises synthesizing separate time domain signals for the left and right stereo channels. 
     The pitch of the voice information in an audio signal can be detected using one of many techniques well known in the art. In one embodiment, it is assumed that the voice component will have the most prominent pitch (monophonic approach). In another embodiment, it is assumed that the voice component will have the first or second most prominent pitch (duophonic approach). In one embodiment, the autocorrelation of the spectral magnitude of the stereo signal is used. The autocorrelation typically exhibits a peak at the most prominent pitch. In one embodiment, a plurality of frequency domain combs is used where each comb is associated with a candidate pitch frequency. The spectral magnitude of the stereo signal is cross-correlated with the frequency domain combs. 
       FIG. 3  is a flowchart illustrating a method used in one embodiment that uses frequency domain combs to perform pitch detection of an audio signal (step  205 ). In step  302 , the center-panned component(s) of the audio signal are extracted. In step  304 , the spectral magnitude M(u,k) of the extracted component(s) of the audio signal is calculated. In one embodiment, step  302  is not performed and M(u,k)=|S R (u,k)+S L (u,k)|, i.e., the magnitude of the combined left and right channel signals is determined in step  304 . In one embodiment, the spectral magnitude is compressed using a compression function (for example, a logarithmic function, a square root function, or an inverse hyperbolic sine function). 
     In step  306 , a plurality of pitch candidates {P m } is selected (for example, every Hz between 80 Hz and 200 Hz). Frequency domain combs C of each pitch candidate are cross-correlated with the spectral magnitude. In one embodiment, the cross-correlation function is defined as 
     
       
         
           
             
               C 
               ⁡ 
               
                 ( 
                 
                   P 
                   m 
                 
                 ) 
               
             
             = 
             
               
                 ∑ 
                 
                   i 
                   = 
                   0 
                 
                 
                   I 
                   ⁡ 
                   
                     ( 
                     
                       P 
                       m 
                     
                     ) 
                   
                 
               
               ⁢ 
               
                 M 
                 ⁡ 
                 
                   ( 
                   
                     k 
                     ⁡ 
                     
                       ( 
                       i 
                       ) 
                     
                   
                   ) 
                 
               
             
           
         
       
     
     where k(i) corresponds to the STFT bin closest to the i th  harmonic of the pitch P m . The sum is carried out up to i=I(P m ) (i.e., for harmonics below a certain limit in Hz, such as 6 kHz). 
     The cross-correlation typically exhibits a large peak at the most prominent pitch and smaller peaks at multiples and submultiples of the most prominent pitch. In step  308 , the location of the maximum value of C(P m ) is identified as the most prominent pitch for that frame. In one embodiment, it is assumed that the voice component will have the most prominent pitch. 
     In some embodiments, a voiced/unvoiced decision is obtained, using techniques well known in the art to determine whether or not a voice component is present. In one embodiment, if the most prominent pitch is outside a predefined range (for example from 80 Hz to 300 Hz for a male singer, or from 200 Hz to 1 kHz for a female singer), the algorithm assumes that no voice is present. 
     In one embodiment, the voiced/unvoiced decision is obtained by comparing the maximum correlation to a predefined threshold. If the maximum correlation is above the threshold, the decision is that the location of the maximum correlation is the voice pitch. If the maximum correlation is below the threshold, the decision is that there is no voice component in the signal. A conservative choice for a threshold is one that biases the decision to be voiced. In one embodiment, if the decision is that there is no voiced information in the signal, system  100  performs no processing (passes input signals s L (t) and s R (t) straight to the output). 
     In one embodiment, pitch detection is performed on a frame-by-frame basis. In some embodiments, previous and future values of the pitch are used to smooth the frame-based estimate via a median filter or a dynamic programming algorithm, both well known in the art. This may help stabilize the voiced/unvoiced decision and remove occasional octave errors. 
       FIG. 4A  is a block diagram illustrating a system used in one embodiment to perform step  302  and step  304  in  FIG. 3 . In one embodiment, the audio signal may comprise a stereo signal comprising a left stereo channel and a right stereo channel, and in one such embodiment the left stereo channel and the right stereo channel are transformed separately into left and right channel short time frequency domain spectra, as shown in  FIG. 1 . The frequency domain spectra of the left and right stereo channels are labeled in  FIG. 1  as S L (u,k) and S R (u,k), respectively. S L (u,k) and S R (u,k) are provided as input to a difference determination block  404 . 
     Difference determination block  404  estimates the degree to which the signal is panned in the center. The output of difference determination block  404  is labeled as ρ(u,k) in  FIG. 4 . In one embodiment, ρ(u,k) is defined as 
     
       
         
           
             
               p 
               ⁡ 
               
                 ( 
                 
                   u 
                   , 
                   k 
                 
                 ) 
               
             
             ≡ 
             
               
                 
                    
                   
                     
                       S 
                       L 
                     
                     - 
                     
                       S 
                       R 
                     
                   
                    
                 
                 2 
               
               
                 
                   
                      
                     
                       S 
                       L 
                     
                      
                   
                   2 
                 
                 + 
                 
                   
                      
                     
                       S 
                       R 
                     
                      
                   
                   2 
                 
               
             
           
         
       
     
     A component of the signal that is panned in the center will exhibit a small ρ(u,k) because the left and right channel short time frequency domain spectra S L (u,k) and S R (u,k) are similar. In contrast, a component of the signal that is not panned in the center will exhibit a larger ρ(u,k). 
     ρ(u,k) is provided as input to a gain determination block  406 . Gain determination block  406  determines a gain G C (u,k) as a function of ρ(u,k). G C (u,k) is appropriately defined so that when it is applied to S L (u,k) and S R (u,k), the center-panned components of the signal are extracted. For example, G C (u,k) may attenuate non-center-panned components of the signal, and thus extract the center-panned components of the signal. In one embodiment, G C (u,k) is defined as 
     
       
         
           
             
               
                 G 
                 C 
               
               ⁡ 
               
                 ( 
                 
                   u 
                   , 
                   k 
                 
                 ) 
               
             
             ≡ 
             
               
                 1 
                 - 
                 
                   
                     G 
                     C 
                     ′ 
                   
                   ⁡ 
                   
                     ( 
                     
                       u 
                       , 
                       k 
                     
                     ) 
                   
                 
               
               
                 1 
                 + 
                 
                   
                     G 
                     C 
                     ′ 
                   
                   ⁡ 
                   
                     ( 
                     
                       u 
                       , 
                       k 
                     
                     ) 
                   
                 
               
             
           
         
       
       
         
           where 
         
       
       
         
           
             
               
                 G 
                 C 
                 ′ 
               
               ⁡ 
               
                 ( 
                 
                   u 
                   , 
                   k 
                 
                 ) 
               
             
             ≡ 
             
               { 
               
                 
                   
                     
                       
                         
                           
                             ( 
                             
                               
                                 ρ 
                                 ⁡ 
                                 
                                   ( 
                                   
                                     u 
                                     , 
                                     k 
                                   
                                   ) 
                                 
                               
                               
                                 ρ 
                                 0 
                               
                             
                             ) 
                           
                           α 
                         
                         ⁢ 
                         for 
                         ⁢ 
                         
                             
                         
                         ⁢ 
                         
                           ρ 
                           ⁡ 
                           
                             ( 
                             
                               u 
                               , 
                               k 
                             
                             ) 
                           
                         
                       
                       ≤ 
                       
                         ρ 
                         0 
                       
                     
                   
                 
                 
                   
                     
                       
                         1 
                         ⁢ 
                         
                             
                         
                         ⁢ 
                         for 
                         ⁢ 
                         
                             
                         
                         ⁢ 
                         
                           ρ 
                           ⁡ 
                           
                             ( 
                             
                               u 
                               , 
                               k 
                             
                             ) 
                           
                         
                       
                       &gt; 
                       
                         ρ 
                         0 
                       
                     
                   
                 
               
             
           
         
       
     
     ρ 0  is an arbitrary threshold and α is used to control the behavior of G C  as a function of ρ. For example, threshold values between 0.001 and 0.5 and an exponent value α=0.7 may be used. This choice of G C  will attenuate any signal not panned to the center, and thus extract the center-panned components of the signal. 
     The gains G C (u,k) provided as output of gain determination block  406  are provided as input to amplifier  402  and amplifier  410 . S L (u,k) and S R (u,k) are provided as inputs to amplifier  402  and amplifier  410 , respectively. Amplifier  402  applies the respective gains G C (u,k) to left channel short time frequency domain spectra S L (u,k) to which they correspond. Likewise, amplifier  410  applies the gains G C (u,k) to S R (u,k). The outputs of amplifier  402  and amplifier  410  are combined and the sum S C (u,k) is provided as input to a spectral magnitude block  408 . 
     Spectral magnitude block  408  is a block used in one embodiment to perform step  304  in  FIG. 3 . Spectral magnitude block  408  calculates the spectral magnitudes M(u,k) of the extracted component(s) of the audio signal. In one embodiment, M(u,k)=|S C (u,k)|. In one embodiment, the spectral magnitude is compressed using a compression function F (for example, a logarithmic function, a square root function, or an inverse hyperbolic sine function). In one embodiment, M(u,k)=F(|S C (u,k)|). 
       FIG. 4B  is a block diagram illustrating a system used in one embodiment to perform step  304  in an embodiment in which pitch detection is performed on the combined left and right channel signals, as opposed to on the extracted center-panned signal, such as in an embodiment in which step  302  is of  FIG. 3  is omitted, as described above. In one embodiment, the audio signal may comprise a stereo signal comprising a left stereo channel and a right stereo channel, and in one such embodiment the left stereo channel and the right stereo channel are transformed separately into left and right channel short time frequency domain spectra, as shown in  FIG. 1 . The frequency domain spectra of the left and right stereo channels are labeled in  FIG. 1  as S L (u,k) and S R (u,k), respectively. S L (u,k) and S R (u,k) are summed and provided as input to spectral magnitude block  408 . Spectral magnitude block  408  is described above with respect to  FIG. 4A . 
       FIG. 5A  is a block diagram illustrating a system used in one embodiment to perform step  306  and step  308  in  FIG. 3 . A plurality of pitch candidates {P m } is selected. Spectral magnitude values M(u,k), determined in one embodiment as described above in connection with  FIG. 4A  and in one alternative embodiment as described above in connection with  FIG. 4B , are provided as input to N cross-correlator blocks  500 - 503 . In one embodiment, for each frame u, each cross-correlator block cross-correlates the spectral magnitude values M(u,k) with a frequency domain comb associated with a pitch candidate to produce cross-correlation values C 1  through C N , where C m =C(P m ). Each cross-correlator block  500 - 503  provides a cross-correlation value as input to a comparator block  520 . Comparator block  520  selects the maximum cross-correlation value C MAX . The pitch associated with C MAX  is the most prominent pitch and assumed to be the voice component. 
       FIG. 5B  is a plot illustrating the cross correlation values C m  as a function of frequency for an audio signal. The cross-correlation values exhibit a large peak at the most prominent pitch and smaller peaks at multiples (and submultiples, not shown in  FIG. 5B ) of the most prominent pitch. In  FIG. 5B , the most prominent pitch is P and its harmonics (multiples) are 2P, 3P, and 4P. In one embodiment, the most prominent pitch P is assumed to be the voice component. 
       FIG. 6A  is a flowchart illustrating a method used in one embodiment to modify portions of frequency domain spectra believed to be voice-related based on a detected pitch or pitches (step  210 ). In step  600 , a range of frequency bins located around each harmonic of the detected pitch or pitches are identified as bins that will be modified. The frequency ranges represented by the ranges of frequency bins defined in step  600  are referred to herein as “harmonic regions”. In one embodiment, the harmonic regions comprise a range of short time Fourier transform frequency bins around each harmonic. These regions may include several bins on each side of the harmonic bin. 
       FIG. 6B  is the plot of  FIG. 5B  with harmonic regions each of length Δ labeled. As discussed above, the harmonic regions are the portions of the signal to be modified. In one embodiment, the harmonic regions each have the same length Δ. In other embodiments, the harmonic regions may have different lengths. 
     As further discussed below, in one embodiment, a gain is calculated for the center (or harmonic) bin of each harmonic region and the same gain so calculated is applied to all of the bins comprising the corresponding harmonic region, so there is one gain for each harmonic region. In one alternative embodiment, a gain is calculated only for the center bin of the harmonic region of the fundamental frequency P, and this same gain is applied to all the harmonic regions. 
     Referring further to  FIG. 6A , in step  605 , a gain is calculated for each frequency bin or set of frequency bins to be modified. In one embodiment, a gain is calculated for each short time Fourier transform frequency bin to be modified. In one such embodiment, the gain is zero for portions associated with a center-panned component and nonzero (e.g., one) for other portions (voice removal). Alternatively, the gain may be nonzero (e.g., greater than one) when a component of the signal is center-panned and one when it is not (voice amplification). 
     In one embodiment, the gain may vary based on the extent to which the left and right channels are center-panned (i.e., the degree of similarity between the left and right channels) and/or other factors. Components of the frequency spectra of the left and right channels that are similar are more center-panned. If voice-related signals are center-panned, attenuating similar components of the left and right channels typically attenuates voice components of the audio signal. In one such embodiment, the gain is defined as: 
     
       
         
           
             
               
                 
                   
                     G 
                     ⁡ 
                     
                       ( 
                       
                         u 
                         , 
                         k 
                       
                       ) 
                     
                   
                   ≡ 
                   
                     { 
                     
                       
                         
                           
                             
                               
                                 
                                   ( 
                                   
                                     
                                       ρ 
                                       ⁡ 
                                       
                                         ( 
                                         
                                           u 
                                           , 
                                           k 
                                         
                                         ) 
                                       
                                     
                                     
                                       ρ 
                                       0 
                                     
                                   
                                   ) 
                                 
                                 α 
                               
                               ⁢ 
                               for 
                               ⁢ 
                               
                                   
                               
                               ⁢ 
                               
                                 ρ 
                                 ⁡ 
                                 
                                   ( 
                                   
                                     u 
                                     , 
                                     k 
                                   
                                   ) 
                                 
                               
                             
                             ≤ 
                             
                               ρ 
                               0 
                             
                           
                         
                       
                       
                         
                           
                             
                               1 
                               ⁢ 
                               
                                   
                               
                               ⁢ 
                               for 
                               ⁢ 
                               
                                   
                               
                               ⁢ 
                               
                                 ρ 
                                 ⁡ 
                                 
                                   ( 
                                   
                                     u 
                                     , 
                                     k 
                                   
                                   ) 
                                 
                               
                             
                             &gt; 
                             
                               ρ 
                               0 
                             
                           
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   1 
                   ) 
                 
               
             
             
               
                 where 
               
               
                 
                     
                 
               
             
             
               
                 
                   
                     p 
                     ⁡ 
                     
                       ( 
                       
                         u 
                         , 
                         k 
                       
                       ) 
                     
                   
                   ≡ 
                   
                     
                       
                          
                         
                           
                             S 
                             L 
                           
                           - 
                           
                             S 
                             R 
                           
                         
                          
                       
                       2 
                     
                     
                       
                         
                            
                           
                             S 
                             L 
                           
                            
                         
                         2 
                       
                       + 
                       
                         
                            
                           
                             S 
                             R 
                           
                            
                         
                         2 
                       
                     
                   
                 
               
               
                 
                   ( 
                   2 
                   ) 
                 
               
             
           
         
       
     
     ρ 0  is an arbitrary threshold and α is used to control the behavior of the gain G as a function of the channel comparison function ρ (which has a low value when the channels are nearly the same and a value of zero when S L =S R ). For example, threshold values between 0.001 and 0.5 and an exponent value α=0.7 may be used. This choice of G will attenuate any signal panned to the center. 
     In one embodiment, the intent is to amplify the voice rather than attenuate it. As used herein, the term “amplify” as applied to a component of an audio signal means to increase the magnitude of that component relative to other components of the audio signal. In one embodiment, the component is amplified by attenuating portions of the audio signal not associated with the component while leaving portions associated with the component unchanged (or substantially unchanged). In one such embodiment, the gain may be defined as: 
     
       
         
           
             
               
                 G 
                 ′ 
               
               ⁡ 
               
                 ( 
                 
                   u 
                   , 
                   k 
                 
                 ) 
               
             
             ≡ 
             
               
                 1 
                 - 
                 
                   G 
                   ⁡ 
                   
                     ( 
                     
                       u 
                       , 
                       k 
                     
                     ) 
                   
                 
               
               
                 1 
                 + 
                 
                   G 
                   ⁡ 
                   
                     ( 
                     
                       u 
                       , 
                       k 
                     
                     ) 
                   
                 
               
             
           
         
       
     
     This choice of gain G′ will attenuate any components panned away from the center, and thereby amplify the center-panned component relative to such attenuated components. In one alternative embodiment, amplification is achieved by increasing the magnitude of portions of the audio signal associated with the component to be amplified while leaving portions not associated with the component unchanged (or substantially unchanged). 
     In one embodiment, time-domain smoothing of the gain values is performed to avoid erratic gain variations that can be perceived as a degradation of the signal quality. 
     In step  610  of the process shown in  FIG. 6 , the gains determined in step  605  are applied to the harmonic regions. In one embodiment, outside the harmonic regions, the gain is set to 1 when voice removal is desired, and a value between zero and one when amplification of the voice component relative to other components is desired. In one embodiment, the gain is smoothed at the boundaries of the harmonic regions as described above. In one embodiment, the gain is applied to a selected set of short time Fourier transform frequency bins to be modified, as described above. In one such embodiment, rather than calculating a gain for each bin, a gain is calculated for one bin (typically the center bin) in each harmonic region and the same gain applied to all the bins in the harmonic region. Alternatively, rather than calculating a gain for each harmonic, a gain may be calculated for one bin (typically the center bin) of the harmonic region located around the fundamental frequency and the gain applied to all the harmonic regions. 
       FIG. 7  is a block diagram illustrating one embodiment of voice removal block  110  in  FIG. 1 . In one embodiment, the audio signal may comprise a stereo signal comprising a left stereo channel and a right stereo channel, and in one such embodiment the left stereo channel and the right stereo channel are transformed separately into left and right channel short time frequency domain spectra, as shown in  FIG. 1 . The frequency domain spectra of the left and right stereo channels are labeled in  FIG. 1  as S L (u,k) and S R (u,k), respectively. S L (u,k) and S R (u,k) are provided as input to a difference determination block  704 . 
     Difference determination block  704  estimates the degree to which the signal is panned in the center. The output of difference determination block  704  is labeled as ρ(u,k) in  FIG. 7 . In one embodiment, ρ(u,k) is defined as in Equation 2. 
     For a frequency bin associated with a component of the signal that is panned in the center, the value of the difference function ρ(u,k) will be small (or zero) because the left and right channel short time frequency domain spectra S L (u,k) and S R (u,k) are similar (or the same). In contrast, for a frequency bin associate with a component of the signal that is not panned in the center, the value of ρ(u,k) will be greater. 
     The difference function values ρ(u,k) are provided as input to a gain determination block  406 . Gain determination block  706  determines, for each frequency bin for which a gain is needed, a gain G(u,k) as a function of ρ(u,k). G(u,k) is appropriately defined so that when it is applied to S L (u,k) and S R (u,k), the desired result is obtained. For example, G(u,k) may attenuate or amplify the center-panned components of the signal, and thus remove or amplify voice. As discussed above, in one embodiment, G(u,k) is defined as in Equation 1. 
     The most prominent pitch values P(u) provided as output of pitch detection block  106  also are provided as input to gain determination block  706 . Gain determination block  706  includes logic to identify the harmonic regions associated with the most prominent P(u), as described above. In one embodiment, gain determination block  706  calculates a gain for the harmonic regions only. For example, in one embodiment the gain is one by default for frequencies that are not within a harmonic region and the gain is G(u,k) for frequencies that are within the harmonic region. In one embodiment, a gain is determined for each short time Fourier transform frequency bin in each harmonic region. In one embodiment, rather than calculating a gain for each bin, a gain is calculated for one bin (typically the center bin) in each harmonic region so that the same gain is associated with all the bins in the harmonic region. Gains outside the harmonic regions are set to one. 
     The output of gain determination block G is provided as input to amplifier  702  and amplifier  710 . S L (u,k) and S R (u,k) are provided as inputs to amplifier  702  and amplifier  710 , respectively. Amplifier  702  applies the gains G(u,k) to the corresponding values S L (u,k) to produce modified left channel spectra Ŝ L (u,k). Likewise, amplifier  710  applies the gains G(u,k) to the corresponding values S R (u,k) to produce modified right channel spectra Ŝ R (u,k). 
     The voice pitch may not be the most prominent pitch in an audio signal. In one embodiment, monophonic, duophonic, or polyphonic pitch detection techniques may be used. Duophonic or polyphonic pitch detection techniques may be desirable if more than one pitch is to be modified (for example, if the most prominent pitch is not voice-related). 
     For example, some tracks have a lot of instruments panned near the center, with a voice signal that is not very prominent. In these frames, the prominent instrument rather than the voice may be attenuated if monophonic pitch detection is used. Duophonic pitch detection detects the two most prominent pitches in an audio signal and may be preferred when the voice pitch is one of the two strongest pitches in an audio signal. In other embodiments, polyphonic pitch detection (in which more than two pitches are detected) may be preferred. 
       FIG. 8  is a flowchart illustrating a duophonic technique used in one embodiment to perform pitch detection of an audio signal. In one embodiment, such duophonic pitch detection is performed in step  205  of the process shown in  FIG. 2 . In step  800 , the first most prominent pitch is identified. In one embodiment, the first most prominent pitch is identified according to the techniques described above in connection with  FIG. 3  (monophonic technique). In step  805 , the cross-correlation values are zeroed out around the most prominent pitch, its multiples (harmonics) and submultiples, to remove cross-correlation values associated with the most prominent pitch from further consideration. In step  810 , the remaining cross-correlation values are used to identify the second most prominent pitch. In one embodiment, step  308  of the process shown in  FIG. 3  is performed using the remaining cross-correlation values and the pitch associated with the maximum value among the remaining cross-correlation values is identified as the second most prominent pitch. 
     As with the monophonic pitch algorithm, in some embodiments, a voiced/unvoiced decision may be used to determine whether or not the first or second most prominent pitch belongs to the voice component. 
     In one embodiment, harmonic regions of the first and second most prominent pitches are modified. In one embodiment, steps  600 ,  605 , and  610  are repeated to modify the harmonic regions of the first and second most prominent pitches, respectively. 
       FIG. 9  is a state diagram illustrating a duck technique. In a karaoke or similar setting, the duck technique results in an improvement in the perceived quality of the audio track and allows the user to rehearse more easily. The duck technique monitors the voice input from a user as measured at a microphone being used by the user to sing or speak along with the rendered audio signal, and only attenuates the recorded voice when the user is speaking or singing. As a result, the recording is not altered unless the user speaks or sings into the microphone. The duck technique includes a “remove voice” state  900  and a “don&#39;t remove voice” state  910 . If the current state is “remove voice” state  900 , the recorded voice component is removed. If the current state is “don&#39;t remove voice” state  905 , the recorded voice component is not removed. The current state transitions from “remove voice” state  900  to “don&#39;t remove voice” state  905  when the microphone level goes below a threshold (e.g., the user has stopped signing or speaking, or is singing or speaking only softly). The current state transitions from “don&#39;t remove voice” state  905  to “remove voice” state  900  when the microphone level goes above a threshold. In one embodiment, the two thresholds are the same. In one embodiment, the two thresholds are different. In one embodiment, the two thresholds may be the same or different, as the user prefers. In one embodiment, the user may control the threshold value(s) via a user input or control. In one embodiment, the duck feature may be disabled. 
       FIG. 10  is a block diagram illustrating a system used in one embodiment to remove or amplify one or more components from a stereo recording incorporating an embodiment of a duck technique. System  1000  is system  100  modified to include a threshold detector  1010 . The user speaks or sings into a microphone. When the microphone level is below a prescribed threshold, the threshold detector  1010  sends a corresponding control signal to voice removal block  110  and in response voice removal block  110  does not modify frequency spectra S L (u,k) and S R (u,k) to remove voice. When the microphone level is above a prescribed threshold, the threshold detector  1010  sends a corresponding control signal to voice removal block  110  and in response voice removal block  110  modifies the frequency spectra S L (u,k) and S R (u,k) as described above to remove voice. As noted above, the threshold for de-activating voice removal block  110  may not be the same as the threshold for activating voice removal block  110 , depending on the embodiment. 
     In one alternative embodiment, not shown in  FIG. 10 , when the microphone level is below the prescribed threshold, a control signal is sent by threshold detector  1010  to disable both the voice removal block  110  and pitch detection block  106 , and neither voice removal nor pitch detection is performed. In one alternative embodiment, not shown in  FIG. 10 , system  1000  performs no processing when the microphone level is below the prescribed threshold. In this embodiment, input signals s L (t) and s R (t) bypass system  1000  and are passed straight to the output. 
     Although the foregoing invention has been described in some detail for purposes of clarity of understanding, it will be apparent that certain changes and modifications may be practiced within the scope of the appended claims. It should be noted that there are many alternative ways of implementing both the process and apparatus of the present invention. Accordingly, the present embodiments are to be considered as illustrative and not restrictive, and the invention is not to be limited to the details given herein, but may be modified within the scope and equivalents of the appended claims.