Abstract:
A binaural sound reproduction apparatus includes a correction filter operable to filter an input sound signal that is recorded using a binaural recording microphone and to supply the filtered signal to a headphone, an adaptive filter to which the input sound signal is supplied, and a difference detector determining a difference between a sound signal that is obtained by collecting a sound reproduced by the headphone using a sound-collecting microphone that is the same as the binaural recording microphone, or that has a similar characteristic to that of the binaural recording microphone, and a sound signal output from the adaptive filter, and for transmitting the difference to the adaptive filter. The adaptive filter determines the inverse of a synthesis characteristic from the headphone to the sound-collecting microphone based on the input sound signal and the difference, and sets the determined characteristic as a characteristic of the correction filter.

Description:
CROSS REFERENCES TO RELATED APPLICATIONS  
       [0001]     The present invention contains subject matter related to Japanese Patent Application JP 2004-131067 filed in the Japanese Patent Office on Apr. 27, 2004, the entire contents of which are incorporated herein by reference.  
       BACKGROUND OF THE INVENTION  
       [0002]     1. Field of the Invention  
         [0003]     The present invention relates to the field of audio signal processing, specifically, to a binaural sound reproduction apparatus and method.  
         [0004]     2. Description of the Related Art  
         [0005]     One known method for recording sound from sound sources placed in an acoustic space with maintaining information about the direction of incoming sound and reproducing the sound is a binaural method.  
         [0006]     In the binaural method, sound signals recorded by a stereo recording method for recording sound using small microphones around or in the right and left ear canals, in principle, these microphones, are reproduced by a headphone or the like with high fidelity in the ear canals around or in which the microphones are placed. The binaural method allows high-fidelity reproduction of a sound field in the recording site. This method can be implemented by a simple sound recording and reproduction apparatus, and is often used in the field of live recording as a convenient stereo acoustic recording and reproduction method.  
         [0007]     The most effective binaural experience is real-head recording and reproduction in which a sound recorded in the ear is reproduced in the ear. This method provides good matching of the so-called head-related transfer function (HRTF), and allows for natural tone (or optimum frequency-amplitude characteristics), optimum sound image localization, high sound image quality, etc.  
         [0008]     Japanese Unexamined Patent Application Publication No. 6-217400 discloses a binaural sound recording and reproducing method that allows a listener to receive a reverberant sound in the sound-collecting system in three-dimensional perception of sound while localizing a sound source with natural tone in front of the listener.  
       SUMMARY OF THE INVENTION  
       [0009]     The real-head recording and reproduction seems to be ideal, but may be actually ineffective in that natural tone reproduction is not achieved, sound is reproduced with a diffuse sound image, sound is unclearly localized, etc.  
         [0010]     These problems arise because the implicit requirements of binaural sound recording and reproduction, that is, flat and equal transfer characteristics from the recording system to the reproduction system in the right and left channels, are not met. Specifically, the characteristic of the microphone used for recording sound is not flat, the characteristics of the right and left microphones differ from each other, the characteristic of the headphone used for reproducing sound is not flat, or the characteristics of the right and left acoustic converters in the headphone differ from each other, thus reducing the binaural effects.  
         [0011]     For example, up and down localization and front and back localization of a sound image are caused by tone to some extent. The difference in level and time (including phase) between the right ear and the left ear greatly affects right and left localization.  
         [0012]     Actually, this problem is not straightforward because, for example, the transfer characteristic to the ear often changes if a user takes off a headphone and puts on the same headphone again. In particular, large circumaural headphones, such as closed headphones, have more play, and therefore suffer changes in the transfer characteristic during listening. Generally, inexpensive headphones do not have uniform characteristics in the right and left channels. Generally, inexpensive microphones do not have uniform characteristics and sensitivity.  
         [0013]     It is therefore desirable to improve the binaural effects even if the transfer characteristic from the recording system to the reproduction system in real-head recording and reproduction is not flat and the transfer characteristics of the right and left channels differ from each other.  
         [0014]     According to an embodiment of the present invention, there is provided a binaural sound reproduction apparatus including a correction filter operable to filter an input sound signal that is recorded using a binaural recording microphone and to supply the filtered signal to a headphone, an adaptive filter to which the input sound signal is supplied, and difference detecting means for determining a difference between a sound signal that is obtained by collecting a sound reproduced by the headphone using a sound-collecting microphone that is the same as the binaural recording microphone or that has a similar characteristic to that of the binaural recording microphone and a sound signal output from the adaptive filter, and for transmitting the difference to the adaptive filter. The adaptive filter determines the inverse of a synthesis characteristic from the headphone to the sound-collecting microphone based on the input sound signal and the difference, and sets the determined characteristic as a characteristic of the correction filter.  
         [0015]     A user wears a microphone used for binaural recording or a microphone whose characteristic is similar to that of the microphone used for binaural recording in the user&#39;s ear in a similar manner to that during recording, and also wears a headphone. When an acoustic measurement signal, such as an impulse, is reproduced from the headphone and is collected by the microphone, the characteristic of the signal from the microphone is a characteristic in which the characteristic of the headphone, the acoustic transfer characteristic (spatial transfer characteristic) from the headphone position to the microphone position, and the characteristic of the microphone are synthesized.  
         [0016]     The characteristic that is similar to the synthesis characteristic and that is the inverse of the synthesis characteristic (hereinafter referred to as an “inverse synthesis characteristic”) is determined. The inverse synthesis characteristic may be used to correct for the characteristic of the headphone, the acoustic transfer characteristic from the headphone position to the microphone position, and the characteristic of the microphone.  
         [0017]     Specifically, a correction filter superimposes the inverse synthesis characteristic onto a binaural signal recorded by a recording microphone. The binaural signal has originally the characteristic of the recording microphone superimposed thereon. Since the inverse synthesis characteristic includes the inverse of the characteristic of the recording microphone, the characteristic of the recording microphone superimposed on the binaural signal is corrected for to produce the same binaural signal as that recorded by a microphone whose characteristic is flat.  
         [0018]     The inverse synthesis characteristic also includes the inverse of the characteristic of the headphone, and a signal to be reproduced by a headphone whose characteristic is flat is therefore obtained as the reproduction signal.  
         [0019]     The inverse synthesis characteristic also includes the inverse of the acoustic transfer characteristic from the headphone to the microphone, and therefore sound is transmitted from the headphone to the microphone with a flat transfer characteristic. Thus, the spatial transfer characteristic that depends upon the individual ear shape when a circumaural headphone is used can be corrected for.  
         [0020]     In the binaural sound reproduction apparatus, therefore, the correction filter superimposes the inverse synthesis characteristic onto a binaural signal obtained by the recording microphone to produce the optimum binaural reproduction signal even if a microphone and a headphone that do not exhibit a flat characteristic or uniform characteristics in the right and left channels are used.  
         [0021]     The recording microphone and the sound-collecting microphone (which is a microphone used for reproducing sound) are not necessarily the same as long as these microphones have similar characteristics.  
         [0022]     The user who binaurally records material and the user who binaurally reproduces sound (or the listener) are not necessarily the same. It is ideal that these users be the same, otherwise, as long as the characteristics of the microphone and the headphone are corrected for in the manner described above, the binaural effects can be improved.  
         [0023]     Furthermore, an ordinary sound signal used as a binaural signal may be reproduced without using a special acoustic measurement signal such as an impulse. The adaptive filter determines the inverse of a synthesis characteristic from the headphone position to the microphone position, and the correction filter superimposes the inverse synthesis characteristic onto the ordinary sound signal used as a binaural signal. Thus, the characteristics of the microphone and the headphone can be corrected for.  
         [0024]     Therefore, binaural sound reproduction can be achieved with little burden on the listener. Moreover, the characteristic of the correction filter can sequentially be updated. A change in characteristics due to the movement of the headphone worn by the listener during reproduction and listening can be dealt with.  
         [0025]     Accordingly, the binaural effects can be improved even if the transfer characteristic from the recording system to the reproduction system in real-head recording and reproduction is not flat and the transfer characteristics of the right and left channels differ from each other. Furthermore, the binaural effects can be improved without using a special acoustic measurement signal. Moreover, a change in characteristics due to the movement of the headphone worn by the listener during reproduction and listening can be dealt with. 
     
    
     BRIEF DESCRIPTION OF THE DRAWINGS  
       [0026]      FIG. 1  is a diagram of a binaural recording mechanism;  
         [0027]      FIG. 2  is an illustration of a listener who wears a microphone and a headphone during binaural sound reproduction;  
         [0028]      FIG. 3  is a block diagram of a binaural sound reproduction apparatus according to an embodiment of the present invention;  
         [0029]      FIG. 4  is an adaptive algorithm for updating the characteristics of a correction filter and an adaptive filter according to an embodiment of the present invention; and  
         [0030]      FIG. 5  is a graph showing estimation and convergence of the adaptive filter according to an embodiment of the present invention.  
     
    
     DESCRIPTION OF THE PREFERRED EMBODIMENTS  
       [0031]     A binaural recording method prior to a binaural sound reproduction method according to an embodiment of the present invention will now be described with  FIG. 1 .  
         [0032]     In this binaural sound recording method, in a recording site, a user  1  wears small recording microphones  3 L and  3 R around or in the left ear canal  2 L and the right ear canal  2 R, and the microphones  3 L and  3 R are used to stereo record sound from sound sources in the acoustic space in the recording site.  
         [0033]     The recording microphones  3 L and  3 R are not necessarily the same as microphones used for reproducing sound as long as the characteristics of the microphones  3 L and  3 R are similar to those of microphones used for reproducing sound, described below. The user  1  (who binaurally records material) may not be a listener (who binaurally reproduces the material). For example, an artificial head or simulated head that simulates the human head may be used.  
         [0034]     Sound signals SL and SR are input as binaural signals to a recording device  10  from the microphones  3 L and  3 R, and are then recorded in a recording medium  4 , such as a disk medium or a memory card.  
         [0035]     Specifically, in the recording device  10 , the input sound signals SL and SR are amplified by sound amplification circuits  11 L and  11 R, respectively, and are then converted into digital sound data DL and DR by analog-to-digital (AD) converters  12 L and  12 R, respectively. The resulting digital sound data DL and DR are input to a recording processor  13 , and are then recorded in the recording medium as sound data having a predetermined format after they are compressed and encoded, if necessary.  
         [0036]     In the recording device  10 , a system controller  15  controls the recording processor  13  and a medium drive  14 .  
         [0037]     Accordingly, the sound reaching the human ear or the ear of the artificial head is recorded in the recording site. The right-channel sound and the left-channel sound in the right and left ear canals or at the microphones  3 R and  3 L are recorded with the difference in level, time, and characteristics depending upon the positional or directional relationship between the microphones  3 R and  3 L and the sound sources.  
         [0038]     In order to reproduce the binaural signals recorded in the manner described above, as shown in  FIG. 2 , a listener  5  wears microphones  7 R and  7 L around the right and left ear canals  6 R and  6 L, respectively, and also wears a headphone  9 .  
         [0039]     The microphones  7 L and  7 R are small and are not circumaural. Preferably, the microphones  7 L and  7 R are the same as the recording microphones  3 L and  3 R shown in  FIG. 1 , but may be different from the microphones  3 L and  3 R as long as the characteristics of the microphones  7 L and  7 R are similar to those of the microphones  3 L and  3 R.  
         [0040]     As shown in  FIG. 2 , the headphone  9  may be of the circumaural type that a right-channel acoustic converter  9 R and a left-channel acoustic converter  9 L covers the right and left ears  6 R and  6 L of the listener  5 , respectively. Alternatively, the headphone  9  may be an open headphone or a semi-open headphone.  
         [0041]      FIG. 3  shows a binaural sound reproduction mechanism allowing the listener  5  to reproduce the binaural signals recorded in the manner described above using the microphones  7 L and  7 R and the headphone  9 .  
         [0042]     In a sound reproduction apparatus  20 , the recorded binaural signals are read from the recording medium  4 . The read binaural signals are subjected to processing, such as extension and decoding, if necessary, by a sound reproduction processor  23 , and digital sound data DL and DR, which are binaural signals, are output from the sound reproduction processor  23 .  
         [0043]     In the sound reproduction apparatus  20 , a media drive  24  and the sound reproduction processor  23  are controlled by a system controller  25 .  
         [0044]     The sound data DL and DR output from the sound reproduction processor  23  of the sound reproduction apparatus  20  are supplied to a binaural sound reproduction apparatus  30 .  
         [0045]     The binaural sound reproduction apparatus  30  is composed of a left-channel system and a right-channel system. The sound data DL is supplied to the left-channel system, and the sound data DR is supplied to the right-channel system.  
         [0046]     In the left-channel system, the sound data DL is supplied as input sound data x(k) to a correction filter  31 L and an adaptive filter  34 L. The sound data output from the correction filter  31 L is converted by a digital-to-analog (DA) converter  32 L into an analog sound signal, and the resulting sound signal is supplied to the left-channel headphone acoustic converter  9 L. Thus, the left-channel sound is reproduced.  
         [0047]     The reproduced left-channel sound is collected using a microphone  7 L near a left-channel headphone acoustic converter  9 L. The sound signal output from the microphone  7 L is converted by an AD converter  33 L into digital sound data d(k). An adder circuit  35 L subtracts sound data y(k) output from the adaptive filter  34 L from the sound data d(k) output from the AD converter  33 L, and difference data e(k) output from the adder circuit  35 L is transmitted to the adaptive filter  34 L.  
         [0048]     The adaptive filter  34 L is composed of a finite impulse response (FIR) filter type adaptive linear coupler (filter unit) and an adaptive algorithm computation unit (filter coefficient update computation unit). The adaptive filter  34 L is operable to estimate a transfer characteristic, in which the characteristic of the correction filter  31 L, the characteristic of the left-channel headphone acoustic converter  9 L, the spatial transfer characteristic (acoustic transfer characteristic) from the left-channel headphone acoustic converter  9 L to the microphone  7 L, and the characteristic of the microphone  7 L are synthesized, according to an adaptive algorithm (filter coefficient updating algorithm) described below. If the characteristic of the correction filter  31 L is flat, the adaptive filter  34 L estimates a transfer characteristic, in which the characteristic of the left-channel headphone acoustic converter  9 L, the spatial transfer characteristic from the left-channel headphone acoustic converter  9 L to the microphone  7 L, and the characteristic of the microphone  7 L are synthesized.  
         [0049]     The correction filter  31 L is composed of, for example, an FIR filter. The filter coefficient of the correction filter  31 L is modified by the adaptive filter  34 L, and the filter characteristic is updated.  
         [0050]     The right-channel system includes a right-channel headphone acoustic converter  9 R, a microphone  7 R, a correction filter  31 R, a DA converter  32 R, an AD converter  33 R, an adaptive filter  34 R, and an adder circuit  35 R, and processes the input sound data DR.  
         [0051]     In ideal estimation performed by the adaptive filter  34 L and update of the characteristic of the correction filter  31 L, first, the characteristic of the correction filter  31 L is made flat, and the adaptive filter  34 L estimates a transfer characteristic, in which the characteristic of the left-channel headphone acoustic converter  9 L, the spatial transfer characteristic from the left-channel headphone acoustic converter  9 L to the microphone  7 L, and the characteristic of the microphone  7 L are synthesized. Then, the filter coefficient of the correction filter  31 L is updated so that the characteristic of the correction filter  31 L is the inverse of the transfer characteristic. The characteristic of the adaptive filter  34 L is also made a flat, thus achieving estimation by the adaptive filter  34 L.  
         [0052]     However, actually, a transfer characteristic, in which the characteristic of the left-channel headphone acoustic converter  9 L, the spatial transfer characteristic from the left-channel headphone acoustic converter  9 L to the microphone  7 L, and the characteristic of the microphone  7 L are synthesized, may contain, in form of a dip, a frequency region having a low signal level. The inverse of such a transfer characteristic has a frequency region with large gain, and therefore is not suitable for a filter.  
         [0053]     Preferably, the characteristic of the correction filter  31 L is updated by limiting the signal level within a certain range using a limiter or the like so as not to contain such a large-gain frequency region. A portion that is not reflected in the characteristic of the correction filter  31 L remains in the characteristic of the adaptive filter  34 L. Thus, the adaptive filter  34 L is convergent.  
         [0054]     While a case where the characteristic of the correction filter  31 L is not updated until the adaptive filter  34 L performs estimation and is convergent has been described, the characteristic of the correction filter  31 L may be updated at any time before the adaptive filter  34 L performs estimation and is convergent.  
         [0055]      FIG. 4  shows an adaptive algorithm for the left-channel system in a case where the characteristic of the correction filter  31 L is updated at any time. Although  FIG. 4  shows only the level characteristic (in each characteristic chart shown in  FIG. 4 , the x-axis indicates the frequency (f) and the y-axis indicates the level in decibels (dB)), the estimation and convergence of the adaptive filter  34 L and the update of the characteristic of the correction filter  31 L are also performed with respect to the phase (including delay) characteristic. This adaptive algorithm also applies to the right-channel system.  
         [0056]     When sound reproduction begins, as indicated by the “initial state” shown in  FIG. 4 , the level characteristic of the correction filter  31 L is 0 dB and flat, which means that an input signal is transmitted, and the level characteristic of the adaptive filter  34 L is lower. Thus, if the input sound data x(k) constantly contains a low-level frequency region, the adaptive filter  34 L can easily perform estimation. Although, in  FIG. 4 , the level characteristic of the adaptive filter  34 L is also flat at the initial state, the level characteristic of the adaptive filter  34 L may not be flat.  
         [0057]     Then, the coefficient of the adaptive filter  34 L is updated to some extent without modifying the correction filter  31 L. Thus, as indicated as “after update of adaptive filter” shown in  FIG. 4 , the level characteristic of the adaptive filter  34 L changes. At this stage, the adaptive filter  34 L does not complete estimation or convergence.  
         [0058]     At this time, the characteristic of the correction filter  31 L is updated. As indicated by “characteristic-ratio calculation” shown in  FIG. 4 , the characteristic ratio of (level characteristic of adaptive filter)/(level characteristic of correction filter) is calculated. Since the level characteristic of the correction filter  31 L is 0 dB and flat, the characteristic ratio of (level characteristic of adaptive filter)/(level characteristic of correction filter) is equal to the level characteristic of the adaptive filter  34 L.  
         [0059]     The resulting characteristic is divided into a non-correction region and a correction region that are bounded at a threshold level of A dB. A frequency region having a level equal to or lower than A dB is referred to as a non-correction region, and a frequency region having a level higher than A dB is referred to as a correction region.  
         [0060]     As indicated by “update of both filter characteristics” shown in  FIG. 4 , the level characteristic of the adaptive filter  34 L and the level characteristic of the correction fitter  31 L are updated. The level characteristic of the adaptive filter  34 L is updated so that, in the non-correction region, the previous level characteristic in this region remains and the level characteristic in the correction region is set to A dB. The level characteristic of the correction filter  31 L is updated so that the level in the non-correction region is maintained at 0 dB and the level in the correction region is set to the level obtained by subtracting the level of the characteristic ratio of (level characteristic of adaptive filter)/(level characteristic of correction filter) from A dB.  
         [0061]     An operation similar to the operation indicated by “after update of adaptive filter”, “characteristic-ratio calculation”, and “update of both filter characteristics” is repeatedly performed to form, as the correction filter  31 L, a practical correction filter exhibiting a level characteristic whose peak is suppressed, and the adaptive filter  34 L is to complete estimation and convergence.  
         [0062]     Moreover, the characteristics of the adaptive filter  34 L and the correction filter  31 L are sequentially updated. If the spatial transfer characteristic from the left-channel headphone acoustic converter  9 L to the microphone  7 L changes due to the movement of the worn headphone  9  during reproduction and listening, the characteristics of the adaptive filter  34 L and the correction filter  31 L are also updated, and the adaptive filter  34 L completes estimation and convergence.  
         [0063]      FIG. 5  is a graph showing estimation and convergence of the adaptive filter  34 L. Sound reproduction begins at ts. At ts, the adaptive filters  34 L and  34 R are in an estimation/convergence incompletion state. The update of the characteristics, described above, allows the adaptive filters  34 L and  34 R to become in an estimation/convergence completion state immediately after ts. At ta, if some disturbance occurs in the binaural sound reproduction system due to, for example, the movement of the worn headphone  9  and, the adaptive filters  34 L and  34 R are in the estimation/convergence incompletion state for a short period of time. However, after the update of the characteristics, described above, the adaptive filters  34 L and  34 R are in the estimation/convergence completion state again.  
         [0064]     In the example shown in  FIG. 4 , a threshold level of A dB between the non-correction region and the correction region is fixed. The sound frequency region may be divided into a plurality of frequency sub-regions, and the threshold level between the non-correction region and the correction region may be different in each frequency sub-region, or the threshold level between the non-correction region and the correction region may be changed depending upon the situation, e.g., at the beginning of sound reproduction, during sound reproduction, etc.  
         [0065]     Although the binaural sound reproduction apparatus  30  shown in  FIG. 3  integrally includes the headphone acoustic converters  9 L and  9 R and the microphones  7 L and  7 R, a binaural sound reproduction apparatus including only a signal processor, which is separate from the headphone acoustic converters  9 L and  9 R and the microphones  7 L and  7 R, may be used. Therefore, a listener can use an existing headphone in combination with the microphones  7 L and  7 R and the binaural sound reproduction apparatus including a signal processor to achieve stable binaural sound reproduction.  
         [0066]     The signal processing in a binaural sound reproduction apparatus according to an embodiment of the present invention may be implemented by software executed by a digital signal processor (DSP) or a central processing unit (CPU). For example, a headphone worn by a listener can be connected to a standard headphone output terminal of a personal computer (PC), and a microphone around the listener&#39;s ear canals can be connected to a standard microphone input terminal of the PC so that the signal processing described above can be executed by the CPU or the like in the PC. The signal processing can be obtained in form of software (program) executable on the PC or the like. Thus, the listener can realize binaural sound reproduction according to an embodiment of the present invention.  
         [0067]     It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and alterations may occur depending on design requirements and other factors insofar as they are within the scope of the appended claims or the equivalents thereof.