Patent Publication Number: US-11647344-B2

Title: Hearing device with end-to-end neural network

Description:
CROSS-REFERENCE TO RELATED APPLICATION 
     This application claims priority under 35 USC 119(e) to U.S. provisional application No. 63/171,592, filed on Apr. 7, 2021, the content of which is incorporated herein by reference in its entirety. 
    
    
     BACKGROUND OF THE INVENTION 
     Field of the Invention 
     The invention relates to hearing devices, and more particularly, to a hearing device with an end-to-end neural network for reducing comb-filtering effect by performing active noise cancellation and audio signal processing. 
     Description of the Related Art 
     It is hard for people to adjust to hearing aids. The fact is that no matter how good a hearing aid is, it always sounds like a hearing aid. A significant cause of this is the “comb-filter effect,” which arises because the digital signal processing in the hearing aid delays the amplified sound relative to the leak-path/direct sound that enters the ear through venting in the ear tip and any leakage around it. The delay is the time that the hearing aid takes to (1) sample and convert an analog audio signal into a digital audio signal; (2) perform digital signal processing; (3) convert the processed signal into an analog audio signal to be delivered to the hearing aid speaker. Prior experiments showed even a delay of around 2 milliseconds (ms) results in clear comb-filtering effect, while ultralow delay below 0.5 ms does not. This delay is perceived as echoes or reverberation by the person wearing a hearing aid and listening to the environmental sounds such as speeches and background noises. The comb-filter effect significantly reduces the sound quality. 
     As well known in the art, the sound through the leak path (i.e., direct sound) can be removed by introducing Active Noise Cancellation (ANC). After the direct sound is cancelled, the comb-filter effect would be mitigated. US Pub. No. 2020/0221236A1 disclosed a hearing device with an additional ANC circuit for cancelling the sound through the leak path. Theoretically, the ANC circuit may operate in time domain or frequency domain. Normally, the ANC circuit in the hearing aid includes one or more time-domain filters because the signal processing delay of the ANC circuit is typically required to be less than 50 μs. For the ANC circuit operating in frequency domain, the short-time Fourier Transform (STFT) and the inverse STFT processes contribute the signal processing delays ranging from 5 to 50 milliseconds (ms), which includes the effect of ANC circuit. However, most state-of-the-art audio algorithms manipulate audio signals in frequency domain for advanced audio signal processing. 
     What is needed is a hearing device for integrating time-domain and frequency-domain audio signal processing, reducing comb-filtering effect, performing ANC and advanced audio signal processing, and improving audio quality. 
     SUMMARY OF THE INVENTION 
     In view of the above-mentioned problems, an object of the invention is to provide a hearing device capable of integrating time-domain and frequency-domain audio signal processing and improving audio quality. 
     One embodiment of the invention provides a hearing device. The hearing device comprises a main microphone, M auxiliary microphones, a transform circuit, at least one processor, at least one storage media and a post-processing circuit. The main microphone and M auxiliary microphones respectively generate a main audio signal and M auxiliary audio signals. The transform circuit respectively transforms multiple first sample values in current frames of the main audio signal and the M auxiliary audio signals into a main spectral representation and M auxiliary spectral representations. The at least one memory including instructions operable to be executed by the at least one processor to perform a set of operations comprising: performing active noise cancellation (ANC) operations over the first sample values using an end-to-end neural network to generate multiple second sample values; and, performing audio signal processing operations over the main spectral representation and the M auxiliary spectral representations using the end-to-end neural network to generate a compensation mask. The post-processing circuit modifies the main spectral representation with the compensation mask to generate a compensated spectral representation, and generates an output audio signal according to the second sample values and the compensated spectral representation, where M&gt;=0. 
     Another embodiment of the invention provides an audio processing method applicable to a hearing device. The audio processing method comprises: providing a main audio signal by a main microphone and M auxiliary audio signals by M auxiliary microphones, where M&gt;=0; respectively transforming first sample values in current frames of the main audio signal and the M auxiliary audio signals into a main spectral representation and M auxiliary spectral representations; performing active noise cancellation (ANC) operations over the first sample values using an end-to-end neural network to obtain multiple second sample values; performing audio signal processing operations over the main spectral representation and the M auxiliary spectral representations using the end-to-end neural network to obtain a compensation mask; modifying the main spectral representation with the compensation mask to obtain a compensated spectral representation; and, obtaining an output audio signal according to the second sample values and the compensated spectral representation. 
     Further scope of the applicability of the present invention will become apparent from the detailed description given hereinafter. However, it should be understood that the detailed description and specific examples, while indicating preferred embodiments of the invention, are given by way of illustration only, since various changes and modifications within the spirit and scope of the invention will become apparent to those skilled in the art from this detailed description. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       The present invention will become more fully understood from the detailed description given hereinbelow and the accompanying drawings which are given by way of illustration only, and thus are not limitative of the present invention, and wherein: 
         FIG.  1    is a schematic diagram of a hearing device according to the invention. 
         FIG.  2    is a schematic diagram of the pre-processing unit  120  according to an embodiment of the invention. 
         FIG.  3    is a schematic diagram of an end-to-end neural network  130  according to an embodiment of the invention. 
         FIG.  4    is a schematic diagram of the post-processing unit  150  according to an embodiment of the invention. 
         FIG.  5    is a schematic diagram of the blending unit  42   k  according to an embodiment of the invention. 
     
    
    
     DETAILED DESCRIPTION OF THE INVENTION 
     As used herein and in the claims, the term “and/or” includes any and all combinations of one or more of the associated listed items. The use of the terms “a” and “an” and “the” and similar referents in the context of describing the invention are to be construed to cover both the singular and the plural, unless otherwise indicated herein or clearly contradicted by context. Throughout the specification, the same components with the same function are designated with the same reference numerals. 
     A feature of the invention is to use an end-to-end neural network to simultaneously perform ANC function and advanced audio signal processing, e.g., noise suppression, acoustic feedback cancellation (AFC) and sound amplification and so on. Another feature of the invention is that the end-to-end neural network receives a time-domain audio signal and a frequency-domain audio signal for each microphone so as to gain the benefits of both time-domain signal processing (e.g., extremely low system latency) and frequency-domain signal processing (e.g., better frequency analysis). In comparison with the conventional ANC technology that is most effective on lower frequencies of sound, e.g., between 50 to 1000 Hz, the end-to-end neural network of the invention can reduce both the high-frequency noise and low-frequency noise. 
       FIG.  1    is a schematic diagram of a hearing device according to the invention. Referring to  FIG.  1   , the hearing device  100  of the invention includes a number Q of microphones  11 ˜ 1 Q, a pre-processing unit  120 , an end-to-end neural network  130 , a post-processing unit  150  and an output circuit  160 , where Q&gt;=1. The hearing device  100  may be a hearing aid, e.g. of the behind-the-ear (BTE) type, in-the-ear (ITE) type, in-the-canal (ITC) type, or completely-in-the-canal (CIC) type. 
     A main microphone  11 , located outside the ear, is used to collect ambient sound to generate a main audio signal au- 1 . If Q&gt;1, at least one auxiliary microphone  12 ˜ 1 Q generates at least one auxiliary audio signal au- 2 ˜au-Q. The pre-processing unit  120  is configured to receive Q audio signals au- 1 ˜au-Q and generate audio data of current frames i of Q time-domain digital audio signals s 1 [n]˜s Q [n] and Q current spectral representations F 1 ( i )˜FQ(i) corresponding to the audio data of the current frames i of time-domain digital audio signals s 1 [n]˜s Q [n], where n denotes the discrete time index and i denotes the frame index of the time-domain digital audio signals s 1 [n]˜s Q [n]. The end-to-end neural network  130  receives input parameters, the Q current spectral representations F 1 ( i )˜FQ(i) and audio data for current frames i of the Q time-domain signals s 1 [n]˜s Q [n], performs ANC and AFC functions, noise suppression and sound amplification to generate a frequency-domain compensation mask stream G 1 (i)˜G N (i) and audio data of the current frame i of a time-domain digital data stream u[n]. The post-processing unit  150  receives the frequency-domain compensation mask stream G 1 (i)˜G N (i) and audio data of the current frame i of the time-domain data stream u[n] to generate audio data for the current frame i of a time-domain digital audio signal y[n], where N denotes the Fast Fourier transform (FFT) size. Finally, the output circuit  160  converts the digital audio signal y[n] into a sound pressure signal in an ear canal of the user. The output circuit  160  includes a digital to analog converter (DAC) 161 , an amplifier  162  and a loudspeaker  163 . 
       FIG.  2    is a schematic diagram of the pre-processing unit  120  according to an embodiment of the invention. Referring to  FIG.  2   , if the outputs of the Q microphones  11 ˜ 1 Q are analog audio signals, the pre-processing unit  120  includes Q analog-to-digital converters (ADC)  121 , Q STFT blocks  122  and Q parallel-to-serial converters (PSC)  123 ; if the outputs of the Q microphones  11 ˜ 1 Q are digital audio signals, the pre-processing unit  120  only includes Q STFT blocks  122  and Q PSC  123 . Thus, the ADCs  121  are optional and represented by dash lines in  FIG.  2   . The ADCs  121  respectively convert Q analog audio signals (au- 1 ˜au-Q) into Q digital audio signals (s 1 [n]˜s Q [n]). In each STFT block  122 , the digital audio signal s j [n] is firstly broken up into frames using a sliding widow along the time axis so that the frames overlap each other to reduce artifacts at the boundary, and then, the audio data in each frame in time domain is transformed by FFT into complex-valued data in frequency domain. Assuming a number of sampling points in each frame (or the FFT size) is N, the time duration for each frame is Td and the frames overlap each other by Td/2, each STFT block  122  divides the audio signal s j [n] into a plurality of frames and computes the FFT of audio data in the current frame i of a corresponding audio signal s j [n] to generate a current spectral representation Fj(i) having N complex-valued samples (F 1,j (i)˜F N,j (i)) with a frequency resolution of fs/N(=1/Td), where 1&lt;=j&lt;=Q. Here, fs denotes a sampling frequency of the digital audio signal s j [n] and each frame corresponds to a different time interval of the digital audio signal s j [n]. In a preferred embodiment, the time duration Td of each frame is about 32 milliseconds (ms). However, the above time duration Td is provided by way of example and not limitation of the invention. In actual implementations, other time duration Td may be used. Finally, each PSC  123  converts the corresponding N parallel complex-valued samples (F 1,j (i)˜F N,j (i)) into a serial sample stream, starting from F 1,j (i) and ending with F N,j (i). Please note that the 2*Q data streams F 1 ( i )˜FQ(i) and s 1 [n]˜s Q [n] outputted from the pre-processing unit  120  are synchronized so that 2*Q elements in each column (e.g., F 1,1 (i), s 1 [ 1 ], . . . , F 1,Q (i), s Q [ 1 ] in one column) from the 2*Q data streams F 1 ( i )˜FQ(i) and s 1 [n]˜s Q [n] are aligned with each other and sent to the end-to-end neural network  130  at the same time. 
     The pre-processing unit  120 , the end-to-end neural network  130  and the post-processing unit  150  may be implemented by software, hardware, firmware, or a combination thereof. In one embodiment, the pre-processing unit  120 , the end-to-end neural network  130  and the post-processing unit  150  are implemented by at least one processor and at least one storage media (not shown). The at least one storage media stores instructions/program codes operable to be executed by the at least one processor to cause the processor to function as: the pre-processing unit  120 , the end-to-end neural network  130  and the post-processing unit  150 . In an alternative embodiment, only the end-to-end neural network  130  is implemented by at least one processor and at least one storage media (not shown). The at least one storage media stores instructions/program codes operable to be executed by the at least one processor to cause the at least one processor to function as: the end-to-end neural network  130 . 
     The end-to-end neural network  130  may be implemented by a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a time delay neural network (TDNN) or any combination thereof. Various machine learning techniques associated with supervised learning may be used to train a model of the end-to-end neural network  130  (hereinafter called “model  130 ” for short). Example supervised learning techniques to train the end-to-end neural network  130  include, without limitation, stochastic gradient descent (SGD). In supervised learning, a function ƒ (i.e., the model  130 ) is created by using four sets of labeled training examples (will be described below), each of which consists of an input feature vector and a labeled output. The end-to-end neural network  130  is configured to use the four sets of labeled training examples to learn or estimate the function ƒ (i.e., the model  130 ), and then to update model weights using the backpropagation algorithm in combination with cost function. Backpropagation iteratively computes the gradient of cost function relative to each weight and bias, then updates the weights and biases in the opposite direction of the gradient, to find a local minimum. The goal of a learning in the end-to-end neural network  130  is to minimize the cost function given the four sets of labeled training examples. 
       FIG.  3    is a schematic diagram of an end-to-end neural network  130  according to an embodiment of the invention. In a preferred embodiment, referring to  FIG.  3   , the end-to-end neural network  130  includes a time delay neural network (TDNN)  131 , a frequency-domain long short-term memory (FD-LSTM) network  132  and a time-domain long short-term memory (TD-LSTM) network  133 . In this embodiment, the TDNN  131  with “shift-invariance” property is used to process time series audio data. The significance of shift invariance is that it avoids the difficulties of automatic segmentation of the speech signal to be recognized by the uses of layers of shifting time-windows. The LSTM networks  132 ˜ 133  have feedback connections and thus are well-suited to processing and making predictions based on time series audio data, since there can be lags of unknown duration between important events in a time series. Besides, the TDNN  131  is capable of extracting short-term (e.g., less than 100 ms) audio features such as magnitudes, phases, pitches and non-stationary sounds, while the LSTM networks  132 ˜ 133  are capable of extracting long-term (e.g., ranging from 100 ms to 3 seconds) audio features such as scenes, and sounds correlated with the scenes. Please be noted that the above embodiment (TDNN  131  with FD-LSTM network  132  and TD-LSTM network  133 ) is provided by way of example and not limitations of the invention. In actual implementations, any other type of neural networks can be used and this also falls in the scope of the invention. 
     According to the input parameters, the end-to-end neural network  130  receives the Q current spectral representations F 1 ( i )˜FQ(i) and audio data of the current frames i of Q time-domain input streams s 1 [n]˜s Q [n] in parallel, performs ANC function and advanced audio signal processing and generates one frequency-domain compensation mask stream (including N mask values G 1 (i)˜G N (i)) corresponding to N frequency bands and audio data of the current frame i of one time-domain output sample stream u[n]. Here, the advanced audio signal processing includes, without limitations, noise suppression, AFC, sound amplification, alarm-preserving, environmental classification, direction of arrival (DOA) and beamforming, speech separation and wearing detection. For purpose of clarity and ease of description, the following embodiments are described with the advanced audio signal processing only including noise suppression, AFC and sound amplification. However, it should be understood that the embodiments of the the end-to-end neural network  130  are not so limited, but are generally applicable to other types of audio signal processing, such as environmental classification, direction of arrival (DOA) and beamforming, speech separation and wearing detection. 
     For the sound amplification function, the input parameters for the end-to-end neural network  130  include, with limitations, magnitude gains, a maximum output power value of the signal z[n] (i.e., the output of inverse STFT  154 ) and a set of N modification gains g 1 ˜g N  corresponding to N mask values G 1 (i)˜G N (i), where the N modification gains g 1 ˜g N  are used to modify the waveform of the N mask values G 1 (i)˜G N (i). For the noise suppression, AFC and ANC functions, the input parameters for the end-to-end neural network  130  include, with limitations, level or strength of suppression. For the noise suppression function, the input data for a first set of labeled training examples are constructed artificially by adding various noise to clean speech data, and the ground truth (or labeled output) for each example in the first set of labeled training examples requires a frequency-domain compensation mask stream (including N mask values G 1 (i)˜G N (i)) for corresponding clean speech data. For the sound amplification function, the input data for a second set of labeled training examples are weak speech data, and the ground truth for each example in the second set of labeled training examples requires a frequency-domain compensation mask stream (including N mask values G 1 (i)˜G N (i)) for corresponding amplified speech data based on corresponding input parameters (e.g., including a corresponding magnitude gain, a corresponding maximum output power value of the signal z[n] and a corresponding set of N modification gains g 1 ˜g N ). For the AFC function, the input data for a third set of labeled training examples are constructed artificially by adding various feedback interference data to clean speech data, and the ground truth for each example in the third set of labeled training examples requires a frequency-domain compensation mask stream (including N mask values G 1 (i)˜G N (i)) for corresponding clean speech data. For the ANC function, the input data for a fourth set of labeled training examples are constructed artificially by adding the direct sound data to clean speech data, the ground truth for each example in the fourth set of labeled training examples requires N sample values of the time-domain denoised audio data u[n] for corresponding clean speech data. For speech data, a wide range of people&#39;s speech is collected, such as people of different genders, different ages, different races and different language families. For noise data, various sources of noise are used, including markets, computer fans, crowd, car, airplane, construction, etc. For the feedback interference data, interference data at various coupling levels between the loudspeaker  163  and the microphones  11 ˜ 1 Q are collected. For the direct sound data, the sound from the inputs of the hearing devices to the user eardrums among a wide range of users are collected. During the process of artificially constructing the input data, each of the noise data, the feedback interference data and the direct sound data is mixed at different levels with the clean speech data to produce a wide range of SNRs for the four sets of labeled training examples. 
     In a training phase, the TDNN  131  and the FD-LSTM network  132  are jointly trained with the first, the second and the third sets of labeled training examples, each labeled as a corresponding frequency-domain compensation mask stream (including N mask values G 1 (i)˜G N (i)); the TDNN  131  and the TD-LSTM network  133  are jointly trained with the fourth set of labeled training examples, each labeled as N corresponding time-domain audio sample values. When trained, the TDNN  131  and the FD-LSTM network  132  can process new unlabeled audio data, for example audio feature vectors, to generate N corresponding frequency-domain mask values G 1 (i)˜G N (i) for the N frequency bands while the TDNN  131  and the TD-LSTM network  133  can process new unlabeled audio data, for example audio feature vectors, to generate N corresponding time-domain audio sample values for the current frame i of the signal u[n]. In one embodiment, the N mask values G 1 (i)˜G N (i) are N band gains (being bounded between Th 1  and Th 2 ; Th 1 &lt;Th 2 ) corresponding to the N frequency bands in the current spectral representations F 1 ( i )˜FQ(i). Thus, if any band gain value G k (i) gets close to Th 1 , it indicates the signal on the corresponding frequency band k is noise-dominant; if any band gain value G k (i) gets close to Th 2 , it indicates the signal on the corresponding frequency band k is speech-dominant. When the end-to-end neural network  130  is trained, the higher the SNR value in a frequency band k is, the higher the band gain value G k (i) in the frequency-domain compensation mask stream becomes. 
     In brief, the low latency of the end-to-end neural network  130  between the time-domain input signals s 1 [n]˜s Q [n] and the responsive time-domain output signal u[n] fully satisfies the ANC requirements (i.e., less than 50 μs). In addition, the end-to-end neural network  130  manipulates the input current spectral representations F 1 ( i )˜FQ(i) in frequency domain to achieve the goals of noise suppression, AFC and sound amplification, thus greatly improving the audio quality. Thus, the framework of the end-to-end neural network  130  integrates and exploits cross domain audio features by leveraging audio signals in both time domain and frequency domain to improve hearing aid performance. 
       FIG.  4    is a schematic diagram of the post-processing unit  150  according to an embodiment of the invention. Referring to  FIG.  4   , the post-processing unit  150  includes a serial-to-parallel converter (SPC)  151 , a compensation unit  152 , an inverse STFT block  154 , an adder  155  and a multiplier  156 . The compensation unit  152  includes a suppressor  41  and an alpha blender  42 . The SPC  151  is configured to convert the complex-valued data stream (G 1 (i)˜G N (i)) into N parallel complex-valued data and simultaneously send the N parallel complex-valued data to the suppressor  41 . The suppressor  41  includes N multipliers (not shown) that respectively multiply the N mask values (G 1 (i)˜G N (i)) by their respective complex-valued data (F 1,1 (i)˜F N,1 (i)) of the main spectral representation F 1 ( i ) to obtain N product values (V 1 (i)˜V N (i)), i.e., V k (i)=G k (i)×F k,1 (i). The alpha blender  42  includes N blending units  42   k  that operate in parallel, where 1&lt;=k&lt;=N.  FIG.  5    is a schematic diagram of a blending unit  42   k  according to an embodiment of the invention. Each blending unit  42   k  includes two multipliers  501 ˜ 502  and one adder  503 . Each blending unit  42   k  is configured to compute complex-valued data: Z k (i)=F k,1 (i)×α k +V k (i)×(1−α k ), where α k  denotes a blending factor of kth frequency band for adjusting the level (or strength) of noise suppression and acoustic feedback cancellation. Then, the inverse STFT block  154  transforms the complex-valued data (Z 1 (i)˜Z N (i)) in frequency domain into audio data of the current frame i of the audio signal z[n] in time domain. In addition, the multiplier  156  sequentially multiplies each sample in the current frame i of the digital audio signal u[n] by w to obtain audio data in the current frame i of an audio signal p[n], where w denotes a weight for adjusting the ANC level. Afterward, the adder  155  sequentially adds two corresponding samples in the current frames i of the two signals z[n] and p[n] to produce audio data in the current frame i of a sum signal y[n]. Next, the DAC  161  converts the digital audio signal y[n] into an analog audio signal Y and then the amplifier  162  amplifies the analog audio signal Y to produce an amplified signal SA. Finally, the loudspeaker  163  converts the amplified signal SA into a sound pressure signal in an ear canal of the user. 
     The above embodiments and functional operations can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. The operations and logic flows described in  FIGS.  1 - 5    can be performed by one or more programmable computers executing one or more computer programs to perform their functions, or by special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). Computers suitable for the execution of the one or more computer programs include, by way of example, can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. 
     While certain exemplary embodiments have been described and shown in the accompanying drawings, it is to be understood that such embodiments are merely illustrative of and not restrictive on the broad invention, and that this invention should not be limited to the specific construction and arrangement shown and described, since various other modifications may occur to those ordinarily skilled in the art.