Abstract:
The present subject application relates to hearing assistance systems and in particular to a method and apparatus for detecting user activities from within a hearing assistance system using micro electro-mechanical structure sensors. Such benefits include the reduction of the ampclusion effect and other excessive sound pressure buildup in the residual air volume of the ear canal for a person wearing a hearing assistance device with an earmold.

Description:
RELATED APPLICATIONS 
       [0001]    This application claims priority under 35 U.S.C 119(e) of U.S. Provisional Patent Application Ser. No. 60/973,399 filed on Sep. 18, 2007 which is incorporated by reference herein in its entirety. 
     
    
     FIELD 
       [0002]    This application relates generally to hearing assistance systems and in particular to a method and apparatus for detecting user activities from within a hearing aid using sensors employing micro electro-mechanical structures (MEMS). 
       BACKGROUND 
       [0003]    For hearing aid users, certain physical activities induce low-frequency vibrations that excite the hearing aid microphone in such a way that the low frequencies are amplified by the signal processing circuitry thereby causing excessive buildup of unnatural sound pressure within the residual ear-canal air volume. The hearing aid industry has adapted the term “ampclusion” for these phenomena as noted in “Ampclusion Management 101: Understanding Variables”  The Hearing Review , pp. 22-32, August (2002) and “Ampclusion Management 102: A 5-step Protocol” The Hearing Review, pp. 34-43, September (2002), both authored by F. Kuk and C. Ludvigsen. In general, ampclusion can be caused by such activities as chewing or heavy footfall motion during walking or running. These activities induce structural vibrations within the user&#39;s body that are strong enough to be sensed by a MEMS accelerometer that is properly positioned within the earmold of a hearing assistance device. Another user activity that can excite such a MEMS accelerometer is simple speech, particularly the vowel sounds of [i] as in piece and [u] is as in rule and annunciated according to the International Phonetic Alphabet. Yet another activity that can be sensed by a MEMS accelerometer is automobile motion or acceleration, which is commonly perceived as excessive rumble by passengers wearing hearing aids. Automobile motion is unique from the previously-mentioned activities in that its effect, i.e., the rumble, is generally produced by acoustical energy propagating from the engine of the automobile to the microphone of the hearing aid. The output signal(s) of a MEMS accelerometer can be processed such that the device can detect automobile motion or acceleration relative to gravity. One additional user activity, not related to ampclusion, that can be detected by a MEMS accelerometer is head tilt. Finally, it should be noted that a MEMS gyrator or a MEMS microphone can be used to detect all of the above-referenced user activities instead of a MEMS accelerometer. It is understood that a MEMS acoustical microphone may be modified to function as a mechanical or vibration sensor. For example, in one embodiment the acoustical inlet of the MEMS microphone is sealed. Other techniques modifying an acoustical microphone may be employed without departing from the scope of the present subject matter. In addition to the translational acceleration estimates provided by a MEMS accelerometer, a MEMS gyrator provides three additional rotational acceleration estimates. 
         [0004]    Thus, there is a need in the art for a detection scheme that can reliably identify user activities and trigger the signal processing algorithms and circuitry to process, filter, and equalize their signal so as to mitigate the undesired effects of ampclusion and other user activities. In all of the activities described in the previous paragraph, the MEMS device acts as a detection trigger to alert the hearing aid&#39;s signal processing algorithm to specific user activities thereby allowing the algorithm to filter and equalize its frequency response according to each activity. Such a detection scheme should be computationally efficient, consume low power, require small physical space, and be readily reproducible for cost-effective production assembly. 
       SUMMARY 
       [0005]    The above-mentioned problems and others not expressly discussed herein are addressed by the present subject matter and will be understood by reading and studying this specification. The present system provides methods and apparatus to detect various motion events that effect audio signal processing and apply appropriate filters to compensate audio processing related to the detected motion events. In one embodiment an apparatus is provided with a micro electro-mechanical structure (MEMS) to sense motion and a processor to compare the sensed motion to signature motion events and provide further processing to adjust filters to compensate for audio effects resulting from the detected motion events. 
         [0006]    This Summary is an overview of some of the teachings of the present application and not intended to be an exclusive or exhaustive treatment of the present subject matter. Further details about the present subject matter are found in the detailed description and appended claims. The scope of the present invention is defined by the appended claims and their legal equivalents. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0007]    Various embodiments are illustrated by way of example in the figures of the accompanying drawings. Such embodiments are demonstrative and not intended to be exhaustive or exclusive embodiments of the present subject matter. 
           [0008]      FIG. 1  shows a side cross-sectional view of an in-the-ear hearing assistance device according to one embodiment of the present subject matter. 
           [0009]      FIG. 1A  illustrates a MEMS sensor mounted halfway into the shell of a hearing assistance device according to one embodiment of the present subject matter. 
           [0010]      FIG. 1B  illustrates a MEMS sensor mounted flush with the shell of a hearing assistance device according to one embodiment of the present subject matter. 
           [0011]      FIG. 2  illustrates a way to mount a MEMS accelerometer to the interior end of the device using a BTE (behind-the-ear) hearing assistance device according to one embodiment of the present subject matter. 
           [0012]      FIG. 3  illustrates a BTE providing an electronic signal to an earmold having a receiver according to one embodiment of the current subject matter. 
           [0013]      FIG. 4 . illustrates a wireless earmold embodiment of the current subject matter. 
           [0014]      FIG. 5  illustrates typical timing relationships for detection of audio related motion events according to one embodiment of the current subject matter. 
       
    
    
     DETAILED DESCRIPTION 
       [0015]    The following detailed description of the present invention refers to subject matter in the accompanying drawings which show, by way of illustration, specific aspects and embodiments in which the present subject matter may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the present subject matter. References to “an”, “one”, or “various” embodiments in this disclosure are not necessarily to the same embodiment, and such references contemplate more than one embodiment. The following detailed description is demonstrative and therefore not exhaustive, and the scope of the present subject matter is defined by the appended claims and their legal equivalents. 
         [0016]    There are many benefits in using the output(s) of a properly-positioned MEMS accelerometer as the detection sensor for user activities. Consider, for example, that the sensor output is not degraded by acoustically-induced ambient noise; the user activity is detected via a structural path within the user&#39;s body. Detection and identification of a specific event typically occurs within approximately 2 msec from the beginning of the event. For speech detection, a quick 2 msec detection is particularly advantageous. If, for example, a hearing aid microphone is used as the speech detection sensor, a (≈0.8 msec) time delay would exist due to acoustical propagation from the user&#39;s vocal chords to the user&#39;s hearing aid microphone thereby intrinsically slowing any speech detection sensing. This 0.8 msec latency is effectively eliminated by the structural detection of a MEMS accelerometer sensor in an earmold. Considering that a DSP circuit delay for a typical hearing aid is ≈5 msec, and that a MEMS sensor positively detects speech within 2 msec from the beginning of the event, the algorithm is allowed ≈3 msec to implement an appropriate filter for the desired frequency response in the ear canal. These filters can be, but are not limited to, low order high-pass filters to mitigate the user&#39;s perception of rumble and boominess. 
         [0017]    The most general detection of a user&#39;s activities can be accomplished by digitizing and comparing the amplitude of the output signal(s) of the MEMS accelerometer to some predetermined threshold. If the threshold is exceeded, the user is engaged in some activity causing higher acceleration as compared to a quiescent state. Using this approach, however, the sensor cannot distinguish between a targeted, desired activity and any other general motion, thereby producing “false triggers” for the desired activity. A more useful approach is to compare the digitized signal(s) to stored signature(s) that characterize each of the user events, and to compute a (squared) correlation coefficient between the real-time signal and the stored signals. When the coefficient exceeds a predetermined threshold for the correlation coefficient, the hearing aid filtering algorithms are alerted to a specific user activity, and the appropriate equalization of the frequency response is implemented. The squared correlation coefficient γ 2  is defined as: 
         [0000]    
       
         
           
             
               
                 γ 
                 2 
               
                
               
                 ( 
                 x 
                 ) 
               
             
             = 
             
               
                 
                   
                     ∑ 
                     s 
                   
                    
                   
                       
                   
                    
                   
                     [ 
                     
                       
                         
                           f 
                           1 
                         
                          
                         
                           ( 
                           s 
                           ) 
                         
                       
                        
                       
                         
                           f 
                           2 
                         
                          
                         
                           ( 
                           s 
                           ) 
                         
                       
                     
                     ] 
                   
                 
                 - 
                 
                   n 
                    
                   
                       
                   
                    
                   
                     
                       
                         
                           f 
                           1 
                         
                          
                         
                           ( 
                           s 
                           ) 
                         
                       
                        
                       
                         
                           f 
                           2 
                         
                          
                         
                           ( 
                           s 
                           ) 
                         
                       
                     
                     _ 
                   
                 
               
               
                 
                   
                     ∑ 
                     s 
                   
                    
                   
                       
                   
                    
                   
                     
                       f 
                       1 
                       2 
                     
                      
                     
                       ( 
                       s 
                       ) 
                     
                   
                 
                 - 
                 
                   n 
                    
                   
                       
                   
                    
                   
                     
                       
                         f 
                         1 
                         2 
                       
                        
                       
                         ( 
                         s 
                         ) 
                       
                     
                     _ 
                   
                    
                   
                     
                       ∑ 
                       s 
                     
                      
                     
                         
                     
                      
                     
                       
                         f 
                         2 
                         2 
                       
                        
                       
                         ( 
                         s 
                         ) 
                       
                     
                   
                 
                 - 
                 
                   n 
                    
                   
                       
                   
                    
                   
                     
                       
                         f 
                         2 
                         2 
                       
                        
                       
                         ( 
                         s 
                         ) 
                       
                     
                     _ 
                   
                 
               
             
           
         
       
     
         [0000]    where x is the sample index for the incoming data, f 1  is the last n samples of incoming data, f 2  is the n-length signature to be recognized, and s is indexed from 1 to n. Vector arguments with overstrikes are taken as the mean value of the array, i.e., 
         [0000]    
       
         
           
             
               
                 
                   f 
                   1 
                 
                  
                 
                   ( 
                   s 
                   ) 
                 
               
               _ 
             
             = 
             
               
                 
                   ∑ 
                   s 
                 
                  
                 
                     
                 
                  
                 
                   
                     f 
                     1 
                   
                    
                   
                     ( 
                     s 
                     ) 
                   
                 
               
               n 
             
           
         
       
     
         [0000]    There are many benefits in using the squared correlation coefficient as the detection threshold for user activities. Empirical data indicate that merely 2 msec of digitized information (an n value of 24 samples at a sampling rate of 12.8 kHz) are needed to sufficiently capture the types of user activities described previously in this discussion. Thus, five signatures having 24 samples at 8 bits per sample require merely 960 bits of storage memory within the hearing aid. It should be noted that the cross correlation computation is immune to amplitude disparity between the stored signatures f 1  and the signature to be identified f 2 . In addition, it is computed completely in the time domain using basic {+ − × ÷} operators, without the need for computationally-expensive butterfly networks of a DFT. Empirical data also indicate that the detection threshold is the same for all activities, thereby reducing detection complexity. 
         [0018]    Although a single MEMS sensor is used, the sensing of various user activities is typically exclusive, and separate signal processing schemes can be implemented to correct the frequency response of each activity. The types of user activities that can be characterized include speech, chewing, footfall, head tilt, and automobile de/acceleration. Speech vowels of [i] as in piece and [u] is as in rule typically trigger a distinctive sinusoidal acceleration at their fundamental formant region of a (few) hundred hertz, depending on gender and individual physiology. Chewing typically triggers a very low frequency (&lt;10 Hz) acceleration with a unique time signature. Although chewing of crunchy objects can induce some higher frequency content that is superimposed on top of the low frequency information, empirical data have indicated that it has negligible effect on detection precision. Footfall too is characterized by low frequency content, but with a time signature distinctly different from chewing. Head tilt can be detected by low-pass filtering and differentiating the output signals from a multi-axis MEMS accelerometer. 
         [0019]    The MEMS accelerometer can be designed to detect any or all of the three translational acceleration components of a rectangular coordinate system. Typically, a dedicated micro-sensor is used in a 3-axis MEMS accelerometer to detect both the x and y components of acceleration, and a different micro-sensor is used to detect the z component. In our application, a 3-axis accelerometer in the earmold could be orientated such that the relative z component is approximately parallel with the relatively-central axis of the ear canal, and the x and y components define a plane that is relatively perpendicular to the surface of the earmold in the immediate vicinity of the ear canal tip. Alternatively, the MEMS accelerometer could be orientated such that the x and y components define any relative plane that is tangent to the surface of the earmold in the immediate vicinity of side of the ear canal, and the z component points perpendicularly inward towards the interior of the earmold. Although specific orientations have been described herein, it will be appreciated by those of ordinary skill in the art that other orientations are possible without departing from the scope of the present subject matter. In each of these orientations, a calibration procedure can be performed in-situ during the hearing aid fitting process. For example, the user could be instructed during the fitting/calibration process to do the following: 1) chew a nut, 2) chew a soft sandwich, 3) speak the phrase: “teeny weeny blue zucchini”, 4) walk a known distance briskly. These events are digitized and stored for analysis, either on board the hearing aid itself or on the fitting computer following some data transfer process. An algorithm clips and conditions the important events and these clipped events are stored in the hearing aid as “target” events. The MEMS detection algorithm is engaged and the (4) activities described above are repeated by the user. Detection thresholds for the squared correlation coefficient and ampclusion filtering characteristics are adjusted until positive identification and perceived sound quality is acceptable to the user. The adjusted thresholds for each individual user will depend on the orientation of the MEMS accelerometer, the number of active axes in the MEMS accelerometer, and the relative strength of signal to noise. For the walking task, the accelerometer can be calibrated as a pedometer, and the hearing aid can be used to inform the user of accomplished walking distance status. In addition, head tilt could be calibrated by asking the user to do the following from a standing or sitting position looking straight ahead: 1) rotate the head slowly to the left or right, and 2) rotate the head such that the user&#39;s eyes are pointing directly upwards. These events are digitized as done previously, and the accelerometer output is filtered, conditioned, and differentiated appropriately to give an estimate of head tilt in units of mV output per degree of head tilt, or some equivalent. This information could be used to adjust head related transfer functions, or as an alert to a notify that the user has fallen or is falling asleep. 
         [0020]    It is understood that a MEMS accelerometer or gyrator can be employed in either a custom earmold in various embodiments, or a standard earmold in various embodiments. Although specific embodiments have been illustrated and described herein, it will be appreciated by those of ordinary skill in the art that other embodiments are possible without departing from the scope of the present subject matter. 
         [0021]      FIG. 1  shows a side cross-sectional view of an in-the-ear (ITE) hearing assistance device according to one embodiment of the present subject matter. It is understood that  FIG. 1  is intended to demonstrate one application of the present subject matter and that other applications are provided.  FIG. 1  relates to the use of a MEMS accelerometer mounted rigidly to the inside shell of an ITE (in-the-ear) hearing assistance device. However, it is understood that the MEMS accelerometer design of the present subject matter may be used in other devices and applications. One example is the earmold of a BTE (behind-the-ear) hearing assistance device, as demonstrated by  FIG. 2 . The present MEMS accelerometer design may be employed by other hearing assistance devices without departing from the scope of the present subject matter. 
         [0022]    The ITE device  100  of the embodiment illustrated in  FIG. 1  includes a faceplate  110  and an earmold shell  120  which is positioned snugly against the skin  125  of a user&#39;s ear canal  127 . A MEMS sensor  130  is rigidly mounted to the inside of an earmold shell  120  and connected to the hybrid integrated electronics  140  with electrical wires or a flexible circuit  150 . The electronics  140  include a receiver (loudspeaker)  142  and microphone  144 . Other placements and mountings for MEMS accelerometer  130  are possible without departing from the scope of the present subject matter. In various embodiments, the MEMS sensor  130  is partially embedded in the plastic of earmold shell  120  as shown in  FIG. 1A , or fully embedded in the plastic so that is it flush with the exterior of earmold shell  120  as shown in  FIG. 1B . With this approach, structural waves are detected by sensor  120  via mechanical coupling to the skin  125  of a user&#39;s ear canal  127 . An analogous electrical signal is sent to electronics  140 , processed, and used in an algorithm to detect various user activities. It is understood that the electronics  140  may include known and novel signal processing electronics configurations and combinations for use in hearing assistance devices. Different electronics  140  may be employed without departing from the scope of the present subject matter. Such electronics may include, but are not limited to, combinations of components such as amplifiers, multi-band compressors, noise reduction, acoustic feedback reduction, telecoil, radio frequency communications, power, power conservation, memory, multiplexers, analog integrators, operational amplifiers, and various forms of digital and analog signal processing electronics. It is understood that the MEMS sensor  130  shown in  FIG. 1  is not necessarily drawn to scale. Furthermore, it is understood that the location of the MEMS accelerometer  130  may be varied to achieve desired effects and not depart from the scope of the present subject matter. Some variations include, but are not limited to, locations on faceplate  110 , sandwiched between receiver  142  and earmold shell  120  so as to create a rigid link between the receiver and the shell, or embedded within the hybrid integrated electronic circuit  140 . 
         [0023]    The embodiment of  FIG. 2  provides a way to mount a MEMS sensor  130  to the interior end of the device  200  using a BTE (behind-the-ear) hearing assistance device  210 . The BTE  210  delivers sound through sound tube  220  to the ear canal  127  at the interior end of earmold  240 . Sound tube  220  also contains an electrical conduit  222  for wired connectivity between the BTE and the MEMS sensor  130 . The remaining operation of the device is largely the same as set forth for  FIG. 1 , except that the BTE  210  includes the microphone and electronics, and earmold  240  contains the sound tube  220  with electrical conduit  222  and MEMS sensor  130 . The entire previous discussion pertaining to variations for the apparatus of  FIG. 1  applies herein for  FIG. 2 . Other embodiments are possible without departing from the scope of the present subject matter. 
         [0024]    The embodiment of  FIG. 3  uses a BTE  310  to provide an electronic signal to an earmold  340  having a receiver  142 . This variation permits a wired approach to providing the acoustic signals to the ear canal  142 . The electronic signal is delivered through electrical conduit  320  which splits at  322  to connect to MEMS sensor  130  and receiver  142 . 
         [0025]    The embodiment of  FIG. 4 , a wireless approach is employed, such that the earmold  440  includes a wireless apparatus for receiving sound from a BTE  410  or other signal source  420 . Such wireless communications are possible by fitting the earmold with transceiver electronics  430  and power supply. The electronics  430  could connect to a receiver loudspeaker  142 . In bidirectional applications, it may be advantageous to fit the earmold with a microphone to receive sound using the earmold. It is understood that many variations are possible without departing from the present subject matter. 
         [0026]    The middle panel of  FIG. 5  shows the instantaneous output voltage of a MEMS accelerometer for a typical user activity such as (1) background circuit noise, (2) crunchy chewing, (3) synthetically generated random noise, (4) a synthetically derived 1 kHz, amplitude-modulated sinusoid, and (5) soft chewing. The top panel of  FIG. 5  shows the instantaneous estimate of the squared correlation coefficient for each particular activity target according to one embodiment, with a horizontal dotted line depicting the detection threshold. The bottom panel shows a Boolean of the detection trigger according to one embodiment. All three panels are synchronized in time, and the vertical dotted lines depict the detection speed and precision of each chewing event. 
         [0027]    The present subject matter relates to a MEMS accelerometer, however, it is understood that other accelerometer designs and MEMS sensors may be substituted for the MEMS accelerometer.