Patent Abstract:
A method for operating a hearing system including a hearing device, a camera and an auxiliary device, the method including the steps of providing an input signal to the hearing device, capturing an image or a sequence of images of at least sections of a surrounding of a user wearing the hearing device, processing the image or the sequence of images in the auxiliary device) for obtaining consolidated data of a sound source being important for the user, transmitting the consolidated data to the hearing device, generating an output signal in the hearing device by processing the audio signal and by taking into account the consolidated data, and feeding the output signal to an output transducer of the hearing device. Thereby, a hearing system having improved capabilities is obtained.

Full Description:
TECHNICAL FIELD OF THE INVENTION 
       [0001]    The present invention is related to a method for operating a hearing system with the aid of a camera as well as to a hearing system. 
       DESCRIPTION OF THE RELATED ART 
       [0002]    One of the most important goals of a hearing system is to enhance the intelligibility of speech also in adverse listening conditions. A beam former is one of the functionalities, which improves the intelligibility of speech, when the speaker is in front of the hearing device user. But in many situations this is not the case. 
         [0003]    It is difficult to analyse acoustically to which person the hearing device user wants to listen to, especially when several individuals are talking. There is no hearing system yet that enables the hearing device user to select the individual the user wants to listen to and that tracks the location of this individual such that the hearing device optimizes the intelligibility exactly to this location. 
         [0004]    There are solutions that propose to use further sensors, especially image sensors, to improve the beam former of a hearing device. For example, the teaching U.S. Pat. No. 6,707,921 B1 discloses an image based solution to determine when a speaker is speaking. Furthermore, DE 10 147 812 B4 discloses a hearing system comprising a camera. The known hearing system may use image processing to determine a number of speakers, perform lip-reading and to control a beam former. 
       SUMMARY OF THE INVENTION 
       [0005]    It is an object of the present invention to provide a method for operating a hearing system as well as a hearing system that are significantly improved with regard to the known solutions. 
         [0006]    It is pointed out that the term “hearing device” covers a hearing aid—such as a BTE-(Behind-The-Ear), an ITE-(In-The-Ear), a CIC-(Completely-In-Channel) hearing device or the like—and also an implantable device that is used to improve the hearing of a hearing impaired person. 
         [0007]    First, the present invention is directed to a method for operating a hearing system comprising a hearing device, a camera and an auxiliary device. The inventive method comprises the steps of:
       providing an input signal for said hearing device,   capturing an image or a sequence of images of at least sections of a surrounding of a user wearing said hearing device,   processing said image or said sequence of images in said auxiliary device for obtaining consolidated data of a sound source being important for said user,   transmitting said consolidated data to the hearing device,   generating an output signal in said hearing device by processing the input signal and by taking into account said consolidated data, and   feeding said output signal to an output transducer of said hearing device.       
 
         [0014]    The step of “providing an input signal for said hearing device” shall be understood as capturing an acoustic signal impinging on the input transducer of the hearing device. The input transducer comprises two or more microphones, for example. Two or more microphones are required if the hearing device shall have beam forming capabilities. 
         [0015]    The present invention results in an improved intelligibility for the hearing system user while energy resources in the hearing device are maintained at the same time. 
         [0016]    In an embodiment of the method according to the present invention, said processing comprises locating said sound source being of importance to said user and said consolidated data of a sound source comprises a directional angle being defined between a sagittal plane of said user and said sound source seen from said user. 
         [0017]    The consolidated data is the result of the processing of the images or sequence of images and may very well be a single parameter as the angle defined between the sagittal plane of the user and the sound source seen from the user. 
         [0018]    Further embodiments of the method according to the present invention further comprise the steps of:
       selecting said sound source being important to said user of said hearing device out of a plurality of sound sources, and   tagging said sound source by activating an input on the auxiliary device.       
 
         [0021]    Further embodiments of the method according to the present invention further comprise the step of tracking said sound source. 
         [0022]    In further embodiments of the method according to the present invention, the auxiliary device is one of the following:
       a smartphone;   a remote control.       
 
         [0025]    In further embodiments of the method according to the present invention, said camera is positioned at the head of the user. 
         [0026]    Further embodiments of the method according to the present invention further comprise the step of augmenting said consolidated data by information of a sensor unit, such as a compass or an additional microphone signal. 
         [0027]    Further embodiments of the method according to the present invention further comprise the steps of:
       detecting a present position of said user relative to the camera, and   calculating a direction in which said sound source being important for said user taking into account the present position of said user.       
 
         [0030]    Further embodiments of the method according to the present invention further comprise the steps of:
       detecting a favorite speaker as said sound source by face recognition and comparison to a corresponding data base, and   tracking said favorite speaker after a tracking command is received by said auxiliary device.       
 
         [0033]    In further embodiments of the method according to the present invention, said input signal to the hearing device originate from at least one input transducer or from a streaming unit. 
         [0034]    Furthermore, the present invention is directed to a hearing system comprising:
       a hearing device comprising an input transducer, an output transducer and a processing unit being operatively connected to the input transducer and the output transducer,   at least one camera for capturing an image or a sequence of images of at least a section of a surrounding of a user wearing said hearing device,   an auxiliary device receiving and processing said image or said sequence of images for obtaining consolidated data of a sound source being important for said user, and   a transmission channel between said auxiliary device and said hearing device for transmitting said consolidated data to said hearing device,       
 
         [0039]    wherein said processing unit is adapted to take into account said consolidated data while generating an output signal for the output transducer. 
         [0040]    Complexity of the image analysis requires high computing resources, which may not be available in a BTE-(Behind-The-Ear) or ITE-(In-The-Ear) hearing devices. Otherwise, image analysis would lead to an excessive use of battery power. Continuous streaming of image data would also require too much power. 
         [0041]    In an embodiment of the hearing system according to the present invention, a low power technology is used for the transmission channel, such as Bluetooth standard or inductive coupling. 
         [0042]    In further embodiments of the hearing system according to the present invention, the transmission channel is adapted to continuously transmit consolidated data from the auxiliary device to the hearing device. 
         [0043]    In further embodiments of the hearing system according to the present invention, the auxiliary device is a smartphone or a remote control. 
         [0044]    It is expressly pointed out that also all combinations of the above-mentioned embodiments are possible and herewith disclosed. Only those embodiments or combinations of embodiments are excluded that would result in a contradiction. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0045]    The present invention is further described by referring to drawings showing exemplified embodiments of the present invention. 
           [0046]      FIG. 1  schematically shows a known BTE-(Behind-The-Ear) hearing device with its main components, 
           [0047]      FIG. 2  schematically shows a side view of a head of a user wearing the hearing device of  FIG. 1  and glasses with a camera, 
           [0048]      FIG. 3 a    and  FIG. 3 b    show two situations a hearing device user may encounter, 
           [0049]      FIG. 4  schematically shows the hearing system with an auxiliary device, and 
           [0050]      FIG. 5 a    and  FIG. 5 b    schematically shows a situation with the user of the hearing device, with the auxiliary device and a person being of interest for the hearing device user. 
       
    
    
     DETAILED DESCRIPTION OF THE INVENTION 
       [0051]      FIG. 1  schematically shows a known BTE-(Behind-The-Ear) hearing device  1  with its main components comprising a battery  2 , a processing unit  3 , a wireless interface unit  4 , a first input transducer  5 , a second input transducer  6  and a receiver unit  7 , to which a tube is connected (not shown in  FIG. 1 ) to conduct sound generated by the receiver unit  7  to an ear of a hearing device user via an ear tip positioned in the ear canal, for example. The battery  2  is providing energy to the wireless interface unit  4  as well as to the processing unit  3 , in which input signals of the first and second input transducers  5  and  6  are processed and in which an output signal is generated for the receiver unit  7 . 
         [0052]      FIG. 2  shows a side view of a head  10  of a hearing device user wearing the hearing device  1  of  FIG. 1 . Besides the internal components of the hearing device  1  a sound tube  8  is also shown that is connected to an ear piece (not shown in  FIG. 2 ) arranged in the ear of the hearing device user. As can be seen from  FIG. 2 , the hearing device user is wearing glasses  13  to which a front camera  11  and a side camera  12  are attached. Furthermore, a sensor unit  14  is also attached to the glasses  13 , the sensor unit  14  being a microphone or a compass, for example. 
         [0053]    The cameras  11 ,  12  and the sensor unit  14  generate output signals that must be processed, for example by applying a tracking algorithm for tracking a person being of interest for the hearing device user. As a matter of fact, such a processing is rather intense and asks for a rather powerful processor. Because the hearing device  1  typically has limited processing power and limited battery capacity, the processing of the output signals of cameras  11 ,  12  and the sensor unit  14  are processed in an auxiliary device  21  ( FIGS. 4, 5   a  and  5   b ). Thereto, the cameras  11 ,  12  and the sensor unit  14  are connected to said auxiliary device  21 , which can be a smartphone or a remote control having ample processing power available. Therefore, the cameras  11  and  12  transmit its raw data to the auxiliary device  21 , in which the raw data is processed according to the envisaged task. In fact, by applying the envisaged tasks, e.g. the tracking algorithm to track a person being important to the hearing device user, consolidated data is generated from the raw data by the auxiliary device  21 . These consolidated data are taken into account in the hearing device  1 , i.e. in the processing unit  3 , while generating the output signal of the hearing device  1  by processing the audio signal. 
         [0054]    The consolidated data can be, for example, an angle or a direction towards a person being important for the hearing device user. The angle is updated on a regular basis in order that said person can be tracked without delay. 
         [0055]    The tracking algorithms run on a powerful processor of the auxiliary device  21 . The result (i.e. an angle) is transmitted to the hearing device  1  via a wireless connection, for example. 
         [0056]    Since the consolidated data is small in comparison to the raw data (e.g. an image or a sequence of images), taking into account the consolidated data in the processing unit  3  of the hearing device  1  only results in a low battery load. 
         [0057]    The sensor unit  14  attached to the glasses  13  (as shown in  FIG. 2 ) or attached to the auxiliary device  21  is used, in a further embodiment of the present invention, to enhance robustness of the algorithms being implemented. If a tracking algorithm is implemented, the auxiliary device  21  may very well be used to initialize the tracking of a person being important to the hearing device user. This can be achieved by pressing a bottom on the auxiliary device  21  while pointing to said person at the same time. 
         [0058]    It has already been described that some embodiments of the present invention comprise more than one camera  11 ,  12 . 
         [0059]    While the first or main camera  11  is pointing to the front, further cameras may be attached at the side of the head of the hearing device user. Preferably the cameras  11 ,  12  are attached on the glasses  13  (as shown in  FIG. 2 ), or on the hearing device  1  (not shown). However, it is important that the cameras  11 ,  12  move with the head of the hearing device user to detect the angle between a sagittal plane of said user and the person (speaker) being important for the hearing device user. 
         [0060]    In a further embodiment, at least one of the cameras  11 ,  12  is a TOF-(Time-of-Flight) camera, such as used in “Microsoft Kinect” or in a smartphone. It is noted that the smartphone may also comprise two cameras that might be used according to the present invention. 
         [0061]    In case that the hearing device  1  is not mechanically coupled to the head of the hearing device user, it is necessary to detect a head movement of the hearing device user in order to track said person being important for the hearing device user (e.g. a speaker). Such an information could be derived from a further sensor in the hearing device  1 , such further sensor being a camera looking at the head of the hearing device user, for example. 
         [0062]    A synchronization of the acoustic detection of speech pauses of the person being important for the hearing device user (speaker or target person) with the visual detection of the conversation activity of the target person (e.g. by detecting whether said person is talking or is silent) by analysing an image taken by the camera, makes the tracking algorithm more robust and may help for a fast adaptation to conversational turns (e.g. changing the target person) by visual detecting speech pauses of the target person in continuous manner. In a further embodiment of the present invention, the microphone of the auxiliary device (e.g. smartphone) may be used for such a detection. 
         [0063]      FIGS. 3 a  and 3 b    schematically show top views of the hearing device user  10  among a number of people  15  to  19  surrounding said user  10 . The hearing device user  10  wears the hearing device  1  illustrated in and described in connection with  FIG. 2 . 
         [0064]    In the situation depicted in  FIGS. 3 a  and 3 b   , it is assumed that the hearing device user  10  is interested in listening to what the speaker A (person  15 ) says. In order that the hearing system according to the present invention is able to track speaker A, the hearing device user  10  must enable tracking of speaker A. Thereto, an initial sign indicating speaker A must be given to the hearing system in order that the hearing system may track speaker A. 
         [0065]    In  FIG. 3 a   , a situation is depicted in which the hearing device user  10  initializes a tracking of speaker A. Such an initialization can be done, for example, by pointing with the camera  11  to the speaker A, i.e. the hearing device user  10  looks to the speaker A. In case the camera  11  is not fixed to the head of the hearing device user  10 , an orientation of the camera relative to the head of the user  10  needs to be known. For this purpose a compass comprised in the hearing device  1  may be used. In further embodiments, similar orientation signals, e.g. a compass signal form the auxiliary device  21  or of a further camera can be used. 
         [0066]    Once the hearing device user  10  decides to start tracking speaker A, one of the following initialization procedures may be performed:
       A button on the auxiliary device  21  is pressed.   Instead of pressing a button, any other input method may be used, such as, for example, a voice command.   If the hearing system incorporates acceleration sensors, a specific head movement by the hearing device user  10 —like a head tilt with a specific angle—can be detected.   If the hearing system incorporates an eye-tracking system, the hearing device user  10  may look into different directions with a specific directional pattern that is detected to start tracking speaker A. Alternatively, twinkling with the eyes may be also detected to start tracking speaker A.       
 
         [0071]    In a further embodiment, the hearing system according to the present invention comprises face recognition capabilities and comprises a database of faces of known speakers (favorites). In such an embodiment, the initialization of tracking a speaker may be done by a voice command, such as “listen to Lisa”. The system would then try to find the face of Lisa. Once the face of Lisa has been identified, Lisa is defined as the sound source being most important to the hearing device user  10  and therefore is tracked. A person that is selected to be tracked but is not in the database could be added to the database, if a proper snapshot of the face of this person can be taken during a conversation. 
         [0072]    Disabling tracking may work similar as enabling tracking. In addition, situation specific actions can disable tracking a speaker, like:
       Speaker A stops talking for a certain while.   Speaker A or the hearing device user leaves the scenery (e.g. speaker A appears smaller in the captured sequence of images or disappears from the images).   No signal coming from the speaker A for a predefined time.       
 
         [0076]    According to the present invention, a number of possible criterions are being implemented for tracking a person being important for the hearing device user. One criterion might be typical characteristics of a face of a person to be tracked, whereas these face characteristics are memorized and used with image processing methods for face recognition. In general, recognizing the presence of a face and tracking the movement of this face, whereas such an algorithm may be enhanced by
       detecting the acoustic direction of any incoming signal/voice, assuming that it is the voice of the tracked person;   the synchronization of the acoustic onsets and offsets of any signal/speech signal (originating from the same direction) with the movement of the mouth or head captured with the camera;   capturing acoustic properties of the voice of the speaker during the enabling process (according to speaker recognition algorithms) and comparing this signal with the video analysis (e.g. moving source and/or face recognition and/or movement of the mouth) (synchronization of speech pauses or/and directionality of the source);   in case the hearing device user wears any movement sensors on the head, any head movements can be incorporated in the calculation of the angle between the speaker and the hearing device user. In case the angle between the body of the speaker and the body of the hearing device user are the same, the tracking of the head movement of the hearing device user is sufficient to calculate the wanted angle.   matching characteristics of the face with characteristics of the voice (e.g. low pitch of the voice and male characteristics in the face).   matching size of the face and acoustic distance of the speech (if possible, signal processing optimizes for distant speech as well, e.g. mixing between streaming-microphone mode toward more percentage of the streaming signal).       
 
         [0083]      FIG. 4  shows the hearing device user  10  wearing a hearing device  1  and glasses  13  as depicted in  FIG. 2  as well as a person  15  being important for the hearing device user  10 . In addition, the auxiliary device  21  is also shown in  FIG. 4  being linked to the cameras  11  and  12 , the sensor unit  14  and the hearing device  1  via a wireless link being proprietary or a standardized link, e.g. Bluetooth or the like. As has been already mentioned, processing of image or sequences of images are processed in the auxiliary device  21  by generating consolidated data (e.g. an angle α being defined between a sagittal plane of the hearing device user  10  and the person  15  being important for the hearing device user  10 . Only the consolidated data are then transmitted to the hearing device  1 , where it is taken into account while processing the signal or signals of the input transducers  5 ,  6 . 
         [0084]    It is pointed out that a sound source being important for the hearing device user is not always a speaker as it is described in connection with  FIGS. 3 a  and 3 b   . A sound source being important for the hearing device user  10  may very well also be an audio source from a Radio or TV set, for example. 
         [0085]      FIGS. 5 a  and 5 b    show two situations with an alternative sound source  22  being of interest to the hearing device user  10 , the situation depicted in  FIG. 5 b    being a mixture of the alternative sound source  22  and a speaker A (reference sign  24 ). 
         [0086]    In  FIG. 5 a   , the hearing device user  10  wearing glasses  13 , as explained in connection with  FIG. 2 , is positioned before a TV set  22  comprising a wireless interface unit  23  for streaming the audio signal of the TV set  22  via the auxiliary device  21  to the hearing device. As a matter of fact, an audio signal that is emitted by the loudspeaker of the TV set  23  and picked up by an input transducer  5 ,  6  ( FIG. 1 ) of the hearing device  1  is replaced by the audio signal of the TV set  23  being streamed via the wireless interface unit  23 . Therewith, the quality of the signal received for processing in the hearing device  1  is higher than for a signal picked up by the input transducers  5 ,  6  as these input transducers  5 ,  6  also pick up noise and surrounding sounds. 
         [0087]    In other words, if the loudspeakers of the TV set  23  comprises streaming means, the signals picked up by the input transducers  5 ,  6  can automatically be substituted by the streamed audio signal. It is pointed out that any device may be upgraded to include streaming means. In particular, a remote microphone may comprise streaming means in order to transmit the picked-up sound directly to its destination or processing unit. 
         [0088]    The presence of streaming means may be recognized by using the camera of the auxiliary device  21 . The camera detects the TV set  23  or may recognize that a speaker is using a remote microphone comprising streaming means. At the same time, the auxiliary device  21  may scan available audio-streams. In case a hearing device user wants to track such a sound source having streaming means, the user may connects his auxiliary device  21  to an audio stream coming from the corresponding sound source. The auxiliary device  21  may than act as a relay station between the streaming device and the hearing device. Alternatively, a direct streaming from the streaming device to the hearing device is possible.

Technology Classification (CPC): 7