Patent Description:
Hearing devices are generally small and complex devices. A typical hearing device comprises a processing unit, e.g. including one or more processors, a sound input module, e.g. a microphone, a sound output module, e.g. an loudspeaker, a memory communicatively coupled to the processing unit, a housing, and other electronical and mechanical components. Some example hearing devices are Behind-The-Ear (BTE), Receiver-In-Canal (RIC), In-The-Ear (ITE), Completely-In-Canal (CIC), and Invisible-In-The-Canal (IIC) devices. A user can prefer one of these hearing devices compared to another device based on hearing loss, aesthetic preferences, lifestyle needs, and budget.

Sometimes when hearing device users use their hearing device in everyday life, they may notice that the hearing device does not always support them well enough. Be it that the hearing device setting is not correctly set for the current acoustic environment, e.g. with respect to audio sources in the environment and/or a listening activity of the user, or an automatic classifier does not classify the situation correctly based on an acoustic input only.

The options, that a user conventionally has, are modification possibilities that are named for the acoustic intention, e.g. reduce background noise, low, high etc., or describe technical possibilities. Another option is to switch to an appropriate manual program if the automatic mode does not classify the acoustic environment correctly.

The translation of hearing intentions or hearing problems with audio sources in complex listening situations into adjustment actions of the modifier is challenging and sometimes impossible for the hearing device user. The range of modifiers that can be used to adjust the hearing device according to the listening intention might be overwhelming. The uncertainty, whether the right modifier is used to improve the performance is high and the risk is high that users adjust not the correct hearing device behavioural part.

Fast and efficient adjustments done by the user may be crucial and a match decision for a future engagement of the user to use the modification/optimization functionality offered may afford the user to react quickly to changes in the acoustic environment. <CIT> describes a system and method for differentially locating and modifying audio sources that includes receiving multiple audio inputs from a set of distinct locations; determining a multi-dimensional audio map from the audio inputs; acquiring a set of positional audio control inputs applied to the audio map, each audio control input comprising a location and audio processing property; and generating an audio output according to the audio control inputs and the audio inputs.

<CIT> describes a device, which includes a processor, at least one camera accessible to the processor, and memory accessible to the processor. The memory bears instructions executable by the processor to identify, at least in part based on input from the at least one camera, a source of sound. The instructions are also executable to, based at toast in part on input from at least one microphone, execute beamforming and provide audio at a hearing aid comprising sound from the source.

<CIT> describes a hearing aid system, which comprises a wearable camera configured to capture images from an environment of a user, a microphone configured to capture sounds from the environment of the user, and a processor. The processor is programmed to receive images captured by the camera; receive audio signals representative of sounds captured by the microphone; operate in a first mode to cause a first selective conditioning of a first audio signal; determine, based on analysis of at least one of the images or the audio signals, to switch to a second mode to cause a second selective conditioning of the first audio signal; and cause transmission of the first audio signal selectively conditioned in the second mode to a hearing interface device configured to provide sound to an ear of the user.

<CIT> describes a method for presenting to a user of a wearable audio device a modified audio scene together with additional information related to the audio scene, comprising: capturing audio signals with a plurality of microphones; outputting an audio signal with a plurality of acoustical transducers; processing the captured audio signals, the processing comprising filtering, equalization, echoes processing and/or beamforming; separating audio sources from the processed audio signals; selecting at least one separated audio source; classifying at least one said selected audio source; retrieving additional information related to the classified audio source; presenting the additional information to the user.

<CIT> describes a hearing aid system including a wearable camera; a microphone; and a processor. The processor is programmed to receive images captured by the camera; receive audio signals representative of sounds received by the at least one microphone; determine a look direction of the user based on analysis of the images; determine an amplitude of a first audio signal associated with an individual or object in a region associated with the look direction of the user; determine an amplitude of a second audio signal from a region other than the look direction of the user; adjust the second amplitude in accordance with the first amplitude; and cause transmission of the second audio signal at the adjusted amplitude to a hearing interface device configured to provide sound to an ear of the user.

It is an objective of the present invention to provide a method and a computer program for operating a hearing system, which enables a quick and/or easy in-the-field-setting of the hearing system for a user of the hearing system, in particular in complex acoustic environments. It is another objective of the present invention to provide the hearing system and a computer-readable medium in which the computer program is stored.

These objectives are achieved by the subject-matter of the independent claims. Further exemplary embodiments are evident from the dependent claims and the following description.

A first aspect relates to a method for operating a hearing system. The hearing system comprises a hearing device configured to be worn at an ear of a user, a user device communicatively coupled to the hearing device and comprising a camera and a display, with the hearing device comprising at least one sound input module for generating an audio signal indicative of a sound detected in an environment of the hearing device, a first processing unit for modifying the audio signal, and at least one sound output module for outputting the modified audio signal. The method comprises: receiving image data from the camera, the image data being representative of a scene in front of the camera; receiving an audio signal from the at least one sound input module, the audio signal being representative of the acoustic environment of the hearing device substantially at a time the image data have been captured, wherein the acoustic environment comprises at least one audio source and wherein the audio signal is at least in part representative for a sound from the audio source; and determining at least one visual object as the audio source, within the scene from the image data and the audio signal.

The method may be a computer-implemented method, which may be performed automatically by the hearing system. The step of determining the at least one visual object as the audio source within the scene from the image data and the audio signal may be carried out by an artificial intelligence and/or a neural network. The artificial intelligence or, respectively the neural network, may be trained with the data set comprising a huge amount of image data representing different scenes with visual objects, wherein at least some of the visual objects are the audio sources, and a corresponding amount of audio signals associated and/or synchronized with the image data.

The hearing system may, for instance, comprise one or two hearing devices used by the same user. One or both of the hearing devices may be worn on or in an ear of the user. A hearing device may be a hearing aid, which may be adapted for compensating a hearing loss of the user. Also, a cochlear implant may be a hearing device or at least a part of it. The hearing system may optionally further comprise at least one connected user device, such as a smartphone, smartwatch, smart glasses, or another device carried by the user or a personal computer of the user etc. The visual objects may be determined from the image data by analysing the image data with the help of a data base comprising several different objects and/or object classes. The time the image data have been captured may be encoded in meta data accompanying the image data. Alternatively or additionally, the image data may be captured and received in real time such that the time of receiving the image data automatically corresponds to the time the audio signal is captured and received. The visual object may be selected by a touch on the display, if the display is a touch screen.

As explained above, the audio signal is representative of the acoustic environment of the hearing device substantially at a time the image data have been captured, wherein "substantially" may mean in this context, that the audio signal is representative of the acoustic environment of the hearing device at the time the image data have been captured, that the audio signal is representative of the acoustic environment of the hearing device during a time interval in which the image data have been captured, or that the audio signal is representative of the acoustic environment of the hearing device during a time interval which overlaps a time interval during which the image data have been captured. The audio signal may comprise meta data representing the time or time interval during which the corresponding sound of the acoustic environment has been captured. Alternatively or additionally, the image may comprise meta data representing the time or time interval during which the image data have been captured. The audio signal and the image data may be synchronized by these meta data.

A second aspect relates to the hearing system. The hearing system comprises: the hearing device configured to be worn at an ear of a user and comprising the at least one sound input module for generating the audio signal, the first processing unit for modifying the audio signal, and the at least one sound output module for outputting the modified audio signal; and the user device communicatively coupled to the hearing device and comprising the display, the camera, and a second processing unit; wherein at least one control unit is coupled to the hearing device and the user device and is configured to carry out the above method. The hearing system may further include, by way of example, a second hearing device worn by the same user. If the user device is a smartphone, the camera may be the camera implemented within the smartphone. Alternatively, the camera may be implemented in smart glasses, if the user device is the smart glasses.

A third aspect relates to a computer program for operating the hearing system, which program, when being executed by a processing unit, e.g. the first and/or second processing unit, is adapted to carry out the steps of the above method.

A fourth aspect relates to a computer-readable medium, in which the above computer program is stored. In general, a computer-readable medium may be a floppy disk, a hard disk, an USB (Universal Serial Bus) storage device, a RAM (Random Access Memory), a ROM (Read Only Memory), an EPROM (Erasable Programmable Read Only Memory) or a FLASH memory. A computer-readable medium may also be a data communication network, e.g. the Internet, which allows downloading a program code. The computer-readable medium may be a non-transitory or transitory medium.

For example, the computer program may be executed in the first processing unit of the hearing device, which hearing device, for example, may be carried by the person behind the ear. The computer-readable medium may be a memory of this hearing device. The computer program also may be executed by the second processing unit of the connected user device, such as a smartphone or any other type of mobile device, which may be a part of the hearing system, and the computer-readable medium may be a memory of the connected user device. It also may be that some steps of the method are performed by the hearing device and other steps of the method are performed by the connected user device.

It has to be understood that features of the method as described above and in the following may be features of the computer program, the computer-readable medium and/or the hearing system as described above and in the following, and vice versa.

Determining the at least one visual object as the audio source within the scene from the image data and the audio signal enable a quick and/or easy in-the-field-setting of the hearing system for the user of the hearing system, in particular in complex acoustic environments. In general, in-the-field-setting is getting more and more important as it has many advantages compared with the conventional hearing device setting in a sound booth. The above method contributes to increasing the trust of a customer as it ensures a quick and easy solution for difficult listening situations, i.e. complex acoustic environments with e.g. several audio and/or noise sources.

According to the invention, the method further comprises: displaying the scene on the display and marking the determined visual object within the scene; receiving an input of the user, the input being representative for the user selecting the marked visual object and for the user wishing to selectively modify the sound from the audio source associated with the selected visual object; selectively modifying the audio signal from the audio source associated with the selected visual object; and outputting the modified audio signal to the user. This gives the user the possibility to visually selecting the corresponding sound sources. With this in-the-field-setting approach of the above method, more individual user data may be obtained. Also, users without a good technical understanding can use that approach very well. Especially data regarding the users' individual visual selection of the visual objects and the associated audio source of interest, what corresponds to a listening intention of the user, may be valuable for an Al-based setting.

According to an embodiment, if the step of determining at least one visual object as the audio source, within the scene from the image data and the audio signal, is not carried out by an artificial intelligence and/or a neural network, the step of determining at least one visual object as the audio source, within the scene from the image data and the audio signal may comprise: determining the at least one visual object, which is a potential audio source, within the scene from the image data; determining the at least one audio source within the acoustic environment from the audio signal; and associating at least one determined visual object with at least one determined audio source, wherein the determined visual object may be associated with the determined audio source. For example, several visual objects are determined and several audio sources are determined. Then, some of the several determined visual objects may be associated with one of the several determined audio sources each.

According to an embodiment, the step of determining the at least one visual object, which is the potential audio source, comprises determining a first spatial relationship between the camera and the visual object, e.g. a direction and/or a distance from the camera to the visual object; the step of determining the at least one audio source within the acoustic environment comprises determining a second spatial relationship between the hearing device and the audio source, e.g. a direction and/or a distance from the hearing device to the audio source, wherein the audio signal may be a stereo-signal; and the step of associating the at least one determined visual object with the at least one determined audio source comprises comparing the first spatial relationships of all determined visual objects with the second spatial relationships of all determined sound sources, and associating that visual object with that audio source such that the corresponding first and second spatial relationship fulfil a predetermined requirement. The predetermined requirement may be the "best fit" of the spatial relationships between the camera and the visual object and the hearing device and the audio source.

According to an embodiment, the method further comprises classifying the acoustic environment from the received audio signal; modifying the audio signal in accordance with the classification; and determining the at least one audio source within the acoustic environment from the modified audio signal. Depending on the classified acoustic environment, a set of feature parameters is selected as a determined sound program. With such an acoustic environment classification, an acoustic situation the wearer is in is classified and consequently categorized in order to automatically adjust the features and/or values of parameters of these features in accordance with the current acoustic situation. Optionally a feature activity may be logged such that it is logged which feature is active at which time. Classifying the acoustic environment and modifying the audio signal in accordance with the classification by the corresponding sound program including a set of features contributes to identify a signal to noise ratio, an object loudness, a type of noise or room acoustic, a pitch of the object or other information which may be required to choose the most effective sound cleaner or frequency dependent gain modification.

According to the invention, the method further comprises determining at least one object class of the determined visual object and labelling the marked visual object in the scene in accordance with the determined object class. The labels assist the user in identifying the marked visual object. The object class may be at least one of the group of human, animal, instrument, speaker, dishes, newspaper, car, and water. For each object class, auto-adjustments, macro modifications and/or a list of modifiers may be defined, which may have an impact on the sound from the corresponding sound source.

According to the invention, the method further comprises providing at least one input field on the display for the input of the user, with the input field being representative for at least one modification of the audio signal with respect to the sound from the audio source assigned to the selected visual object, wherein the audio signal is selectively modified with respect to the sound from the audio source assigned to the selected visual object, if the user activates the input field. The input field may be activated by a direct pressure on the input field or by a gesture above, on or next to the input field. The input field on the display represents an intuitive possibility for the user to quickly and easily set the preferred modification.

According to an embodiment, the input field is provided depending on the classification of the acoustic environment, with the input field being representative for at least one modification in accordance with the classification of the acoustic environment.

According to the invention, the input field is provided depending on the object class of the determined visual object, with the input field being representative for at least one modification in accordance with the object class of the determined visual object. Providing the input field depending on the classification enables to provide the user with the optimal and/or preferred option for modifying the audio signal in the current acoustic situation.

According to an embodiment, the input is representative for the user wishing to increase the volume of the sound from the selected visual object and/or to decrease the volume or effectuate a dampening of the sound from all other determined visual objects, or to decrease the volume or effectuate a dampening of the sound from the selected visual object and/or to increase the volume of the sound from all other determined visual objects, and the audio signal is selectively modified such that the volume of the audio source associated with the selected visual object is increased or, respectively decreased, or that the volume of the audio sources of all other determined visual objects is decreased or, respectively, increased.

According to an embodiment, the method further comprises monitoring the visual object by the camera and stopping to selectively modify the audio signal with respect to the sound from the audio source assigned to the monitored visual object if the visual object disappears from a field of view of the camera. This enables to save processing resources for selectively modifying the audio signal with respect to the sound from the audio source assigned to the monitored visual object, if the visual object and as such the audio source disappears. The audio signal from the visual object may be modified again as soon as the visual object is recognized within the scene again.

According to an embodiment, the method further comprises detecting at least one gesture of the user on or above the display; and selecting the marked visual object in accordance with the gesture; and/or selectively modifying the audio signal with respect to the sound from the audio source associated with the selected visual object in accordance with the gesture. Detecting the gesture provides a very intuitive input possibility for the user.

According to an embodiment, the hearing system further comprises a remote server communicatively coupled to the hearing device and/or the user device and being configured to carry out at least a part of the above method. The provision of the server enables to outsource processing tasks from the hearing device and/or the user device to the server. This is especially advantageous, if the corresponding processing tasks need huge processing resources and/or if the hearing device and, respectively, the user device have to be relieved.

According to an embodiment, the control unit is implemented in the first processing unit, the second processing unit or the remote server. In other words, the processing of the above method may be controlled by the hearing device, the user device, or, respectively, the remote server. The user device may be connected to a cloud and/or the internet. Some of the steps of the method described here and further below may be executed on the hearing device, the user device or in the cloud, or any combination thereof.

Below, embodiments of the present invention are described in more detail with reference to the attached drawings.

<FIG> schematically shows a hearing system <NUM> according to an embodiment of the invention. The hearing system <NUM> includes a hearing device <NUM> and a user device <NUM> connected to the hearing device <NUM>. As an example, the hearing device <NUM> is formed as a behind-the-ear device carried by a user (not shown) of the hearing device <NUM>. It has to be noted that the hearing device <NUM> is a specific embodiment and that the method described herein also may be performed with other types of hearing devices, such as e.g. an in-the-ear device or one or two of the hearing devices <NUM> mentioned above. The user device <NUM> may be a smartphone, a tablet computer, and/or smart glasses.

The hearing device <NUM> comprises a part <NUM> behind the ear and a part <NUM> to be put in the ear channel of the user. The part <NUM> and the part <NUM> are connected by a tube <NUM>. The part <NUM> comprises at least one sound input module <NUM>, e.g. a microphone or a microphone array, a sound output module <NUM>, such as a loudspeaker, and an input mean <NUM>, e.g. a knob, a button, or a touch-sensitive sensor, e.g. capacitive sensor. The sound input module <NUM> can detect a sound in the environment of the user and generate an audio signal indicative of the detected sound. The sound output module <NUM> can output sound based on the audio signal modified by the hearing device <NUM>, wherein the sound from the sound output module <NUM> is guided through the tube <NUM> to the part <NUM>. The input mean <NUM> enables an input of the user into the hearing device <NUM>, e.g. in order to power the hearing device <NUM> on or off, and/or for choosing a sound program or any other modification of the audio signal.

The user device <NUM> comprises a display <NUM>, e.g. a touch-sensitive display, providing a graphical user interface <NUM> including control element <NUM>, e.g. a slider, which may be controlled via a touch on the display <NUM>, and a camera <NUM>. The control element <NUM> may be referred to as input means of the user device <NUM>. The camera <NUM> may be a photo camera and/or video camera. If the user device <NUM> is the smart glasses, the use device <NUM> may comprise a knob or button instead of the display <NUM> and/or the graphical user interface <NUM>.

<FIG> shows a block diagram of components of the hearing system <NUM> according to <FIG>.

The hearing device <NUM> comprises a first processing unit <NUM>. The first processing unit <NUM> is configured to receive the audio signal generated by the sound input module <NUM>. The hearing device <NUM> may include a sound processing module <NUM>. For instance, the sound processing module <NUM> may be implemented as a computer program executed by the first processing unit <NUM>. The sound processing module <NUM> may be configured to modify, in particular increase or decrease a volume of and/or delay, the audio signal generated by the sound input module <NUM>, e.g. some frequencies or frequency ranges of the audio signal depending on parameter values of parameters, which influence the amplification, the damping and/or, respectively, the delay, e.g. in correspondence with a current sound program. The parameter may be one or more of the group of frequency dependent gain, time constant for attack and release times of compressive gain, time constant for noise canceller, time constant for dereverberation algorithms, reverberation compensation, frequency dependent reverberation compensation, mixing ratio of channels, gain compression, gain shape/amplification scheme. A set of one or more of these parameters and parameter values may correspond to a predetermined sound program, wherein different sound programs are characterized by correspondingly different parameters and parameter values. The sound program may comprise a list of sound processing features. The sound processing features may for example be a noise cancelling algorithm or a beamformer, which strengths can be increased to increase speech intelligibility but with the cost of more and stronger processing artifacts. The sound output module <NUM> generates sound from the modified audio signal and the sound is guided through the tube <NUM> and the in-the-ear part <NUM> into the ear channel of the user.

The hearing device <NUM> may include a control module <NUM>. For instance, the control module <NUM> may be implemented as a computer program executed by the first processing unit <NUM>. The control module <NUM> may be configured for adjusting the parameters of the sound processing module <NUM>, e.g. such that an output volume of the sound signal is adjusted based on an input volume. For example, the user may select a modifier (such as bass, treble, noise suppression, dynamic volume, etc.) and levels and/or values of the modifiers with the input mean <NUM>. From this modifier, an adjustment command may be created and processed as described above and below. In particular, processing parameters may be determined based on the adjustment command and based on this, for example, the frequency dependent gain and the dynamic volume of the sound processing module <NUM> may be changed.

All these functions may be implemented as different sound programs stored in a first memory <NUM> of the hearing device <NUM>, which sound programs may be executed by the sound processing module <NUM>. The first memory <NUM> may be implemented by any suitable type of storage medium, in particular a non-transitory computer-readable medium, and can be configured to maintain, e.g. store, data controlled by the first processing unit <NUM>, in particular data generated, accessed, modified and/or otherwise used by the first processing unit <NUM>. The first memory <NUM> may also be configured to store instructions for operating the hearing device <NUM> and/or the user device <NUM> that can be executed by the first processing unit <NUM>, in particular an algorithm and/or a software that can be accessed and executed by the first processing unit <NUM>.

A sound source detector <NUM> may be implemented in a computer program executed by the first processing unit <NUM>. The sound source detector <NUM> is configured to determine at least the one sound source from the audio signal. In particular, the sound source detector <NUM> may be configured to determine a spatial relationship between the hearing device <NUM> and the corresponding sound source. The spatial relationship may be given by a direction and/or a distance from the hearing device <NUM> to the corresponding audio source, wherein the audio signal may be a stereo-signal and the direction and/or distance may be determined by different arrival times of the sound waves from one audio source at two different sound input modules <NUM> of the hearing device <NUM> and/or a second hearing device <NUM> worn by the same user.

A first classifier <NUM> may be implemented in a computer program executed by the first processing unit <NUM>. The first classifier <NUM> can be configured to evaluate the audio signal generated by the sound input module <NUM>. The first classifier <NUM> may be configured to classify the audio signal generated by the sound input module <NUM> by assigning the audio signal to a class from a plurality of predetermined classes. The first classifier <NUM> may be configured to determine a characteristic of the audio signal generated by the sound input module <NUM>, wherein the audio signal is assigned to the class depending on the determined characteristic. For instance, the first classifier <NUM> may be configured to identify one or more predetermined classification values based on the audio signal from the sound input module <NUM>. The classification may be based on a statistical evaluation of the audio signal and/or a machine learning (ML) algorithm that has been trained to classify the ambient sound, e.g. by a training set comprising a huge amount of audio signals and associated classes of the corresponding acoustic environment. So, the ML-algorithm may be trained with several audio signals of acoustic environments, wherein the corresponding classification is known.

The first classifier <NUM> may be configured to identify at least one signal feature in the audio signal generated by the sound input module <NUM>, wherein the characteristic determined from the audio signal corresponds to a presence and/or absence of the signal feature. Exemplary characteristics include, but are not limited to, a mean-squared signal power, a standard deviation of a signal envelope, a mel-frequency cepstrum (MFC), a mel-frequency cepstrum coefficient (MFCC), a delta mel-frequency cepstrum coefficient (delta MFCC), a spectral centroid such as a power spectrum centroid, a standard deviation of the centroid, a spectral entropy such as a power spectrum entropy, a zero crossing rate (ZCR), a standard deviation of the ZCR, a broadband envelope correlation lag and/or peak, and a four-band envelope correlation lag and/or peak. For example, the first classifier <NUM> may determine the characteristic from the audio signal using one or more algorithms that identify and/or use zero crossing rates, amplitude histograms, auto correlation functions, spectral analysis, amplitude modulation spectrums, spectral centroids, slopes, roll-offs, auto correlation functions, and/or the like. In some instances, the characteristic determined from the audio signal is characteristic of an ambient noise in an environment of the user, e.g. a noise level, and/or a speech, e.g. a speech level. The first classifier <NUM> may be configured to divide the audio signal into a number of segments and to determine the characteristic from a particular segment, e.g. by extracting at least one signal feature from the segment. The extracted feature may be processed to assign the audio signal to the corresponding class.

The first classifier <NUM> may be further configured to assign, depending on the determined characteristic, the audio signal generated by the sound input module <NUM> to a class of at least two predetermined classes. The classes may represent a specific content in the audio signal. For instance, the classes may relate to a speaking activity of the user and/or another person and/or an acoustic environment of the user. Exemplary classes include, but are not limited to, low ambient noise, high ambient noise, traffic noise, music, machine noise, babble noise, public area noise, background noise, speech, nonspeech, speech in quiet, speech in babble, speech in noise, speech in loud noise, speech from the user, speech from a significant other, background speech, speech from multiple sources, calm situation and/or the like. The first classifier <NUM> may be configured to evaluate the characteristic relative to a threshold. The classes may comprise a first class assigned to the audio signal when the characteristic is determined to be above the threshold, and a second class assigned to the audio signal when the characteristic is determined to be below the threshold. For example, when the characteristic determined from the audio signal corresponds to ambient noise, a first class representative of a high ambient noise may be assigned to the audio signal when the characteristic is above the threshold, and a second class representative of a low ambient noise may be assigned to the audio signal when the characteristic is below the threshold. As another example, when the characteristic determined from the audio signal is characteristic of a speech, a first class representative of a larger speech content may be assigned to the audio signal when the characteristic is above the threshold, and a second class representative of a smaller speech content may be assigned to the audio signal when the characteristic is below the threshold.

At least two of the classes can be associated with different sound programs, in particular with different sound processing parameters, which may be applied by the sound processing module <NUM> for modifying the audio signal. To this end, the class assigned to the audio signal, which may correspond to a classification value, may be provided to the control module <NUM> in order to select the associated audio processing parameters, in particular the associated sound program, which may be stored in the first memory <NUM>. The class assigned to the audio signal may thus be used to determine the sound program, which may be automatically used by the hearing device <NUM>, in particular depending on the audio signal received from the sound input module <NUM>.

The hearing device <NUM> may further comprise a first transceiver <NUM>. The first transceiver <NUM> may be configured for a wireless data communication with a remote server <NUM>. Additionally or alternatively, the first transceiver <NUM> may be adapted for a wireless data communication with a second transceiver <NUM> of the user device <NUM>. The first and/or the second transceiver <NUM>, <NUM> each may be e.g. a Bluetooth or RFID radio chip.

Each of the sound processing module <NUM>, the control module <NUM>, the sound source detector <NUM>, and the first classifier <NUM> may be embodied in hardware or software, or in a combination of hardware and software. Further, at least two of the modules <NUM>, <NUM>, <NUM>, <NUM> may be consolidated in one single module or may be provided as separate modules. The first processing unit <NUM> may be implemented as a single processor or as a plurality of processors. For instance, the first processing unit <NUM> may comprise a first processor in which the sound processing module <NUM> is implemented, and a second processor in which the control module <NUM> and/or the sound source detector <NUM> and/or the first classifier <NUM> are implemented.

The user device <NUM>, which may be connected to the hearing device <NUM>, may comprise a second processing unit <NUM>, a second memory <NUM>, a second transceiver <NUM>, a second classifier <NUM> and/or a visual object detector <NUM>.

The second processing unit <NUM> may comprise one or more processors, e.g. for running the second classifier <NUM> and/or the visual object detector <NUM>. If the hearing device <NUM> is controlled via the user device <NUM>, the second processing unit <NUM> of the user device <NUM> may be seen at least in part as a controller of the hearing device <NUM>. In other words, according to some embodiments, the first processing unit <NUM> of the hearing device <NUM> and the second processing unit <NUM> of the user device <NUM> may form the controller of the hearing device <NUM>. A processing unit of the hearing system <NUM> may comprise the first processing unit <NUM> and the second processing unit <NUM>. The processing units <NUM>, <NUM> may communicate data via the first and second transceivers <NUM>, <NUM>.

The second classifier <NUM> may have the same functionality as the first classifier <NUM> explained above. The second classifier <NUM> may be arranged alternatively or additionally to the first classifier <NUM> of the hearing device <NUM>. The second classifier <NUM> may be configured to classify the acoustic environment of the user and the user device <NUM> depending on the received audio signal, as explained above with respect to the first classifier <NUM>, wherein the acoustic environment of the user and the user device <NUM> corresponds to the acoustic environment of the hearing device <NUM> and wherein the audio signal may be forwarded from the hearing device <NUM> to the user device <NUM>.

The visual object detector <NUM> is configured to identify one or more visual objects <NUM> (see <FIG>) within the scene <NUM> taken by the camera <NUM>. Further, the visual object detector <NUM> may be configured to communicate with a display control (not shown) of the user device of <NUM>, such that the display control controls the display <NUM> in order to visually mark the determined visual objects <NUM>. The visual object detector <NUM> may be implemented as an algorithm and/or an artificial intelligence. In case of an artificial intelligence, the algorithm has to be trained, e.g. with the help of the huge set of image data representing pictures each including one or more visual objects. A list of potential visual objects may be stored in a database, which may be stored in the first and/or second memory <NUM>, <NUM>. Further, a set of adjustable audio sources and corresponding object classes may be predefined, like e.g. human, instrument, speaker, dishes, newspaper, car, water, noise, etc. For each of these audio sources, auto-adjustments, macro modifications and/or a list of modifiers may be predefined, which may have an impact on the corresponding sound object. An exemplary auto-adjustment may be decreasing an overall gain, e.g. of <NUM>. 5dB, increasing a noise reduction strength, e.g. to <NUM>, and increasing a beamformer strength, e.g. to <NUM>.

There are several conventional high-performing algorithms available, which are able to detect visual objects within an image. One of the best performing algorithms is the YOLO (You Only Look Once) method. This algorithm is able to run on a smartphone in real time as it processes only one image at the same time. Hence, it is proposed to use the image data of the camera <NUM> or alternatively from a smart glasses camera, a smartwatch camera, a TV camera, or a recorded video, to run the visual object detection algorithm on the user device <NUM>.

With the hearing system <NUM> it is possible that the above-mentioned modifiers and their levels and/or values are adjusted with the user device <NUM> and/or that the adjustment command is generated with the user device <NUM>. This may be performed with a computer program run in the second processing unit <NUM> and stored in the second memory <NUM> of the user device <NUM>. This computer program may also provide the graphical user interface <NUM> on the display <NUM> of the connected user device <NUM>. For example, for adjusting the modifier, such as volume, the graphical user interface <NUM> may comprise the control element <NUM>, such as a slider. When the user adjusts the slider, an adjustment command may be generated, which will change the sound processing of the hearing device <NUM> as described above and below. Alternatively or additionally, the user may adjust the modifier with the hearing device <NUM> itself, for example via the input mean <NUM>.

The hearing device <NUM> and/or the user device <NUM> may communicate with each other and/or with the remote server <NUM> via the Internet <NUM>. The method explained below with respect to <FIG> and/or <NUM> may be carried out at least in part by the remote server <NUM>. For example, processing tasks, which require a huge amount of processing resources, may be outsourced from the hearing device <NUM> and/or the user device of <NUM> to the remote server <NUM>. For example, the determination of the visual objects from the image data and/or of the audio sources from the audio signal, may be outsourced to the remote server <NUM>. Further, the processing units (not shown) of the remote server <NUM> may be used at least in part as the controller for controlling the hearing device <NUM> and/or the use device <NUM>.

<FIG> shows a flow diagram of a method for operating the hearing system <NUM>. The method may be carried out by the first and/or the second processing unit <NUM>, <NUM> and/or by the remote server <NUM>, wherein some of the steps of the method may be carried out by the first and/or the second processing unit <NUM>, <NUM> and/or some other steps of the method may be carried out by the remote server <NUM>.

In a step S2, image data from the camera <NUM> are received, e.g. by the first or second processing unit <NUM>. The image data are representative for a scene <NUM> in front of the user. In particular, a picture or video of the front of the user may be taken by the camera <NUM> and the camera <NUM> generates the image data representing the scene <NUM> in front of the user.

In a step S4, an audio signal from the at least one sound input module <NUM> may be received, e.g. by the first or second processing unit <NUM>. The audio signal is representative for the acoustic environment of the user at a time the image data have been captured. The acoustic environment comprises at least one audio source and the audio signal is at least in part representative for a sound from the audio source.

In a step S6, at least one visual object <NUM> is determined as the audio source, within the scene <NUM> from the image data and the audio signal. Step S6 may be carried out by an artificial intelligence and/or by a "traditional" algorithm.

In a step S8, the scene <NUM> is displayed on the display <NUM>.

<FIG> shows an example of the scene <NUM> including several visual objects <NUM>. In particular, the scene <NUM> comprises four persons <NUM>, wherein two of the persons <NUM> are seen in the upper left in the background of the scene <NUM>, one person <NUM> is shown at the right in the background of the scene <NUM> and one person <NUM> is shown in the middle in the front of the scene <NUM>. Further, the scene <NUM> comprises an instrument <NUM> and a speaker <NUM> both representing visual objects <NUM> within the scene <NUM> and potential audio sources.

In a step S10, at least one determined visual object <NUM> is marked within the scene <NUM>. Further, the user may be prompted to select at least one marked visual object <NUM> within the scene <NUM>. If one of the visual objects <NUM> is detected as an adjustable audio source, the user has to become aware of it. One way to indicate a detected audio source is to highlight the corresponding visual object <NUM> within the scene <NUM>. This may be done with augmenting a boundary around the detected visual object <NUM> within the scene <NUM>. Another way to indicate the detected visual object <NUM> acting as an audio source is to show a corresponding object related modifier directly within the scene <NUM> (see <FIG> and/or <NUM>).

<FIG> shows the scene <NUM> of <FIG> with several marked visual objects <NUM>. In particular, the persons <NUM>, the instrument <NUM> and the speaker <NUM> are identified and marked as visual objects <NUM> potentially acting as audio sources. The visual objects <NUM> are marked by the fine dashed rectangles, as an example.

In a step S12, an input of the user is received. The input is representative for that the user selected at least one of the marked visual objects <NUM>. Each marked visual object <NUM>, which has been selected by the user, is referred to as selected visual object <NUM> in the following. The input is further representative for that the user wishes to selectively modify the sound from the audio source associated with the selected visual object <NUM>. Optionally, the input may be representative for the user wishing to increase the volume of the sound from the selected visual object <NUM> and/or to decrease the volume or effectuate a dampening of the sound from all other determined visual objects <NUM>. Alternatively, the input may be representative for the user wishing to decrease the volume or effectuate a dampening of the sound from the selected visual object <NUM> and/or to increase the volume of the sound from all other determined visual objects <NUM>. If the visual objects <NUM> potentially acting as an audio source are highlighted within the scene <NUM>, e.g. with an augmented boundary around the corresponding visual object <NUM>, the augmented boundary itself may be a hitbox to select the corresponding visual object <NUM>. If the object-based modifier is shown directly when a visual object <NUM> acting as an audio source is detected within the scene <NUM>, this step becomes obsolete, because the user automatically selects one of the visual objects <NUM>, if he/she activates the corresponding object-based modifier.

<FIG> shows the scene of <FIG> with two selected visual objects <NUM>. In particular, in <FIG>, the instrument <NUM> and the speaker <NUM> are marked as selected visual objects <NUM>, e.g. by course dashed rectangles.

In a step S14, the audio signal from the audio source associated with the selected visual object <NUM> is selectively modified. Optionally, the audio signal may be selectively modified such that the audio source associated with the selected visual object <NUM> is amplified or that the volume of the audio sources of all other determined visual objects <NUM> is decreased or a dampening thereof is effectuated. Alternatively the audio signal may be selectively modified such that the volume of the audio source associated with the selected visual object <NUM> is decreased or a dampening thereof is effectuated, and/or that the volume of the audio sources of all other determined visual objects <NUM> is increased.

In a step S16, the modified audio signal is outputted to the user.

Optionally, the method further comprises the step of monitoring the visual object(s) <NUM> by the camera <NUM> and stopping to selectively modify the audio signal with respect to the sound from the audio source assigned to the monitored visual object <NUM>, if the visual object <NUM> disappears from a field of view of the camera <NUM>.

A pure object detection might not be sufficient for some applications as it is not able to detect the signal to noise ratio, object loudness, type of noise or room acoustic, pitch of the object or other information which may be required to choose the most effective sound cleaner or frequency dependent gain modification. Therefore, the above method may further comprise the steps of classifying the acoustic environment from the received audio signal, modifying the audio signal in accordance with the corresponding classification, and step S6, e.g. determining the at least one audio source within the acoustic environment, may be carried out using the modified audio signal.

A combination of a visual and an acoustic classification may be even better able to provide the optimal adjustment parameter. Therefore, the above method may further comprise the steps of determining at least one object class of the determined visual object <NUM>. A list of adjustable visual objects acting as audio sources or corresponding sound object classes may be available in a corresponding database stored within the first and/or second memory <NUM>, <NUM> and/or within the remote server <NUM>. The corresponding visual and acoustic object detection algorithm may be trained to estimate the most possible and effective sound object class and its adjustment suggestion. This adjustment suggestion may be applied automatically or for manual adjustments offered to the user.

<FIG> shows a flow diagram of a sub-method of the method of <FIG>. In particular, <FIG> shows a sub-routine of the above method in case the method is implemented by a "traditional" algorithm and not by an artificial intelligence and/or a neuronal network. In this case step S6, i.e. the step of determining at least one visual object <NUM> as the audio source within the scene <NUM> from the image data and the audio signal, may comprise the following steps S18, S20 and S22.

In step S18 at least one visual object <NUM>, which is a potential audio source, is determined within the scene <NUM> from the image data. For example, step S18 comprises determining a first spatial relationship between the camera <NUM> and the visual object <NUM>.

In step S20, at least one audio source is determined within the acoustic environment from the audio signal. For example, step S20 comprises determining a second spatial relationship between the hearing device <NUM> and the audio source.

In step S22, at least one determined visual object <NUM> is associated with at least one determined audio source. For example, the first spatial relationships of all determined visual objects <NUM> may be compared with the second spatial relationships of all determined sound sources. Then, that visual object <NUM> of all determined visual objects <NUM> may be associated with that audio source of all determined sound sources such that the corresponding first and second spatial relationship fulfil a predetermined requirement. The predetermined requirement may be the "best fit" of the spatial relationship between the camera and the visual object and the hearing device and the audio source.

<FIG> shows another example of a scene <NUM> including several marked and labelled visual objects <NUM> in accordance with the present invention, in particular in accordance with the above method. In particular, all marked visual objects <NUM> within the scene <NUM> are labelled in accordance with the determined object class. For example, the woman in the background in the upper left has the label <NUM> "Female", the man in the middle in the front has also the label "Male", the man in the background on the right has the label "Male", and the clapping hands of the man in the background on the right have the label "Clapping". The labelling may be carried out within steps S8 or S10 of the above method.

Further, one, two or more state indicators <NUM> may be shown within the scene <NUM>. For example, the state indicators <NUM> may indicate the classification of the acoustic environment, if the acoustic environment is classified. In particular, the state indicators <NUM> may be representative for the noisy acoustic environment ("Noisy" in <FIG>) and/or that the acoustic environment is within a room or house (house-symbol in <FIG>).

According to the invention, the above method further comprises the step of providing the at least one input field <NUM> on the display <NUM> for the input of the user such that the input field <NUM> is representative for at least one modification of the audio signal with respect to the sound from the audio source assigned to the selected visual object <NUM>, wherein the audio signal is selectively modified with respect to the sound from the audio source assigned to the selected visual object <NUM>, if the user activates the input field <NUM>. For example, an input field <NUM> may be provided on the display <NUM> within the scene <NUM>. The input field <NUM> may comprise a one, two or more, e.g. four, buttons <NUM>. The user may use of the input field <NUM> and in particular the buttons <NUM> for the user input of step S12 of the above method. For example, some of the buttons <NUM> may be representative for that the proper visual object <NUM> is selected or not, e.g. such that the user can confirm or deny that the proper visual object <NUM> is selected. Alternatively or additionally, some of the buttons <NUM> may be representative for the predetermined modification of the audio signal with respect to the audio source of the selected visual object <NUM>. For example, the user may indicate that he wishes to increase or decrease the volume of the sound from the selected visual object <NUM> by pressing the corresponding button <NUM>.

Further, the input field <NUM> and in particular the buttons <NUM> represent object-based modifiers. In general, an object-based modifier may be an input field <NUM> and/or a button <NUM>, which appearance and/or function depends on the object class of the selected visual object <NUM>. For example, in a first case some of the buttons <NUM> are representative for increasing or decreasing the volume of the sound of the selected visual object <NUM>, if the selected visual object <NUM> is the man in the front of the scene <NUM>, and in contrast the buttons <NUM> at the same position within the input field <NUM> may be representative for increasing or decreasing the beamformer strength, if the selected visual object <NUM> is the clapping hands.

Optionally, an overview of the detected visual objects <NUM> potentially acting as an audio source and/or the corresponding recommended sound adjustments, which may be performed in the background of a sound object-based modifier (see <FIG>) may be shown to the user on the display <NUM>, e.g. within the scene <NUM>. In this way, the user can be instructed how to self-adjust the sound of the audio source corresponding to certain visual objects <NUM> properly.

<FIG> shows the scene of <FIG> and two examples of input fields <NUM>, wherein the input fields <NUM> of <FIG> may represent object-based modifiers. For example, the input fields <NUM> each comprise a label <NUM> and two buttons <NUM> for increasing and, respectively decreasing, the value of one of the parameters for modifying the audio signal with respect to the sound from the audio source.

Optionally, one or more of the above input fields <NUM> are provided depending on the classification of the acoustic environment, with the corresponding input field <NUM> being representative for at least one modification in accordance with the classification of the acoustic environment. Additionally, the corresponding input field <NUM> is provided depending on the object class of the determined visual object <NUM>, with the input field <NUM> being representative for at least one modification in accordance with the object class of the determined visual object <NUM>, wherein in this case the input field <NUM> may be also referred to as object-based modifier.

<FIG> shows another example of a scene <NUM> including a labelled and selected visual object <NUM> and several examples for selecting a proper modification of the corresponding audio source. For example, the above method may further comprise the steps of detecting at least one gesture <NUM> of the user on or above the display <NUM>, wherein the marked visual object <NUM> may be selected in accordance with the gesture <NUM> and/or the audio signal may be selectively modified with respect to the sound from the audio source associated with the selected visual object <NUM> in accordance with the gesture <NUM>. The gesture <NUM> may be a movement of the thumb relative to the forefinger, wherein an increase of the distance between the thumb and the forefinger may be representative for an increase of the value of the corresponding parameter for modifying the audio signal with respect to the selected visual object <NUM> and/or a decrease of the distance between the thumb and the forefinger may be representative for a decrease of the value of the corresponding parameter for modifying the audio signal with respect to the selected visual object <NUM>. Further, a beamformer symbol <NUM>, e.g. a triangle, may be laid over the selected visual object <NUM> and the user may increase or decrease an upper base of the triangle and as such the value of the corresponding parameter by the gesture <NUM>. Alternatively or additionally the slider <NUM> may be provided on the display <NUM> within the scene <NUM>, wherein the slider <NUM> may be used to increase or decrease the value of one of the parameters for modifying the audio signal.

<FIG> shows an example of a visual object-based audio signal modifier implementation. In particular, <FIG> shows an example of a visual object Objk acting as an audio source, wherein the visual object Objk may be taken from a lookup table. When such a visual object Objk is detected, the corresponding acoustical information is analysed and the most suitable sub-object Objk,<NUM>, Objk,<NUM>, Objk,<NUM> with its set of weights wa,. , wg may be chosen, e.g. depending on the corresponding acoustical context. The weight matrices may be added to the modifiers to consider the object-depended impact of each modifier as well as the acoustical context of the corresponding object Objk. So, each visual object Objk may have an individual set of weights wa,. , wg, which depends also on the acoustical information of the visual object. In addition, each visual object Objk may have the same modification possibilities Moda,. Depending on the visual object Objk and, in case the acoustic environment is classified, the acoustic feature activity the relevant modification parameters will be chosen.

One possible realization could be the set of weights wa,. , wg of the predefined modifiers Moda,. , Modg as shown in <FIG>. For each connection, there may be one of the weights wa,. , wg, which depends on the detected visual object Objk. This means, for each visual object and/or object class in the database there is the set of weights wa,. If the user wants to increase the sound from one of the visual objects Objk, there are modifiers, which may help for this purpose.

For example, the appropriate weights wa,. , wg may get a rather high value close to <NUM>, which means that the visual object-based modifier may have a high impact on this modifier for this specific visual object or object class. If the modifier is expected to not help for this purpose, the appropriate weights wa,. , wg may get a rather low value close to <NUM>, which means that the sound object-based modifier will have a low impact or no impact on this modifier for this specific visual object and/or object class. It is also possible to set the weights wa,. , wg in the mid-range so that there is a moderate impact on this modifier for a specific visual object and/or object class. A level dependency or other properties of the acoustic environment may be added to consider that the impact of each modifier on the same visual object <NUM> may have a different strength depending on the sound properties of the visual object <NUM> and the corresponding audio source as well as the acoustic environment.

Claim 1:
A method for operating a hearing system (<NUM>), the hearing system (<NUM>) comprising a hearing device (<NUM>) configured to be worn at an ear of a user,
a user device (<NUM>) communicatively coupled to the hearing device (<NUM>) and comprising a camera (<NUM>) and a display (<NUM>),
the hearing device (<NUM>) comprising at least one sound input module (<NUM>) for generating an audio signal indicative of a sound detected in an environment of the hearing device, a first processing unit (<NUM>) for modifying the audio signal, and at least one sound output module (<NUM>) for outputting the modified audio signal,
the method comprising:
receiving image data from the camera (<NUM>), the image data being representative of a scene (<NUM>) in front of the camera (<NUM>);
receiving an audio signal from the at least one sound input module (<NUM>), the audio signal being representative of the acoustic environment of the hearing device (<NUM>) substantially at a time the image data have been captured, wherein the acoustic environment comprises at least one audio source and wherein the audio signal is at least in part representative of a sound from the audio source;
determining at least one visual object (<NUM>) as the audio source within the scene (<NUM>) from the image data and the audio signal;
displaying the scene (<NUM>) on the display (<NUM>);
marking the determined visual object (<NUM>) within the scene (<NUM>);
determining at least one object class of the determined visual object (<NUM>) and labelling the marked visual object (<NUM>) in the scene (<NUM>) in accordance with the determined object class;
providing at least one input field (<NUM>) on the display (<NUM>) for an input of the user;
receiving the input of the user to select the marked visual object (<NUM>), wherein the input field (<NUM>) is representative for at least one modification of the audio signal with respect to the sound from the audio source assigned to the selected visual object (<NUM>);
selectively modifying the audio signal from the audio source associated with the selected visual object (<NUM>), if the user activates the input field (<NUM>);
outputting the modified audio signal to the user;
characterized in that the input field (<NUM>) is provided depending on the object class of the determined visual object (<NUM>), with the input field (<NUM>) being representative for at least one modification in accordance with the object class of the determined visual object (<NUM>).