title: Personalization of Hearing Aids using Daily Routine Recognition and Sensor Fusion
Creators: name: Kübert, Thomas
Publisher: name: UNSPECIFIED
Year: 2022

description: According to the World Health Organization, disabling hearing loss affects over 400 million people worldwide. Modern hearing aids (HA) can reduce this burden but require a situation-dependent control, which classifies the incoming sounds in a person-independent manner into predefined acoustic categories, such as speech in noise. For each category a corresponding setting, e.g., frequency gains, exists. However, the complex audio signal processing shows the highest benefits if the algorithms are optimally adapted to the respective acoustic situations and personal preferences. A stable and reliable situation identification is necessary for a natural and subtle HA control. To enhance the user satisfaction and increase the temporal prediction stability, we propose to personalize the classification system for each user by considering the recurring situations and environments of the daily routine. For this purpose, we link the daily routine situations and environments to preferred HA settings. Therefore, we are first to propose a combination of acceleration (ACC) and microphone data to recognize the daily routine in hearing aids. While acoustic classification systems are typically trained on selected real and controlled situations, we solely perform our analysis on realistic unconstrained situations and environments of HA wearers following their personal daily routine. Therefore, we create two realistic large data sets that form the basis for our comprehensive investigations. The first set DT contains one subject over 9 days with coarse time diary annotations for our feasibility experiments. The second data set D7 includes seven subjects over 104 days with intention-based user annotations for the model generalization investigations. For the recordings, we build on a HA prototype allowing to stream the ACC and audio data to a mobile phone. To show the feasibility of the approach and analyze which situations are distinguishable within the feature space, we perform clustering and visualization approaches on the data set DT. Thereby, we demonstrate that the ACC and audio features discriminate and group various daily routine situations and environments. These are visualized and clustered in data embeddings and feature plots over time. Using the knowledge of the discriminative routine situations and the coarse time diary annotations, we label the DT set by an extended semi-supervised algorithm. After that, the routine activities are recognized and the effect of three input variants, namely ACC, audio, and both, are analyzed. Within these experiments, we show the strong contribution of the audio features. After showing the proof-of-concept, the second set D7 is used to analyze the model generalization abilities across subjects. We train several classifiers in an offline, online, and sequence manner. To achieve this, we build an efficient feature representation, which describes the recurring daily situations and environments well. To recognize the daily routine, we apply and test various classification methods in an online manner. The multi-layer perceptron and random forest (RF) trained in a person-dependent way show the best F-measure performance. We confirm for high-level activities that the person-dependent model outperforms the independent one. In our online experiments, the goal is to improve the offline classification results and simulate a real system. Therefore, we personalize a model that was pretrained in a person-independent manner by daily online updates with the predicted or true user labels. Thereby, multiple incremental learners and an online RF are tested. We demonstrate that the RF can self-improve the F-measure compared to the offline baselines. In our sequence simulations, we model the temporal relationships of the sequential data to improve the routine detection. The results show a strong improvement in the daily routine recognition with the proposed sequence classification techniques. The sequence learners enhance the temporal stability of the predictions. In this dissertation, we show the potential of a personalized daily routine classification for an optimal HA configuration. Our efficient processing scheme allows to detect the routine classes linked to a preferred HA setting based on audio and acceleration features. To this end, our work contributes to support hearing aid wearers with an enhanced classification system to improve the user satisfaction.