Improving the robustness of machine learning-based tasks through optimal model selection using data quality
Authors:   , de Hoog Jens 
Published: October 2024

Deep Learning (DL) is being deployed in many different domains such as robotics, aerospace, automotive, healthcare and waste sorting. In each of these applications, sen- sors are employed to capture the environment. In turn, the DL models process this sensor data in order to infer intelligent decisions. Interesting to note is that, as a result of context changes or uncertainties, the environment in which the DL model is deployed could differ from the environment used for training. This phenomenon is known as domain shift or distribution shift. As the data between the training and deployment environment has changed due to this shift, one could argue that the quality of this data has decreased. As a result of this quality decrease, DL models show suboptimal performances. To address this challenge of quality degradation due to distribution shift, previous works have proposed various strategies, such as achieving generalisation through extensive datasets, model adaptation methods or Ensemble Learning (EL). However, each of these methods have their own complexities. Achieving generalisation through extensive datasets involves the inclusion of different contexts and uncertainties. This could be precar- ious: the class imbalance problem needs to be taken into account, while training on such datasets could be very computationally expensive. Adapting DL models has shown promising results, but could be less efficient regarding dealing with unseen shifts in distri- butions. Finally, EL has also shown generalisation improvements, but executing parallel DL models increases the computational cost. Hence, there is a need for a strategy that can cope with varying contexts or uncertainties, while no adaptations of the DL models are required and the additional computational complexity is limited. Research has indicated that each DL model approximates tasks differently due to varia- tions in initial states and hyperparameters, which leads to different generalisation capa- bilities. This means that each DL model has its own strengths and weaknesses. In this thesis, we investigate if these strengths and weaknesses can be leveraged intelligently based on the given situation, described by its context and uncertainties. That is, through a divide-and-conquer approach the most optimal model for an unseen situation is selected. The main research hypothesis of this thesis states that utilising the characteristics of DL models to select the most optimal model for an unseen situation leads to an approximation of the most optimal performance. To validate this hypothesis, several sub-challenges are addressed. First, the concept of data quality and the impact of distribution shift is defined. Subsequently, we investigate the relation between different sorts of quality degradations and the resulting effect on model performance measures. Third, we propose a method- ology to carry out performance measurements of DL models under varying degradations of quality, such that we gain deeper insights into the strengths and weaknesses of dif- ferent models. Fourth, we propose a methodology that, given the measured insights of the previous step, estimates the performance of DL models for unseen situations. In this way, performance estimations could be obtained in a more efficient way. Finally, through leveraging these performance estimations, a strategy is proposed to select the most suitable model for a given unseen situation, aiming to approximate the most optimal performance in this situation. In summary, this thesis aims to address the challenges associated with domain shift in DL models by leveraging the unique characteristics of each individual model. We indeed show that, through selection of the most suitable model, the most optimal performance for unseen situations can be approximated. Hence, this work offers a promising avenue for increased robustness and performance in environments exposed to distribution shift.