## 4 Results
### AD Definition as a Data Preprocessing Method (Substudy 1)
#### 4.1.2 Effects of X-outliers on the Model Performance  
To better understand the effects of X-outliers on the performance of the models, different amounts of X-outliers (0%, 10%, 30% and 50%) are eliminated from the preprocessed data on the basis of their anomaly score (N.B. the elimination of a percentage of outliers in this case reduces to the elimination of the same percentage of all molecules, with the highest anomaly score). The results are presented for enthalpy in Figure 5a-c, respectively, for _iForest_, _RF confidence_ and _ISNE2D/kNN_. In addition, in Figure 6a,b, the results of all three methods are compared for the training and test sets, respectively. Note that the same parameter initialization was used for the three methods (hence, the same \(MAE\) value at 0% removal). First of all, it can be clearly observed that the elimination of X-outliers with high anomaly scores improves the model performances on both the training and test datasets for _iForest_ and _RF confidence_ but not for _tSNE2D/kNN_. For example, in _RF confidence_, for which the most important improvement is observed, the elimination of the 10% most important outliers enables us to reduce the test \(MAE\) by 40%. These outliers correspond to 180 molecules approximately, for which the _RF confidence_ anomaly scores are above 150 kJ/mol. A possible explanation of the good performance of _RF confidence_ could be related to the simultaneous elimination of a higher percentage of y-outliers along with the X-outliers, as shown previously for enthalpy. In case of extreme elimination percentages of data points, as X-outliers (e.g., 70%, 90%), the test \(MAE\) will start to increase at some point as the number of remaining molecules will become too low for the model to learn the relationship between descriptors and property and therefore to generalize well. More generally, the elimination of X-outliers (when they are correctly identified with an adapted method) will improve the model's performance; however, care must be taken to ensure that they are properly distinguished from other \"useful\" data points. Indeed, X-outliers will affect the response surface and therefore deteriorate the predictions for the other data. Figure 5: Evolution of Lasso model performance in predicting the **enthalpy** under different proportions of X-outliers removed from the preprocessed data: **(a)**_iForest_, **(b)**_RF confidence_, **(c)**_tSNE2D/kNN_. The regression model can therefore be highly influenced by the presence of X-outliers, but their identification depends on the employed method. However, this improvement in the model performance, caused by the removal of outliers, comes with a certain cost, namely the loss of molecules from the dataset (cf. red curves in Figure 5) and, consequently, a modification of the AD of the models. Note that the removal of the X-outliers also reduces overfitting (i.e., observed as a decrease in the difference between the training and test performances), but again only for _iForest_ and _RF confidence_ methods. Indeed, on the one hand, if an outlier is present in the training set, the regression model will extend in order to include it, thus lowering its generalization ability. On the other hand, if an outlier is present in the test set, the trained model will most probably perform poorly in predicting its property, as it will not have been trained on similar descriptor values. Finally, the tSNE 2D representation and the selected anomaly score seem to be inefficient in identifying outliers whose elimination will lead to an improvement in the model performance. Indeed, the distance to the k-th nearest neighbors is probably not explicit enough as a metric to identify such points, given also the observed differences in the property values of neighboring points (i.e., property cliffs). Besides, the tSNE 2D representation displays some limits. These observations and the limits of some of the employed methods, such as tSNE 2D, are further discussed in Sections 4.2 and 4.3.