Authors: Sergey Kucheryavskiy | Oxana Ye. Rodionova | Alexey L. Pomerantsev
Source: VBN Forskningsportal (Aalborg Universitet) (article, 2023-01-01)
Field: Chemistry — Analytical Chemistry

Unblind cross-validation Validation is one of the most important steps in chemometric modelling, which can drastically change the final results and the performance of a model, especially when it is being applied for the prediction of new data. The best solution for validation of a final model with optimized hyperparameters is to use a validation set – an independent set of measurements comprising the same population as the calibration set.<br/><br/>However, validation is also required for the optimization of the model, for selection of the best set of preprocessing methods and for the selection of the most important variables. In this case, the well-established solution is to use cross-validation, which is based on iterative resampling. However, cross-validation has several drawbacks. It does not actually validate the model of interest. The cross-validation outcomes (e.g. predicted values) are computed for a set of local models, which are built using initial calibration subsets. Another issue is how to split the objects from the calibration set into segments. Many different techniques exist, however, it is difficult to prove which one is correct in a particular case, as the only outcome cross-validation provides are predictions. Finally, conventional cross-validation can be time-consuming, especially in the case of large datasets, as the resampling procedure must be repeated over and over until the optimization is done.<br/><br/>These drawbacks can be overcome by using a recently proposed Procrustes Cross- Validation (PCV), which takes the best of the two methods – validation with an independent set and cross-validation [1, 2]. The main idea of PCV is to use cross- validation for estimation of sampling error and then introduce this error into the calibration set, which results in a new set of measurements – PV-set. PV-set can then be used for validation of the global models in the same way as the independent validation set. It also provides several additional tools that help to assess the PV-set quality and find the best splitting strategy. One could say that PCV is the way to "unblind" conventional cross-validation.<br/><br/>In this presentation, we will show the main PCV principles and how it can be used for validation of a wide range of models, including PCA/SIMCA, PCR, and PLS regression. We will also present various diagnostic tools that can be used to assess the quality of the PV-set and optimize the splitting strategy of the calibration set. Finally, we will propose a new splitting strategy that was developed with the help of these tools.<br/><br/>References<br/>1. Kucheryavskiy S, Zhilin S, Rodionova O, Pomerantsev O, (2020) Procrustes Cross- Validation—A Bridge between Cross-Validation and Independent Validation Sets. Analytical Chemistry, 92 11842–11850.<br/>2. Kucheryavskiy S, Rodionova O, Pomerantsev O, (2023) Procrustes cross-validation of multivariate regression models. Analytica Chimica Acta, 1255 Cross-validation ; Computer science ; Artificial intelligence Spectroscopy and Chemometric Analyses ; Soil Geostatistics and Mapping ; Statistical and numerical algorithms