# Multivariate calibration of non-destructive spectral sensors with a particular focus on food applications: Validation issues and guidelines

Tom Fearn, Claudia Beleites, Juan Antonio Fernández Pierna, Vincent Baeten, Martin Lagerholm, Jean-Michel Roger, Anastasios Koidis

<figure><img src="image_1.png" /></figure>

## To cite this version:

Tom Fearn, Claudia Beleites, Juan Antonio Fernández Pierna, Vincent Baeten, Martin Lagerholm, et al.. Multivariate calibration of non-destructive spectral sensors with a particular focus on food applications: Validation issues and guidelines. Trends in Analytical Chemistry, 2025, 192, pp.118410. 10.1016/j.trac.2025.118410. hal-05280300

HAL Id: hal-05280300

https://hal.inrae.fr/hal-05280300v1

Submitted on 24 Sep 2025

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_3.png" /></figure>

# Multivariate calibration of non-destructive spectral sensors with a particular focus on food applications: Validation issues and guidelines

Tom Fearn<sup>a,*</sup>, Claudia Beleites<sup>b,c</sup>, Juan Antonio Fernández Pierna<sup>d</sup>, Vincent Baeten<sup>d</sup>, Martin Lagerholm<sup>e</sup>, Jean-Michel Roger<sup>f,g</sup>, Anastasios Koidis<sup>h</sup>

## ARTICLE INFO

### Keywords:

Chemometrics

Multivariate calibration

Tuning

Validation

Cross-validation

## ABSTRACT

Multivariate calibration methods have enabled the use of non-destructive spectral sensors in a wide range of applications but carry a risk of overfitting to the available training samples. For this reason, the prediction of unseen samples plays a vital role both in tuning the prediction algorithm and in assessing its performance, two activities that need to be carefully distinguished. Methods employed include data-splitting, cross-validation, and the use of genuinely independent sets of data. These approaches are described and some common issues with them are identified. The focus is on food applications but the methods discussed are widely used in other areas.

## 1. Introduction

Non-destructive spectral sensors (NDSS) such as near infrared (NIR), Raman and hyperspectral imaging make indirect measurements of the properties of interest in foods, based on the structural and chemical information that is contained in their spectra due to light-matter interactions. These measurements need to be calibrated to enable determination of the food properties. This calibration is typically carried out by assembling a training set of samples and measuring for each sample its spectrum using the NDSS and the property of interest using a laboratory reference method. Then a multivariate calibration method such as partial least squares regression (PLSR) [1,2] is used to model the relationship between spectrum and property of interest and thus derive a rule that predicts reference measurements from spectral measurements for the training samples. Other popular methods include support vector machines [3] and, more recently, deep learning [4]. In the case of PLSR, the prediction rule is a relatively simple linear equation, but for other methods it may be a more complex computation for converting the

chemical information in a spectrum into a prediction of a property of the food sample on which the spectrum was measured. The property to be predicted may be quantitative, such as the amount of protein in a sample of wheat flour, or qualitative, such as the variety of coffee bean in the sample, robusta or arabica. In the qualitative case the reference data will often be based on trust in the source of the sample rather than a laboratory measurement.

Multivariate calibration is a powerful methodology that has enabled the successful use of NDSS in a wide range of applications [5], but it needs to be used with care. All multivariate calibration methods are prone to overfitting. This means producing a rule that is too finely tuned to the training samples and which will work much less well when applied to other samples [2]. The risk of overfitting increases as the number of variables in the measured spectrum increases. It also increases as more sophisticated and flexible calibration algorithms are employed. It decreases as the number of independent samples in the training set increases. Rules for classification are particularly prone to overfitting. The performance of the rule can be strongly influenced by a

<footnote><sup>a</sup> Department of Statistical Science, University College London, Gower Street, London, WC1E 6BT, UK</footnote>

<footnote><sup>b</sup> Chemometrix GmbH, Södeler Weg 19, 61200, Wölfersheim, Germany</footnote>

<footnote><sup>c</sup> Julius Kühn-Institute, Institute for Ecological Chemistry, Plant Analysis and Stored Product Protection (ÖPV), Königin-Luise-Str. 19, 04195, Berlin, Germany</footnote>

<footnote><sup>d</sup> Walloon Agricultural Research Centre, Knowledge and Valorisation of Agricultural Products Department, Quality and Authentication of Agricultural Products Unit, Henseval Building, 24 Chaussée de Namur, 5030 Gembloux, Belgium</footnote>

<footnote><sup>e</sup> Perten Instruments AB, P.O. Box 9006, SE-129-09, Hägersten, Sweden</footnote>

<footnote><sup>f</sup> ITAP INRAE, Institut Agro, University Montpellier, Montpellier, 34196, France</footnote>

<footnote><sup>g</sup> ChemHouse Research Group, Montpellier, 34196, France</footnote>

<footnote><sup>h</sup> Institute for Global Food Security, School of Biological Sciences, Queen's University Belfast, Belfast, UK</footnote>

<footnote>This article is part of a special issue entitled: Chemometrics published in Trends in Analytical Chemistry.</footnote>

<footnote>* Corresponding author.
E-mail address: t.fearn@ucl.ac.uk (T. Fearn).</footnote>
relatively small number of samples near the boundaries of the classes being separated. A sophisticated algorithm can adapt to these samples in the training set if it is allowed to. Thus, although the overall sample set may appear to be large, it may be a much smaller subset that is driving the calibration.

However, in most NDSS calibration scenarios it is rare to be able to obtain enough calibration samples to avoid the risk of overfitting. The way to protect against this risk is to assess rules by using them to predict unseen samples. These are samples that were not involved in the training of the rules, nor closely related to any of samples that were involved, for example by being another subsample from the same jar. The prediction of unseen samples is used at two stages of the development and evaluation of a calibration model. The first stage, which will be called tuning here, involves the optimisation of the tuning parameters that control the calibration algorithm and of other choices that are made as part of the calibration. In the case of PLSR the most obvious tuning parameter is the number of factors to be used, though the choices of the wavelength range to be used and the spectral pre-treatment to be applied are also part of the tuning and may be just as important. The second stage, referred to here as validation, involves the assessment of the performance of the final tuned rule. Although some of the ideas described in what follows are common to both tuning and validation, it is important to maintain the distinction between the two.

The term validation is used with different meanings in chemometrics and in the broader area of analytical chemistry. In chemometrics, validation usually means the assessment of the performance of a multivariable calibration via the prediction of a set of samples not used in the training. This procedure would usually be called verification by an analytical chemist. In analytical chemistry generally, the validation of an analytical method would involve much more than simply an assessment of performance on a set of “nice” samples. It would investigate linearity, selectivity, and ruggedness for example [6]. In this review, the term validation is used in the chemometric sense. The term verification will not be used again. The paper by Fernandez Pierna, Dardenne and Baeten [7] describes the validation of an NIR imaging method for detecting animal proteins in animal feed which goes much further than a typical chemometric validation.

Similar differences exist in the terminology of what constitutes internal and external validation between chemometrics on the one hand and analytical and clinical chemistry on the other hand. In the general analytical and clinical chemistry terminology, an external validation is performed by an independent third party on samples they ensure to be independent. Any validation procedures carried out by the same lab that developed or implemented the analytical method would be considered internal. In contrast, in chemometrics sometimes less emphasis is put on this organizational method of ensuring independence, and “external validation” is commonly used to describe results obtained on a single dataset split off from a larger data set that was available for rule development, see for example [8] or [9].

The aim of this review is to describe the current state of the methodology for validation of NDSS calibrations, review some of the literature, and provide some guidelines for users. As a review it is somewhat less than comprehensive, as the focus is on important issues only. Its main contributions are to bring together these issues, discuss them critically, highlight problems with current practice, and conclude with some basic guidelines. While the basic principles for proper validation have been known for decades, a recent review [8] reveals that these principles are widely ignored.

Much of the discussion below applies equally to both quantitative and qualitative calibrations, though particular issues associated with the qualitative case will be highlighted at some points. The focus throughout is on tuning and validating calibration models for a particular application using a given set of data. This is a different problem from trying to assess whether one algorithm is in general better than another [10].

## 2. Tuning and validation

Unfortunately, there is no common agreement on nomenclature. The names training, tuning, and test will be used consistently throughout this review for the three data sets involved in training, tuning, and assessing a prediction rule respectively. The use of “test set” to describe the set used to assess performance is in line with the terminology of ISO standard EN12099 [11]. The alternative label “validation set” for the third set has been avoided here because it is used by some authors to denote the data used for optimising the rule and by others to denote the data used to assess its performance. This inconsistency in nomenclature may be one reason why distinction between the two functions is not always properly appreciated.

### 2.1. Tuning

Tuning is an intrinsic part of the development of the rule. The fact that during this process the rule is optimised to perform well on the unseen samples means that by the end of the tuning the samples have stopped being unseen ones. The performance statistics of the final tuned rule on these samples will be over-optimistic. The extent of this over-optimism will depend on the complexity of the algorithm used, and hence its ability to adapt to these samples during the tuning, and on the number of different options investigated during tuning. The root of the problem is that the statistic that is optimised during the tuning is, like any quantity calculated from data, subject to error. Doing the data splitting correctly, as discussed below, will avoid systematic biases in the performance statistic, but it is still subject to random error. This is true for any performance statistic, but for one based on a count of successful classifications the random error can be particularly large [12]. In the classification context the use of proper scoring rules can help somewhat, see Ref. [13] for an example of their use, because their values change more smoothly, but the underlying problem remains. The systematic and random error behaviour of various validation schemes have been studied in a general machine learning context by Kohavi [14], Dougherty [15] and Kim [16], while Beleites [17] focuses on data shapes common in spectroscopic applications.

A tuning regime that compares many options and selects the apparently best one will exploit this random error to achieve an apparent performance that will not be repeatable on other samples [18].

To get an unbiased estimate of likely future performance of the final rule, it is necessary to predict a further set of unseen samples, the test set. Here the temptation to carry forward several rules, for example rules based on a handful of spectral pre-treatments, to this stage and pick the best according to performance on the test set should be resisted. Doing this would mean that the test set would effectively be acting as a tuning set and the performance statistics will be over-optimistic. A recent review of 200 publications applying NIR spectroscopy to food [8] reported that in at least 15 % of the papers the same data were used both to tune the algorithm and to assess its performance. The real figure is almost certainly higher than this: for another 25 % of the surveyed papers, the description of the methodology was inadequate so independence of the test data could not be evaluated.

In the context of cross-validation, splitting into separate sets for tuning and testing is known as nested cross-validation [19]. In the chemometric literature, the same procedure has also been termed double cross-validation [20]. Nowadays, this procedure is mostly discussed as splitting a given data set into three partitions for training, tuning and testing, respectively. Judged by the number of papers reported by Ref. [8] that do not implement this separation between testing and tuning correctly, there is still considerable confusion about the techniques almost two decades after they were introduced. Fig. 1 (a) and (b) attempts to clarify the situation.

The confusion may be rooted in the indiscriminate use of the term training for both a low-level and a high-level step involved. High-level training includes any tuning, while low-level training comprises only
<figure><img src="image_4.png" /><figcaption>**Fig. 1.** a) Data flow and splitting for tuning and validation: The available data is input to the testing procedure. The testing procedure splits it into a (high-level) training and a test set. The (high-level) training data is then handed over to the high-level training procedure, which in turn splits that data set into low-level training and tuning sets. The low-level training procedure fits models (triangles) from its (low-level) training data and the hyperparameters, and hands the models back to the high-level training procedure. The high-level training procedure uses the tuning data to determine which of the tentative models is the best, and outputs this (large, dark blue triangle). Now, the testing procedure applies this model to the test data and outputs the observed performance as result. The scopes of the three different procedures help ensure that no data leakage occurs due to inadequate access to data that needs to be kept separately: each procedure has access only to its own data, which entered its scope along the solid arrows. The splitting may be done in a repeated fashion. I.e., while for the sake of clarity the illustration depicts single splits, repeated splitting procedures such as repeated cross-validation may be used. The governing principles for correct splitting are the same. Also, the two splitting procedures are independent of each other: the testing procedure may use one particular set-up for its splitting, while the high-level training function may use a different setting. b) The common recommendation to split into independent training, tuning and test data produces (vertical) corresponding data sets, but without the scopes it is less clear how to practically implement the splitting correctly. c) The final model trained from the whole data set (black), by applying the same high-level training procedure as in a). The performance estimate from b) which is really obtained on the model within the testing procedure scope (medium-sized dark-blue triangle) is taken as approximation for the performance of the actual final model (large black triangle). Illustration reproduced with permission by Chemometrix GmbH. (For interpretation of the references to colour in this figure legend, the reader is referred to the Web version of this article.)</figcaption></figure>

the fitting of a rule to given training data with specified hyper-parameters. High-level training will usually employ low-level training on a variety of different options e.g. for pre-processing, complexity, etc. These options are formalized as so-called hyperparameters. The high-level training procedure then selects the optimal option and returns the corresponding rule. In order to determine which option is best, such a high-level training method will usually evaluate performance of the tentative rules, and in order to do so, it may split off a tuning set from the data it received. Thus, we now have a high-level training function that returns a ready-to use rule. The performance of the high-level training function and the rule it generates may be tested in the usual way. In the case of a calibration that is to be used, rather than being simply a feasibility study, there is another important distinction to be made between tuning and validation, as discussed below.

## 2.2. Validation

Validation is the step at which the robustness of the calibration needs to be established. For tuning purposes, the approach usually adopted is to use various methods of splitting the training data to provide the unseen samples. In many published papers the same data-splitting methods are used to provide a test set. However, for validation in a situation in which one needs to be confident that the calibration is a robust one, a much more challenging, genuinely independent, set of unseen samples is desirable. A split of a single set of data, especially if the samples were obtained from a limited number of sources and measured at the same time under well-controlled conditions, will always result in a test set containing samples and spectra that are in many ways very similar to the training samples and their spectra. Future samples will be less similar, and the calibration should be challenged with some of these to see how robust it is. The review in Ref. [8] found that only around 5 % of the papers studied used genuinely independent test sets rather than some
form of data splitting.

In the case of quantitative calibrations it is important to include samples that lie near to the classification boundaries, i.e. ones that will be harder to classify. This is especially true for so-called one-class classification rules, where the result of a prediction is that the sample is declared to be either a member or not a member of the class of interest. If the class of interest is Parmigiano cheese, then the calibration needs to be challenged with Padano, not just with Cheddar.

Not only should the ability of the calibration to make good predictions for samples that fall within its declared scope be tested, but also its ability to refuse to make predictions for samples that are quite different from the samples on which it was trained. Screening procedures of this type typically involve comparing the spectrum of the sample to be predicted with those of the samples in the training set and flagging a problem if it is too far away. In the case of PLSR the x-residual may be used for this purpose [21]. To test such a procedure, it should be challenged with samples of types that it may accidentally encounter when in routine use, for example other raw materials in the same production system. This is particularly important when the scope of the calibration is narrow, protein in plain white flour for example, and there are other similar products that might accidently be presented to the instrument, brown flour for example. An obvious first step might be to predict any samples that were removed as outliers when the calibration was being developed: are the predictions satisfactory, and if not, are they rejected by the screening algorithm?

### 3. Data splitting for tuning or validation

As remarked above, enabling the prediction of unseen samples by splitting the set of training samples is commonly used for both tuning and validation, though for a serious attempt at validation this is very much a second-best option. For simplicity we first describe the methods as they would be used to produce just one set of unseen samples. The concepts for properly splitting data once are the same that need to be obeyed also when splitting multiple times. Or, to put it the other way round: the mistakes one can make during splitting which cause data leakage and in consequence overoptimistically biased performance estimates are the same whether a single split is performed or repeated splits as in cross-validation.

#### 3.1. A single split of a data set into two parts

The simplest approach is to split the set of available training samples once, giving a training set and a tuning set for example. The rule is trained on the training set, using its performance on the samples in the tuning set to guide the optimisation of the algorithm. There is no universally accepted rule for what proportions to use, though splits in the region of 70:30, with the training set being the larger one, are common [8]. When there are plenty of samples available the proportions of the split do not really matter too much. For performance estimation it is the absolute size of the split-off set that becomes important [12], particularly when samples are scarce. With, say, 40 samples, the problem with splitting is that a tuning set with 10 or so samples would give a very unreliable estimate of performance, especially in the case of a qualitative calibration where each extra success corresponds to a 10 % improvement in estimated accuracy. Thus, the performance estimate is subject to high random uncertainty (variance). This consideration applies regardless of whether the purpose for the performance estimate is tuning or validation. For tuning, the risk of overfitting (to the split off set as well) increases, see Section 2.1. For verification, the resulting large uncertainty will be unacceptable since it precludes judging the fitness for purpose. On the other hand a larger tuning set would leave an unsatisfactory remainder for training. Such small training sets should be avoided if possible, but if they have to be used it is better to use cross-validation to repeatedly split off a few samples, as described below, rather than a single split. It could be argued that cross-validation

is a better option than a single split more generally, as it will give a performance estimate with lower variance. However a single split is simpler, and may be preferred when the number of available samples is large.

#### 3.1.1. Respecting data structures in the split

Arguably more important than the exact proportion of any split is the method of splitting. If overfitting is to be avoided the samples in the tuning set need to be genuinely different to those in the training set. This means that the split should take account of any structure in the overall sample set. A simple example of such a structure would be when the overall set, of size 150, is made up of three replicate subsamples from each of 50 larger samples. Then the temptation to make the split by putting two subsamples from each set of three in the training set and the other one in the tuning set should be resisted, strongly. Each sample in the tuning set would then have two very close relatives in the training set, so that the rule, though it may not have seen the one to be predicted, will have seen a two very similar samples. A random split should also be avoided in this situation, because this would still result in many of the replicates being split across the two sets. A better strategy would be to split at the level of the 50 larger samples, keeping each set of three subsamples in one or other of the two sets. This split could be done randomly, assuming there is no other structure to be taken account of. Lopez et al. [9] analyse three examples with replicate spectra where not splitting properly leads to highly misleading results.

More complicated structures are not uncommon. For example, the available samples might comprise 10 apples from each of 5 trees from each of 5 orchards. The higher up such a nested structure the data split is made, the more the rule is made to predict samples genuinely different from those it was trained on. Putting three orchards in the training set and two in the tuning set will probably lead to worse performance statistics, but it is more likely to result in a calibration that is robust enough to predict samples from other orchards in future, and also in a realistic estimate of the performance that one can expect when using this calibration in other orchards. At the other extreme, a random split at the level of apples will mean that during tuning the rule is mostly predicting apples for which one or more apples from the same tree were seen during training.

Another example of structure is where the available samples are artificial mixtures of two constituents, involving 5 independent examples of each constituent. Then the most challenging, and therefore most desirable, split would be to separate off 2 of the examples of each constituent and use only combinations of these in the tuning set.

If the aim of the is to produce a robust calibration, any hierarchical structure in the samples should in general be used to make the tuning set harder for the rule to predict, not easier. There is a more detailed discussion of rules for correctly splitting data, with an emphasis on the case of applications to food and agriculture in another paper in this issue [22]. Westad and Marini [23] also discuss the choice of level at which to split data in a paper that has much useful guidance on validation in general.

In the case of qualitative calibrations it is generally desirable to carry out any splitting in a way that maintains the proportions of samples in the various classes at least roughly equal in each of the sets. Many figures of merit, such as overall accuracy, will depend on these proportions, and having substantially different mixes at different stages of the tuning and testing procedure can lead to apparently inconsistent results. Alternatively, the observed figures of merit can be rescaled to match the desired proportions of the different classes. Such rescaling becomes necessary if the test set cannot be matched with the expected proportions (e.g., prevalence or incidence) during production use, or putting it more generally: if the test data distribution does not match the data distribution under real-world production use. This issue is also well-known in the medical context, see e.g. Ref. [24] for a discussion on the impact misleading validation has had on cancer diagnostics.

Defining the scope of the calibration at the outset not only informs
the selection of training and validation samples but should also guide any data splitting. In the apples example above there may be no need to split at the level of orchards if these 5 orchards will be the future universe for application of the calibration. Even then though, the more robust calibration that will result from splitting at the top level may be more stable over time. Examples where the whole population of interest would be known and available at the time of calibration do exist, see Ref. [25] for an example involving plant breeding, but they are rare. Most spectroscopic calibrations will be used for the prediction of future samples whose circumstances are less well known.

One important consequence of such data structure is that when correctly splitting according to the structure, the splitting options may become very restricted, e.g. splitting off one or two of the five orchards.

Two practical recommendations follow:

1. Application requirements need to be specified in sufficient detail to establish which data structures are important. E.g., is the method to be used for unknown orchards, or only within the set of orchards available during rule development?

2. Report which influencing factors or structures in the data were taken into account and how independence was ensured. E.g., data was split into training and testing orchards.

### 3.1.2. Avoiding data leakage due to pre-processing

It is important that the split-off set to be used for tuning or testing is kept separate from the training set. A problem sometimes called data leakage will occur if data processing before the split transfers information between the two sets. The simplest example of this is the mean centring of the columns of data, i.e. mean centring of spectra at each individual wavelength, which needs to use a mean that is calculated from the training portion of the data only and is then applied to both the training portion and the left-out portion of the data. A more serious violation of the need to keep processing separate would be to carry out a principal component analysis (PCA) on the full data set and then split it. As with mean centring, the PCA must be applied to the training set only and the scores for the split-off samples calculated using the loadings from this PCA. The rule is simple: split first and process afterwards [19].

### 3.1.3. Kennard-Stone and similar systematic sample selection algorithms

There exist several algorithms for systematically selecting a subset of samples from a larger set based on their spectra [26–28]. The one due to Kennard and Stone [29] is probably the most popular in chemometric applications. This algorithm starts by finding the pair of samples whose spectra are farthest apart, then at each further step adds the sample whose spectrum is furthest from those already selected. It stops when the desired number of samples has been selected. The result of this, and of the alternative algorithms, is a subset of samples whose spectra are uniformly distributed over the high-dimensional space in which they lie. This subset will typically include the spectra on the boundary of the space. The paper on sampling [22] in this issue discusses Kennard-Stone as well as other related algorithms.

The scenario in which these algorithms have their greatest value arises when it is feasible to measure spectra on a large number of potential training samples but desirable to carry out reference measurements on only a subset. One reason for this would be to save on the cost of the reference chemistry. Another arises in some plant breeding situations when the whole population of interest is available for scanning, but the proportion subjected to a destructive reference analysis needs to be minimised. In both cases the reference measurements can be carried out only on the reduced number of selected samples.

However, there is another common application of sample selection algorithms of this type which is less than ideal, although it is suggested in the early literature (see Ref. [28] for example) as appropriate. They are often used to perform a split into training and tuning or training and test sets. The selected training set will indeed be a good one, in that it will have an even spread and include the extremes of the range of

interest [21]. The unselected samples on the other hand will exclude these extremes and tend to have more density in the centre of the distribution than the original set. Using them as a tuning or test set makes the prediction of the unseen samples a little too easy. Imagine a simple case with one predictor variable x and one dependent variable y and five data points with equally spaced x-values. Using Kennard-Stone to split this into a training set of three samples and a test set of two samples would put the two extreme points plus the central one into the training set and the remaining two into the test set. This test set is inside the training set and predicting it is not a particularly stiff test for the calibration. The review in Ref. [8] found roughly 15 % of the surveyed papers employing Kennard-Stone, SPXY, Duplex or Onion algorithms for the testing, all of which share this problem when the selected samples are used for training and the remainder for tuning or testing. In high dimensions the situation is more complicated, but the test samples will still tend to be in the interior of the training set. This is probably a relatively small effect unless there is some structure in the data set that is being ignored. For example, in the extreme case of three replicate spectra on each sample, naive use of Kennard-Stone to make a 2:1 split of the samples into training and test sets would probably get quite close to putting two from each set of three replicates in the training set and the other in the test set. It is of course easy to modify such an algorithm to avoid serious problems like this data leakage by making it respect any structure.

The fundamental problem with using these approaches for data splitting is that the design of the tuning or test set, which is arguably as important as that of the training set, is not being given adequate consideration. A test set from which the tails of the natural distribution have been removed in the interests of optimising the training set is not a good test set. Essentially, this is another variant of the problem discussed earlier for classification: the test data distribution not matching the data distribution encountered in application-use.

## 3.2. Multiple splits of a data set into two parts

### 3.2.1. Cross-validation

A more commonly used approach to tuning is to employ cross-validation [30]. This makes multiple splits of the data in such a way that each of the N samples in the training set is predicted exactly once. In K-fold cross-validation the training samples are divided into K blocks of approximately equal size. Then each of the blocks in turn is removed from the training set and predicted by a calibration developed using the remaining K-1 blocks. This method makes very efficient use of the available samples. The only real drawback is the extra computational cost of fitting K calibrations.

In addition to the fact that using $K < N$ requires less computation, there is a further intuitive argument for not using the obvious choice of $K=N$, i.e. leave-one-out. The larger the number of samples in each left-out block, the stronger will be the protection against overfitting. Tuning using cross-validation rewards calibrations that are robust to perturbations of the training data. The more samples that are removed at each iteration, the bigger is the perturbation, thus one might expect leaving out larger blocks to lead to the selection of more robust calibrations [31]. On the other hand, if you remove $N/K$ of the N training samples at each step, the calibrations being optimised are fitted to training sets of size $N(K-1)/K$. With a large training set this may not matter very much, with a small one the calibration might be quite sensitive to losing the information in some of the training samples. Thus, the choice of K is a matter of balancing the desire for robustness against the risk of over-reducing the training set.

A more statistical version of the above argument presents this as a balance between the variance and bias of the performance measure computed using cross-validation. It is generally thought by no means universally accepted that the variance reduces as the left-out blocks get larger, i.e. as the number of blocks K gets smaller. Conversely, the bias resulting from calibrating on smaller training sets gets larger as K gets
smaller, hence the need to balance the two effects. There are some
theoretical arguments in the literature supporting various choices, but
none of them seems entirely convincing. Empirical evidence suggests
that choices of 5 or 10 for K represent a good compromise in most sit-
uations [32,33]. With a large training set, 1000 samples say, the use of 5
blocks still leaves 800 samples in each reduced training set and saves on
computational effort. With a much smaller training set one might want
to use more blocks. With only 30 samples, not something to be
encouraged but sometimes unavoidable, cross-validation leaving out
small numbers of samples at each iteration is the obvious choice. In this
situation repeated random cross-validation as described below might
also be considered as an option.

Most software offers multiple choices for how the split into blocks is
done. Commonly available options are random splits or systematic splits
using either contiguous blocks, e.g. the first block is the first 20 samples,
the second is the next 20 and so on, or so-called venetian blinds, where
the first of 5 blocks comprises samples 1, 6, 11, ..., and the second block
is made up of samples 2, 7, 12, ... A single random split into blocks that is
different each time the program runs can have the annoying effect that
results keep changing slightly, though multiple random splits, as dis-
cussed in Section 3.2.2, can provide valuable information about the
uncertainty in the results. There seems little to choose between the
various systematic versions, with the proviso that, as with a split into
two sets, it is important to consider any structure in the training data set.
If there is structure it may be necessary to define the blocks so that, for
example, a block corresponds to all the samples from one orchard or one
tree. Some, but unfortunately not all, software will allow the user to
define the blocks. Even when there is no obvious structure, the ordering
of the samples in the training set may be informative. For example, they
may be arranged in time order of measurement. In this case the use of
venetian blinds, which would ensure that each left-out block had a
measurement time structure matching that of those used for training,
should be avoided.

One mistake that is quite often made with cross-validation is to forget
that any data treatments that work on more than one row of the data
matrix, i.e. one spectrum, at a time need to be recalculated with each
split. This is the data leakage problem described in Section 3.1.2. Any
data processing that involves more than one sample at a time needs to be
done inside the cross-validation loop, not outside it. As before, the
simplest example of this is mean centring of the columns of data [34].

3.2.2. *Cross-validation using repeated random splits*

Some authors recommend repeated random splits of the data, i.e.
carrying out M repetitions of a K fold random cross-validation [35]. The
first advantage of this method is that it avoids the risk of an unlucky
single split. Secondly, it provides several different cross-validation re-
sults, giving an idea of the variation and uncertainty in the computed
performance measure. Not only is each sample predicted many times but
it is predicted by a different model each time, because the training set
varies with each random split. This allows the variance due to model
instability to be estimated [17]. See Refs. [36,37] for further discussion
of the uncertainty of cross-validation results.

In addition, this technique makes it possible to use a low value of K, i.
e. a high proportion of test samples, e.g. 50 % in the case of K = 2, and to
reduce the effect of chance inherent in drawing a small number of
blocks. Last but not least, this method can be used to test a large number
of calibration/test splits with a small number of samples. For example,
with 30 samples, a K = 2, M = 30 repeated cross-validation enables 60
configurations to be explored.

3.3. *Creating sets for tuning and validation*

In practice, two sets of unseen samples are needed, one for tuning
and one for validation. The usual method of achieving this by data
splitting is to split off one set for validation and then use cross-validation
on the remaining samples for tuning. This avoids ending up with a

training set of substantially reduced size. Anselmo et al. [38] use a
scheme of this type in a study of the use of vibrational spectroscopy to
measure insect meal in compound feed. If there are not enough samples
to split off even one set, there are basically two options. One is to use a
nested cross-validation [30] in which an outer loop leaves out a
sequence of blocks, each of which is predicted by a rule tuned by
cross-validation on the remaining samples as described in Section 3.2.1.
Iqbal et al. [39] use a scheme in which the outer loop comprises three
batches of samples, each left out in turn. The issues with this approach
are computation time and the fact that if the outer loop has more than a
very few blocks the tuning will need to be completely automated
because it will happen inside this loop.

The other option is to postpone the validation. Start by using cross-
validation for tuning and trying to avoid overfitting by employing
simple algorithms, using whatever spectroscopic knowledge is available
to avoid searching over too many options, and not aggressively chasing
small improvements in performance during the tuning process. If the
result is a calibration that seems promising it will still need to be vali-
dated before one could risk using it, and this limitation needs to be
stated unambiguously.

4. The final calibration

If the development of the calibration has involved data splitting, it
makes sense to put the data back together and produce the final working
calibration using all the samples. Reassembling the data is particularly
important if it has been split at a high level of some hierarchy, orchards
in the earlier example say. Separating off one or two orchards is a good
way to assess the generalisability of the calibration, but having done
this, one would want to use all the orchards at the final step. A question
that does not have a simple answer is how to choose the hyper-
parameters for this final calibration. Imagine, for example, a nested
cross-validation leaving out one of the five orchards at a time in the
outer loop and then tuning by cross-validation in the inner one. The
optimal parameter choices may well differ depending on which orchard
is left out. If they differ a lot, this may be a warning sign that the
modelling is unstable. If they differ only a little, one has two choices.
One is to use compromise values for the hyperparameters in the final
calibration. This is arguably the best option if robustness has been shown
and it is desired to retain this. For example, if the optimal number of PLS
factors only varies by one or two between different splits using the
higher number may be judged reasonable. The other, more relevant to
the case in which predicting a left-out orchard has led to poor results,
would be to tune at least some of the parameters again with a lower-level
split. If extrapolation to unseen orchards has been shown to be risky,
obtaining the best calibration for the ones we have may be the best
option. With PLS this is likely to result in the use of more factors as the
differences between orchards are modelled. This is not a problem so long
as the calibration is accompanied by a warning about the dangers of
extrapolation to unknown orchards.

In case the tuning cross validation is pushed into a separate high-
level training procedure (Fig. 1 a), this same high-level training pro-
cedure can be applied to the whole data set (Fig. 1 c). Again, if that final
model ends up differing much from the internal models of the testing,
that is a serious warning that the training (possibly the tuning) is not
stable. Such warning signs need to be taken seriously since they indicate
situations in which also the tuning has problems.

The metadata accompanying the calibration should include details of
how the validation was carried out and any implications for its scope.
For example, if leaving out an orchard led to poor predictions for that
orchard, a user should be very cautious about using it on orchards other
than the ones in the training set without carefully checking its perfor-
mance on them first. On the other hand, if it worked well on unseen
orchards, the user might be rather more confident as regards extrapo-
lation, though they would still be advised to check. The ISO standard
EN12099 [11] and the paper by Williams, Dardenne and Flinn [40]
contain recommendations for metadata that it is important to associate with a calibration.

## 5. The need for rigorous validation

There is an important distinction to be made between spectroscopic calibration exercises that are no more than a feasibility study showing that a spectroscopic method has the potential to solve a particular problem and those in which calibrations are developed for actual use. Most of the published literature describes exercises of the former type and many of them are of very limited value. The researchers have typically taken every care to make sure that the test set looks as much like the training set as possible, thus guaranteeing that the calibration does not have to face any unexpected challenges when it is validated, as it certainly would do if used routinely for any length of time. There is a place for reporting studies of limited scope, so long as that scope is clearly spelled out in the report, but there is a danger that this is seen as standard practice by newcomers to the field. It would be good to see more published examples of rigorous validation.

If you plan to use the calibration in a situation where mistakes could be costly, the validation needs to be much more rigorous than this.

As a first step it is reasonable to use a split-off test set, because if the performance is not adequate on this it is not going to improve when faced with tougher challenges. However, splitting a data set provides many opportunities for the test samples to resemble the training samples, even when the obvious problems described in Section 3.1 are avoided. For example, all the samples may have had their spectra measured by the same operator under the same conditions and over a relatively short period of time.

If the calibration does appear to work on a split-off test set, then it should be tested on a set with genuinely independent samples, acquired and measured at a different time to the training samples, and chosen to represent as much as possible of the variability in samples that the calibration is likely to have to cope with.

If there are known factors that may interfere with the calibration, sample temperature for example, a specific test of robustness to these factors is also desirable.

Papers that nicely demonstrate the difference between “pseudo-independent” validation using random splits and genuinely independent validation using samples measured later in time than the training samples include Bobelyn et al. [41], Diaz-Olivares et al. [42] and Fonseca-Diaz et al. [43]. The applications are to apples and milk.

## 6. Ongoing monitoring

However carefully the validation was performed, there is always a risk that, with time, changes in the instrument, the measurement environment, or the type of sample being measured will lead to errors in the measurement. Regular monitoring of the calibration performance should be carried out, by comparing the instrument's measurements with reference measurements on a small number of samples. Regular monitoring of the quality of analytical data, for example by participating in a proficiency testing scheme [44], is not only good practice but is required in some application fields. Although there is extensive literature on monitoring by spectroscopic instruments, there appears to be very little on the monitoring of the instruments themselves. An early book on the application of NIR to food analysis [45] suggests monitoring the performance of a calibration using a control chart. At regular intervals, weekly say, a small number of samples are analysed both by the instrument and by the reference method and the mean difference plotted on the chart. The construction of such charts in general will be covered in almost any book on statistical quality control [46], for example. Their particular application to monitoring spectroscopic calibrations is an area which perhaps warrants further investigation. Another more recent paper recommending the use of control charts is [7]. In the case of a known event that may cause problems, for example a change to some

component of the instrument, or a need to expand the scope of the calibration to cover types of samples that may not have been seen before, a check using a rather larger number of samples will be necessary. In the case where the scope of application of a calibration needs to be expanded to include new types of samples, it is common to add some of these samples to the training set and produce a new calibration. Methods for tackling this problem are described in another paper in this issue [47]. Here we simply note that if a major change is made to a calibration the validation exercise should be repeated. If the reason for the recalibration was to expand the scope of the application, it is important to check that the new calibration still predicts well the samples in the earlier test set. It is not usually possible to save the samples, which typically change over time, but at least the spectra and reference values on these samples can be and should have been saved in a format that enables their later retrieval and use as a supplementary test set for the new calibration.

## 7. Discussion and summary

In this section we discuss the coverage of the existing literature and summarise some of the key rules for validation of multivariate calibrations.

### 7.1. Gaps in the literature

Many of the references cited in this paper are quite old. While current research on the statistics of these methods does produce novel and important insight (e.g. Ref. [37]), the basic principles of validation of multivariate calibrations have been well-known for decades [48]. The main problem in this area is not a lack of established knowledge, but a failure to apply it in too many cases. This tendency is probably reinforced by the number of feasibility studies in the literature, often with modest numbers of samples and minimal validation. There is of course a role for such studies, but the danger is that this is seen as the norm. Another possible cause is that the principles are typically discussed separately from each other with very neat example data (no structure), but rarely is an example of putting everything together for messy real-life data given.

The terminology in the field is confusing (Sections 1 and 2), with several terms being applied in ambiguous fashion. There is no sign in the literature of any resolution of this problem and the reader must continue to be wary.

The fundamental principles for splitting structured data have been known for decades. Yet, it may be possible to arrive at more clear and immediate guidance by applying design of experiments to performance evaluation with structured data. Some first insights are discussed in Ref. [22]. Performance evaluation is known to be subject to systematic and random error like any other measurement. Systematic error has been studied in detail for many evaluation schemes [14,15] and results are nowadays available in textbooks [32]. In contrast, the random uncertainty of cross validation is less well understood, and subject to contemporary study [37]. At least two factors are crucial. Firstly the treatment of the random uncertainty needs to differentiate two different scenarios. One is the comparison of algorithms trained on data of a particular type. The other is the estimation of production-use performance of a particular model trained on the data at hand [10,36]. While the latter is the usual situation in validation of a chemometric model, the machine learning literature concentrates on the former. Secondly, cross validation and similar schemes for performance estimation involve approximating the performance of the final model (Fig. 1 c) from observations for a set of similar models (Fig. 1 a), and the related random uncertainty is far from straightforward.

One type of paper it would be good to see more of in the literature would describe in detail the development of NDSS applications that have been implemented in industrial settings. Perhaps these do not appear because the work has been done in a commercial setting and publication
is seen as either irrelevant or undesirable.

## 7.2. Recommendations

It is difficult to lay down precise rules because the exact strategy adopted for validation of multivariate calibrations is context dependent. However, there are some principles that apply quite generally.

* Using the same samples for tuning a rule and assessing its performance will lead to an over-optimistic assessment and should be avoided.

* The same rules for achieving independence apply for splitting a given data set once or repeatedly, e.g. in cross-validation.

* Splitting a single independent test set off a given data set runs the same risk of data leakage as cross-validation involving repeated such splits.

* Ensuring independence by organizational measures (acquiring the test set after calibration is finished, involving a third party for external evaluation) can mitigate these risks. Testing must account for the production use scenario of the model. E.g., a classification rule must be evaluated for the prevalence or incidence expected under application conditions, results on artificially balanced test data are misleading at best.

* A separately acquired test set allows to directly measure performance for relevant subgroups of samples. E.g., for a rule to detect food adulteration, known, common types food fraud may be directly studied.

* The gold standard for the test set is that it should consist of genuinely independent samples. The use of a split-off set is more common. This is acceptable in the first stages of developing an application but not adequate for a calibration that will be used in practice. In splitting data or constructing cross-validation blocks, any hierarchical structure in the data needs to be taken into account.

* Ask yourself: is this test data in any respect unduly easy to predict? Properly accounting for structure usually makes the predictions harder, not easier.

### CRediT authorship contribution statement

**Tom Fearn:** Writing – review & editing, Writing – original draft. **Claudia Beleites:** Writing – review & editing. **Juan Antonio Fernández Pierna:** Writing – review & editing. **Vincent Baeten:** Writing – review & editing. **Martin Lagerholm:** Writing – review & editing. **Jean-Michel Roger:** Writing – review & editing. **Anastasios Koidis:** Writing – review & editing.

### Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

### Acknowledgements

All the authors acknowledge the SensorFINT project (COST Action No. CA19145). Claudia Beleites acknowledges partial funding by BMBF project In4Food (FKZ 031B0959C).

### Data availability

No data was used for the research described in the article.

### References

[1] S. Wold, M. Sjöström, L. Eriksson, PLS-regression: a basic tool of chemometrics, Chemometr. Intell. Lab. Syst. 58 (2) (2001) 109–130.

[2] H. Martens, T. Næs, Multivariate Calibration, Wiley, Chichester, 1989.

[3] J.A. Fernandez Pierna, V. Baeten, A. Michotte Renier, R.P. Cogdill, P. Dardenne, Combination of support vector machines (SVM) and near infrared (NIR) imaging spectroscopy for the detection of meat and bone meal (MBM) in compound feeds, J. Chemometr. (18) (2004) 341–349.

[4] P. Mishra, D. Passos, F. Marini, J. Xu, J.M. Amigo, A.A. Gowen, J.J. Jansen, A. Biancolillo, J.-M. Roger, D. Rutledge, A. Norden, Deep learning for near-infrared spectral data modelling: hypes and benefits, Trends Anal. Chem. 157 (2022) 116804.

[5] Y. Ozaki, C. Huck, S. Tsuchikawa, S. Balling Engelson, in: Near Infrared Spectroscopy, Springer, Singapore, 2021.

[6] M.E. Swartz, I.S. Krull, Handbook of Analytical Validation, CRC Press, Boca Raton, FL, USA, 2012.

[7] J.A. Fernández Pierna, P. Dardenne, V. Baeten, In-house validation of a near infrared hyperspectral imaging method for detecting processed animal proteins in compound feed, J. Near Infrared Spectrosc. 18 (2010) 121–133.

[8] J. Ezenarro, D. Schorn-García, How are chemometric models validated? A systematic review of linear regression models for NIRS data in food analysis, J. Chemometr. 39 (2025) e70036.

[9] E. Lopez, J. Etxebarria-Elezgarai, J.M. Amigo, A. Siefert, The importance of choosing a proper validation strategy in predictive models. A tutorial with real examples, Anal. Chim. Acta 1275 (2023) 341532.

[10] T.G. Dietterich, Approximate statistical tests for comparing supervised classification learning models, Neural Computing 10 (1998) 1895–1923.

[11] EN ISO 12099:2107 E, Animal Feeding Stuffs, Cereals and Milled Cereal Products – Guidelines for the Application of near Infrared Spectroscopy, European Committee for Standardization, 2017.

[12] C. Beleites, U. Neugebauer, T. Bocklitz, C. Krafft, J. Popp, Sample size planning for classification models, Anal. Chim. Acta 760 (2013) 25–33.

[13] T. Fearn, D. Pérez Marín, A. Garrido Varo, J.E. Guerrero Ginel, Classifying with confidence using Bayes rule and kernel density estimation, Chemometr. Intell. Lab. Syst. 189 (2019) 81–87.

[14] R. Kohavi, A study of cross-validation and bootstrap for accuracy estimation and model selection, in: C.S. Mellish (Ed.), Artificial Intelligence Proceedings 14th International Joint Conference, 20-25 August 1995, Morgan Kaufman, USA, Montreal, Quebec, Canada, 1995.

[15] E.R. Dougherty, C. Sima, J. Hua, B. Hanczar, U.M. Braga-Neto, Performance of error estimators for classification, Curr. Bioinf. 11 (2010) 53–67.

[16] J.-H. Kim, Estimating classification error rate: repeated cross-validation, repeated hold-out and bootstrap, Comput. Stat. Data Anal. 53 (11) (2009) 3735–3745.

[17] C. Beleites, R. Salzer, Assessing and improving the stability of chemometric models in small sample size situations, Anal. Bioanal. Chem. 390 (5) (2008) 1261–1271.

[18] G.C. Cawley, N.L.C. Talbot, On overfitting in model selection and subsequent selection bias in performance evaluation, J. Mach. Learn. Res. 11 (2010) 2079–2107.

[19] S. Varma, R. Simon, Bias in error estimation when using cross-validation for model selection, BMC Bioinf. 7 (2006) 91.

[20] P. Filzmoser, B. Liebmann, K. Varmuza, Repeated double cross validation, J. Chemometr. 23 (4) (2009) 160–171.

[21] T. Næs, T. Isaksson, T. Fearn, T. Davies, A User Friendly Guide to Multivariate Calibration and Classification, IM Publications, Chichester, 2017.

[22] J. Gajdos Klusurić, V. Baeten, A. Koidis, C. Beleites, Sampling for non-destructive spectroscopy with a particular focus on agriculture, food and feed, Trends Anal. Chem., submitted to this special issue.

[23] F. Westad, F. Marini, Validation of chemometric models – a tutorial, Anal. Chim. Acta 893 (2015) 14–24.

[24] L. Buchen, Cancer: missing the mark, Nature 471 (7339) (2011) 428–432.

[25] D. von Maydell, C. Beleites, A.-M. Stache, D. Riewe, A. Krämer, F. Marthe, Genetic variation of annual and biennial caraway (Carum carvi) germplasm offers diverse opportunities for breeding, Ind. Crop. Prod. 208 (2024) 117789.

[26] X. Chu, Y.-H. Yun, Y. Huang, X. Bian, Chemometric Methods in Analytical Spectroscopy Technology, Springer, Singapore, 2022.

[27] T. Næs, The design of calibration in near infrared reflectance analysis by clustering, J. Chemometr. 1 (1987) 121–134.

[28] R.D. Snee, Validation of regression models: methods and examples, Technometrics 19 (4) (1977) 415–428.

[29] W. Kennard, L.A. Stone, Computer aided design of experiments, Technometrics 11 (1969) 137–148.

[30] M. Stone, Cross-validatory choice and assessment of statistical predictions, J. Roy. Stat. Soc. B 36 (2) (1974) 111–147.

[31] B.D. Ripley, Pattern Recognition and Neural Networks, CUP, Cambridge, 2008.

[32] T. Hastie, R. Tibshirani, J. Friedman, The Elements of Statistical Learning, Springer, New York, 2001.

[33] G. James, D. Witten, T. Hastie, R. Tibshirani, An Introduction to Statistical Learning: with Applications in R, second ed., Springer, New York, 2021.

[34] S. Dochow, C. Beleites, T. Henkel, G. Mayer, A. Jens, J. Clement, C. Krafft, J. Popp, Quartz microfluidic chip for tumour cell identification by Raman spectroscopy in combination with optical traps, Anal. Bioanal. Chem. 405 (8) (2013) 2743–2746.

[35] J.D. Rodrigo, A. Perez, J.A. Lozano, Sensitivity analysis of k-fold cross validation in prediction error estimation, IEEE Trans. Pattern Anal. Mach. Intell. 32 (3) (2009) 569–575.

[36] Y. Bengio, Y. Grandvalet, No unbiased estimator of the variance of K-fold cross validation, J. Mach. Learn. Res. 5 (2004) 1089–1105.

[37] S. Bates, T. Hastie, R. Tibshirani, Cross-validation: what does it estimate and how well does it do it? J. Am. Stat. Assoc. 119 (2023) 1434–1445.
[38] A. Anselmo, A. Pissard, D. Vincke, Q. Arnould, B. Lecler, S. Gofflot, D. Michez, V. Baeten, Contribution of vibrational spectroscopy to the characterisation and detection of insect meal in compound feed, Ital. J. Anim. Sci. 24 (1) (2025) 763-771.

[39] Z. Iqbal, N.K. Afseth, A. Postelmans, J.P. Wold, P.V. Andersen, J. Kusnadi, W. Saeys, Detection and quantification of pork adulteration in beef meatballs with Raman spectroscopy and near infrared spectroscopy, Spectrochim. Acta Mol. Biomol. Spectrosc. 337 (2025) 126069.

[40] P. Williams, P. Dardenne, P. Flinn, Tutorial: items to be included in a report on a near infrared spectroscopy project, J. Near Infrared Spectrosc. 25 (2) (2017) 85-90.

[41] E. Bobelyn, A.S. Serban, M. Nicu, J. Lammertyn, B.M. Nicolai, W. Saeys, Postharvest quality of apple predicted by NIR spectroscopy: study of the effect of biological variability on spectra and model performance, Postharvest Biol. Technol. 55 (3) (2010) 133-143.

[42] J.A. Diaz-Olivares, I. Adriaens, E. Stevens, W. Saeys, B. Aeronouts, Online milk composition analysis with an on-farm near-infrared sensor, Comput. Electron. Agric. 178 (2020) 105734.

[43] V. Fonseca-Diaz, B. Aeronouts, E. Bobelyn, B. Nicolai, W. Saeys, Monitoring of multivariate calibration models in the absence of new reference values: the regression case, Chemometr. Intell. Lab. Syst. 240 (2023) 104884.

[44] R. Walker, M. Thompson, R. Lawn, Proficiency Testing in Analytical Chemistry, Royal Society of Chemistry, London, 1997.

[45] B.G. Osborne, T. Fearn, P.H. Hindle, Practical NIR Spectroscopy, second ed., Longman, Harlow, 1993.

[46] D.C. Montgomery, Introduction to Statistical Quality Control, seventh ed., Wiley, Hoboken, NJ, 2013.

[47] J.M. Roger, V.F. Diaz, R. Nikzad-Langerodi, Cloning instruments, model maintenance and calibration transfer, Trends Anal. Chem. 191 (2025) 118319.

[48] B. Kowalski, S. Wold, in: P.R. Krishnajah, L.N. Kanal (Eds.), Pattern Recognition in Chemistry, in Pattern Recognition and Reduction of Dimensionality, Handbook of Statistics, vol. II, 1982, pp. 673-697. II, North-Holland, Amsterdam.