# A general differential split-sample test to select sub-periods of discontinuous years gathering similar to different climate conditions

Hamouda Dakhlaoui, Denis Ruelland, Yves Tramblay

► To cite this version:

Hamouda Dakhlaoui, Denis Ruelland, Yves Tramblay. A general differential split-sample test to select sub-periods of discontinuous years gathering similar to different climate conditions. MethodsX, 2020, 575, pp.470 - 486. 10.1016/j.mex.2020.101008. hal-03143061

HAL Id: hal-03143061

https://hal.umontpellier.fr/hal-03143061

Submitted on 16 Feb 2021

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_2.png" /></figure>

ELSEVIER

Contents lists available at ScienceDirect

MethodsX

journal homepage: www.elsevier.com/locate/mex

<figure><img src="image_3.png" /></figure>

Method Article

# A general differential split-sample test to select sub-periods of discontinuous years gathering similar to different climate conditions

<figure><img src="image_4.png" /></figure>

Hamouda Dakhlaoui<sup>a,b,*</sup>, Denis Ruelland<sup>c</sup>, Yves Tramblay, PhD<sup>c</sup>

## ABSTRACT

This article introduces a Matlab® code to implement the General Differential Split Sample Test (GDSST) (Dakhlaoui et al. [5]). As an illustration, the GDSST is applied to five catchments in northern Tunisia over 30-year reference period and compared to three benchmark Split Sample Test (SST) methods. The techniques are compared as regards to the number of validation exercises and to the differences in temperature (ΔT) and precipitation (ΔP) between the sampled sub-periods, whose length was set to 8-year. The GDSST allows a larger number of discontinuous periods to be sampled, and is computationally more effective than the basic bootstrap to identify the most climatically contrasting conditions. In addition, the GDSST offers a larger continuum of climatic conditions and a better spread of validation periods than the benchmark techniques, which is essential to test the parameter transferability of hydrological models. As supplementary material, a package file containing MATLAB® scripts to run the three benchmark SSTs and the proposed GDSST, as well as an application example on the five catchments, can be freely downloaded.

* An enhanced split-sample test based on an oriented bootstrap to assess transferability of hydrological models.

* The proposed split-sample test is computationally more effective than the basic bootstrap to identify the most climatically contrasting conditions.

* MATLAB® code of the proposed GDSST and four benchmark SST, with application example.

© 2020 The Authors. Published by Elsevier B.V.

This is an open access article under the CC BY license. ([http://creativecommons.org/licenses/by/4.0/](http://creativecommons.org/licenses/by/4.0/))

## ARTICLE INFO

Method name: General differential split-sample test (GDSST)

Keywords: Rainfall-runoff modeling, Split-sample tests, Parameter transferability, Semi-arid climate

Article history: Received 21 October 2019; Accepted 20 July 2020; Available online 23 July 2020

DOI of original article: 10.1016/j.jhydrol.2019.05.056

* Corresponding author at: LMHE, Ecole Nationale d'Ingénieurs de Tunis, University of Tunis El Manar, BP 37, 1002 Tunis le Belvedère, Tunisia.

E-mail addresses: hammouda.dakhlaoui@laposte.net (H. Dakhlaoui), yves.tramblay@ird.fr (Y. Tramblay).

https://doi.org/10.1016/j.mex.2020.101008

2215-0161/© 2020 The Authors. Published by Elsevier B.V. This is an open access article under the CC BY license.

([http://creativecommons.org/licenses/by/4.0/](http://creativecommons.org/licenses/by/4.0/))

<footnote><sup>a</sup> LMHE, Ecole Nationale d'Ingénieurs de Tunis, University of Tunis El Manar, BP 37, 1002 Tunis le Belvedère, Tunisia</footnote>

<footnote><sup>b</sup> Ecole Nationale d'Architecture et d'Urbanisme, University of Carthage, Rue El Quods, 2026 Sidi Bou Said, Tunisia</footnote>

<footnote><sup>c</sup> HydroSciences Montpellier (Univ. Montpellier, CNRS, IRD), Montpellier, France</footnote>
Specifications Table

<table><tr><td>Subject Area</td><td>Earth and Planetary Sciences</td></tr><tr><td>More specific subject area</td><td>Hydrology<br/>Hydroinformatic</td></tr><tr><td>Method name</td><td>General differential split-sample test (GDSST)</td></tr><tr><td>Name and reference of original method</td><td>Coron, L., Andréassian, V., Perrin, C., Lerat, J., Vaze, J., Bourqui, M., Hendrickx, F., 2012. Crash testing hydrological models in contrasted climate conditions: an experiment on 216 Australian catchments. Water Resour. Res., 48, W05552. doi:10.1029/2011WR011721<br/>Coron, L., 2013. Les modèles hydrologiques conceptuels sont-ils robustes face à un climat en évolution ? PhD Thesis, ISIVE, AgroParisTech, 364 p.<br/>Dakhlaoui, H., Ruelland, D., Tramblay, Y., Bargaoui, Z., 2017. Evaluating robustness of conceptual rainfall-runoff models under climate variability in northern Tunisia. J. Hydrol., 550, 201–217. doi:10.1016/j.jhydrol.2017.04.032<br/>Dakhlaoui, H., Ruelland, D., and Tramblay Y. (2019). A bootstrap-based differential split-sample test to assess the transferability of conceptual rainfall-runoff models under past and future climate variability. Journal of Hydrology. https://doi.org/10.1016/j.jhydrol.2019.05.056.</td></tr></table>

Introduction

This article introduces a transferable package (see supplementary material) of the General Differential Split Sample Test (GDSST) to select sub-periods of discontinuous years gathering similar to different conditions in terms of precipitation and temperature. The GDSST was originally proposed in Dakhlaoui et al. [5] to assess the transferability of conceptual rainfall-runoff models under past and future climate variability. In this paper, we showed that compared to three existing benchmark techniques, the GDSST allowed a larger number of climatically contrasted discontinuous periods to be sampled, and was computationally more effective than the basic bootstrap to identify the most contrasted periods. When applied to three hydrological models in five catchments in northern Tunisia, the GDSST provided clear transferability limits of the models under changing precipitation (P) and temperature (T) conditions towards drier and hotter conditions. We also showed that some climate projections of temperature and precipitation from the EURO-CORDEX exercise fell outside these transferability limits.

Since a specific research method was customized for the above article, we thought readers might be interested in accessing the codes developed to run the proposed GDSST, the three benchmark techniques, as well as the application example on the five studied catchments. The current paper thus focuses on this technical part of our work as a description of a MATLAB package.

Description of the split-sample techniques included in the package

Three benchmark SST techniques

The SST methods included in the package and selected for comparison with the proposed GDSST are (Fig. 1): (i) a sliding-window SST [2]; (ii) a random bootstrap SST [1,3]; and (iii) a 4-sub-period DSST [4]. These three techniques were selected because they enable simultaneous investigation of the effect of T and P on model transferability under climate variability.

The sliding-window SST technique [2] consists in using calibration-validation tests on independent sub-periods of equal length, considering all possible pairs of sub-periods. The sampling method used to generate sub-periods is based on sliding windows applied over the reference period. The technique enables the identification of n-l + 1 calibration sub-periods, where l is the number of years composing each sub-periods and n is the number of years of the reference period.
(a) Sliding-window SST

<figure><img src="image_5.png" /></figure>

(b) Random bootstrap SST

<figure><img src="image_6.png" /></figure>

(c) 4-sub-period DSST

<figure><img src="image_7.png" /><figcaption>**Fig. 1.** Split-sample methods according to (a) a sliding-window SST technique, (b) a random bootstrap SST technique, and (c) a 4-sub-period DSST technique [5].</figcaption></figure>

The random bootstrap SST technique [1,3] relies on a sub-period sampling technique which is based on a random combination of discontinuous years (bootstrap). This sampling technique is time consuming since the possible number of calibration sub-periods is equal to $C_h$. For example, the random bootstrap SST technique results in around six million possible 8-year sub-periods if applied to a 30-year reference period. Its application then requires a priori selection of the number of permitted calibration exercises, due to limited time budget for model calibration and validation.

The implementation of the 4-sub-period DSST [4] requires the calculation of the annual precipitation and mean temperature for each hydrological year of the reference period. The sub-periods are thus made up of groups of climatically contrasted years. To create these groups, the hydrological years are first distributed into two equal groups of hydrological years (dry years and wet years) according to the annual precipitation median for the reference period (Fig. 1c). Dry and wet years are defined as years with respectively less or more total precipitation than the median of the reference period. For each group, the median of the mean annual temperature is then calculated, which serves to distinguish hot and cold years. The four final groups of hydrological years are: hot/dry (HD), hot/wet (HW), cold/dry (CD) and cold/wet (CW) years (Fig. 1).

Using the three above techniques makes it possible to identify different numbers of calibration sub-periods of *n* years. All *n*-year periods which do not have any year in common with a given *n*-year calibration period can thus be considered as independent validation exercises. As a result, the number of validation exercises may not be the same for all calibration periods selected with the sliding-window and random bootstrap SST. For the 4-sub-period SST, there are three possible validation exercises for each of the 4 calibration sub-period.

### Proposed general differential split-sample test (GDSST)

Based on the existing SST methods, we developed a technique which can take benefit from the random bootstrap SST technique to provide a large number of validation exercises while accounting
<figure><img src="image_8.png" /><figcaption>**Fig. 2.** Processing steps to sample *k* climate contrasted *l*-year sub-periods from *n*-year reference period, in the proposed GDSST. Each point represents a hydrological year from the reference period in the climate space (*T*, *P*). The years circled are those selected [5].</figcaption></figure>

for the much contrasted Δ*T* and Δ*P* detected with the 4-sub-period DSST. In other words, the idea was to design a method which uses the sampling of the random bootstrap SST technique, but which is oriented so as to obtain the extreme climate contrast provided by the 4-sub-period DSST. The proposed method was called general differential split-sample test (GDSST) and is described in Fig. 2.

The procedure used to generate *k* *n*-year sub-periods from the *l* hydrological years (from the 1st of September to the 31st of August) of reference period, is as follows. The first year of the *n*-year sub-period to be sampled is randomly selected from the *l* years of the reference period (step 1 in Fig. 2). The *l*-1 remaining years of the reference period are then sorted based on the order of increasing distance of Mahalanobis [6] to the first selected year in the space of mean annual temperature (*T*) and total annual precipitation (*P*) (step 2 in Fig. 2). Using the Mahalanobis distance aims at rescaling the *T* and *P* axes in order to account for the correlations between the two variables and to calculate standard Euclidean distance in a transformed space having unit variance. In other words, it aims to reduce the dominance of one climatic variable over the other when computing “climatic” distance between years. A trapezoidal probability is then assigned to the *l*-1 remaining years of the reference period, as follows (step 3 in Fig. 2):

$$P(i) = \frac{2(m+1-i)}{m+1}, \quad i=1, \dots, m \qquad (1)$$

$$P(i) = 0, \quad i = m+1, \dots, l-1 \qquad (2)$$
where $P(i)$ is the probability assigned to the year with rank $i$; $i$ is the rank of the remaining years of the reference period sorted in order of increasing Mahalanobis distance to the originally selected year; $m$ is a number selected randomly at each sub-period selection from the interval $[n-1, L-1]$. The year closest to the year originally retained has the highest probability ($P(1)= 2/m + 1$) and the farthest years has the lowest probability ($P(m) = 2/m(m + 1)$ and $P(i) = 0$ for $i > m$).

The $n-1$ remaining years of the sub-period are then selected from the $L-1$ remaining years of the reference period according to the trapezoidal probability distribution giving more chance to be selected to the years which are the closest to the initial year according to the Mahalanobis distance defined in the $T$ and $P$ space (step 4 in Fig. 2). The trapezoidal distribution allows only the $m$ years closest to the initial year retained, to be selected in the sub-period. This gives more chance to years with similar climatic conditions to be selected in order to generate more climatically contrasted sub-periods. However, varying randomly $m$ for each sub-period generation also allows years with different climatic conditions to be selected. This aims at creating a continuum of climatic conditions, from similar to contrasted, between the sampled sub-periods in view of evaluating the model transferability under increasing climate contrasts. In case the new created sub-period was already sampled, it is not retained (step 5 in Fig. 2). The procedure (steps 1 to 5 in Fig. 2) is repeated until the required number of sub-periods is reached (step 7 in Fig. 2).

The random selection of years in the proposed procedure allows a larger number of sub-periods to be selected than with a deterministic procedure (where the closest years to the originally retained year are selected). In fact, in the best case, the deterministic procedure provides a number of sub-periods equal to the number of observed years (e.g. 30 sub-periods for a 30-year reference period). The number of calibration sub-periods which can be generated by the proposed technique is similar to the random bootstrap SST technique ($C_n^l$). That is why its application requires a priori selection of the number of permitted calibration exercises. Similarly to the three benchmark SST (See Section "Three benchmark SST techniques"), all $n$-year periods which do not have any year in common with a given $n$-year calibration period can be considered as independent validation exercises with the GDSST.

MATLAB code of the split sample tests

The routine "SST.m", presented below and contained in the GDSST package (see supplementary material), allows generating sub-periods by GDSST [5] and by three benchmark Split Sample Tests: (i) sliding-window SST [2]; (ii) random bootstrap SST [1,3]; and (iii) 4-sub-period DSST [4].

The arguments of the routine are:

OptSST: variable used to set the SST to be used. It must be set to 'GSST' for GDSST, 'Mobile' for sliding-window SST, 'Rand_part' for random bootstrap SST, and '4PDSST' for 4-sub-period DSST.

AnnualPrecip: array of $2 \times n$ dimension. The first column is for years and the second column for annual precipitation. $n$ the number of years of the reference period.

AnnualTemp: array of $2 \times n$ dimension. The first column is for years and the second column for mean annual temperature. $n$ the number of years of the reference period. nsousperiod: number of sub-periods to be generated by GDSST or random bootstrap SST durationSubP: duration of the sub-periods expressed in years.

The routine gives the following outputs: echantillon: a four-column array containing all the independent calibration-validations exercises. Each line contains one calibration-validations exercise. The first column contains the order of calibration period. The second column represents the order of validation period. The third and fourth columns represent the changes in temperature ($\Delta T$) and in precipitation ($\Delta P$), respectively, between calibration and validation period. The order of subperiod is the same that Combination array.

Combination: contain the years composing the generated sub-periods. Each line contains one subperiod. The order of subperiod in this array is used in echantillon array.
The MATLAB® code of the routine "SST.m", is presented as follows:

```matlab
function [echantillon,Combination] = SST(OptSST, AnnualPrecip, AnnualTemp, nsousperiod, durationSubP)
```

```
% GENERAL INFORMATION
```

% This subroutine generates sub-periods by General Differential Split Sample
% Test (Dakhlaoui et al., 2019) and by three benchmark Split Sample Tests:
% (i) sliding-window SST (Coron et al., 2012);
% (ii) random bootstrap SST (Coron, 2013; Arsenault et al., 2018);
% (iii) 4-sub-period DSST (Dakhlaoui et al., 2017).
```

```
% Arguments
```

% OptSST equal to 'GSST' for GDSST, 'Mobile' for sliding-window SST,
% 'Rand_part' for random bootstrap SST, and '4PDSST' for
% 4-sub-period DSST.
% AnnualPrecip array of nx2 dimension the first column is for years
% and the second column for annual precipitation. n the number
% of years of the reference period
% AnnualTemp array of nx2 dimension the first column is for years
% and the second column for mean annual temperature .
% n the number of years of the reference period
% Nsousperiod number of subperiods to be generated by GDSST
% or random bootstrap SST
```

```
% Outputs
```

% echantillon a four column array containing all the independent
% validations exercises. The first column for the order
% of calibration period , the second the order of validation
% period, the third contains the changes in temperature
% between calibration and validation period DT and the
% last contains the changes in precipitation DP.
% Combination contain the years composing the generated subperiods.
% Each line contains one subperiod.
```

% Variable affectation

Years = AnnualPrecip(:,1);
ANDEB = AnnualPrecip(1,1);
Temperature = AnnualTemp(:,2);
Precipitation = AnnualPrecip(:,2);

% Calculation of Hot-Dry, Hot Wet,Cold Dry and Cold Wet years

medianep = median(AnnualPrecip(:,2));
medianet = median(AnnualTemp(:,2));
WetYears = AnnualPrecip(AnnualPrecip(:,2)>medianep,1);
WetYearst = AnnualTemp(AnnualPrecip(:,2)>medianep,2);
medianetw = median( WetYearst);
DryYears = AnnualPrecip(AnnualPrecip(:,2)<= medianep,1);
DryYearst = AnnualTemp(AnnualPrecip(:,2)<=medianep,2);
medianetd = median( DryYearst);
HotDryYears = AnnualTemp(AnnualTemp(:,2)>= medianetd&AnnualPrecip(:,2)<= medianep,1);
HotWetYears = AnnualTemp(AnnualTemp(:,2)>= medianetw&AnnualPrecip(:,2)> medianep,1);
ColdDryYears = AnnualTemp(AnnualTemp(:,2)< medianetd&AnnualPrecip(:,2)<= medianep,1);
ColdWetYears = AnnualTemp(AnnualTemp(:,2)< medianetw&AnnualPrecip(:,2)> medianep,1);

% Calculation of subperiods according the selected SST

```
switch OptSST
```

case 'Rand_part'

Combination1 = nchoosek(Years,durationSubP);
sizeCombination1 = size(Combination1);
Totcombin1 = sizeCombination1(1);
a = unique(round(rand(nsousperiod,1)*Totcombin1));
Combination2 = Combination1(a,:);
Combination = sort(Combination2,2);
Combination = unique(Combination,'rows');
sizeCombination = size(Combination);
Totcombin = sizeCombination(1);
```R
case 'GDSST'

nyears      = length (Precipitation);
a          = (ceil(rand (nsousperiod,1)*nyears));
selectyears = Years(a);
Iclimat     = [Precipitation Temperature];
[Combin, Dist] = knnsearch(Iclimat,Iclimat,'k',nyears,'distance', 'mahalanobis');

i = 0;

while I < nsousperiod
    i = i+1;
    yearalea   = ceil(rand (1,1)*nyears);
    nyears2    = max(ceil(rand*nyears),durationSubP-1);
    if nyears2==durationSubP-1
        lcs = 1:durationSubP;
    else
        lcs(1) = 1;
        for k3 = 2:durationSubP;
            for iter = 1:1000;
                lpos = 1 + floor(nyears2+0.5-sqrt((nyears2+0.5)^2 - nyears2*(nyears2+1)*rand));
                idx = find(lcs(1:k3-1)==lpos); if isempty(idx); break; end;
            end;
            lcs(k3) = lpos;
        end;
        lcs = sort(lcs);
    end;

    Combination2 (i,: ) = Years(Combin(yearalea,lcs));
    Combination2       = unique(Combination2,'rows');
    i=size(Combination2,1);
end

Combination   = sort(unique(Combination2,'rows'),2);
sizeCombination = size(Combination);
Totcombin      = sizeCombination(1);


case '4PDSST'

HotDryYears2  = NaN(durationSubP,1);
HotWetYears2  = NaN(durationSubP,1);
ColdDryYears2 = NaN(durationSubP,1);
ColdWetYears2 = NaN(durationSubP,1);

sizeHotDryYears = size(HotDryYears);
sizeHotWetYears = size(HotWetYears);
sizeColdDryYears = size(ColdDryYears);
sizeColdWetYears = size(ColdWetYears);
```
```R
if (sizeHotDryYears(1)==durationSubP)
  HotDryYears2 = HotDryYears;
else if (sizeHotDryYears(1)>durationSubP)
  HotDryYears2 = HotDryYears(1:durationSubP);
else
  HotDryYears2(1:sizeHotDryYears(1)) = HotDryYears;
end

if (sizeHotWetYears(1)==durationSubP)
  HotWetYears2 = HotWetYears;
else if (sizeHotWetYears(1)>durationSubP)
  HotWetYears2 = HotWetYears(1:durationSubP);
else
  HotWetYears2(1:sizeHotWetYears(1)) = HotWetYears;
end

if (sizeColdDryYears(1)==durationSubP)
  ColdDryYears2 = ColdDryYears;
else if (sizeColdDryYears(1)>durationSubP)
  ColdDryYears2 = ColdDryYears(1:durationSubP);
else
  ColdDryYears2(1:sizeColdDryYears(1)) = ColdDryYears;
end

if (sizeColdWetYears(1)==durationSubP)
  ColdWetYears2 = ColdWetYears;
else if (sizeColdWetYears(1)>durationSubP)
  ColdWetYears2 = ColdWetYears(1:durationSubP);
else
  ColdWetYears2(1:sizeColdWetYears(1)) = ColdWetYears;
end

Combination2 = [ reshape(HotDryYears2 (1:durationSubP),1,durationSubP);
  reshape(HotWetYears2 (1:durationSubP),1,durationSubP);
  reshape(ColdDryYears2(1:durationSubP),1,durationSubP);
  reshape(ColdWetYears2(1:durationSubP),1,durationSubP) ];
Combination = unique(Combination2, 'rows');
sizeCombination = size(Combination);
Totcombin = sizeCombination(1);

case 'Mobile'
  Totcombin = length(Years)-durationSubP +1;
  Combination = zeros(Totcombin, durationSubP);

  for i=1:Totcombin
    for j=1:durationSubP;
      Combination(i,j) = Years(i+j-1);
    end
end

end

% Subperiod Climatic characteristics calculation
for i=1:Totcombin
  TMeanSubP(i) = mean(AnnualTemp.find(ismember(AnnualTemp(:,1),Combination(i,:)),2));
  PMeanSubP(i) = mean(AnnualPrecip.find(ismember(AnnualPrecip(:,1),Combination(i,:)),2));
end
VariableClim = [TMeanSubP; PMeanSubP];

% Constitution of validation exercises
iechantillon=0;
for i=1:Totcombin %cal
  for j=1:Totcombin %val
    if (length(intersect (Combination(i,:),Combination(j,:))) == 0)
      iechantillon = iechantillon+1;
      echantillon(iechantillon,1) = i;
      echantillon(iechantillon,2) = j;
      echantillon(iechantillon,3) = TMeanSubP(j)-TMeanSubP(i);
      echantillon(iechantillon,4) = (PMeanSubP(j)-PMeanSubP(i))/PMeanSubP(i);
    end
  end
end
end
```
## Application example

The GDSST package (see supplementary material) includes an application example of the four split-sampling methods using the climatic data from five catchments in northern Tunisia (PTfile.mat), it can be run via the Matlab script (MainSST.m).

The data file: "PTfile.mat"

This matlab file contains the dataset of an application example from five catchments in northern Tunisia (Rhezala, Melah, Maaden, Joumine and El Abid). See Dakhlaoui et al. [5] for more details about the catchments. It contains an array PT of 30 × 11 dimension. The first column of the PT array contains the years, for each catchment two columns are reserved, one for the annual precipitation and the second for mean annual temperature.

The reference period is from 1st September 1970 to 31st August 2000. It was based on the hydrological years (from the 1st of September to the 31st of August).

The PT array of the "PTfile.mat" of the application example of the five catchments of the northern Tunisia, is presented below. The first column contains the years (1971–2000), the second and third columns are reserved respectively to annual precipitation and mean annual temperature of the catchment 1, the fourth and fifth columns are reserved respectively to annual precipitation and mean annual temperature of the catchment 2, etc. The users can implement the new and benchmark split-sample approaches with their own data, by adapting the dimension of this table to the number of their study catchments and years.

<table><tr><td>1971</td><td>510,2</td><td>17,3</td><td>1047,2</td><td>15,7</td><td>1032,5</td><td>15,7</td><td>885,0</td><td>15,9</td><td>911,8</td><td>16,1</td></tr><tr><td>1972</td><td>661,0</td><td>16,6</td><td>1009,6</td><td>14,9</td><td>946,5</td><td>14,8</td><td>743,3</td><td>15,0</td><td>775,1</td><td>15,1</td></tr><tr><td>1973</td><td>730,6</td><td>17,0</td><td>1193,1</td><td>15,2</td><td>1084,3</td><td>15,2</td><td>897,1</td><td>15,4</td><td>905,7</td><td>15,6</td></tr><tr><td>1974</td><td>451,2</td><td>17,4</td><td>636,3</td><td>15,7</td><td>730,2</td><td>15,8</td><td>596,6</td><td>15,9</td><td>620,6</td><td>16,1</td></tr><tr><td>1975</td><td>655,5</td><td>16,5</td><td>851,6</td><td>14,9</td><td>1005,5</td><td>15,1</td><td>743,0</td><td>15,3</td><td>796,6</td><td>15,5</td></tr><tr><td>1976</td><td>504,0</td><td>16,8</td><td>819,3</td><td>15,0</td><td>944,6</td><td>15,2</td><td>804,5</td><td>15,4</td><td>805,3</td><td>15,6</td></tr><tr><td>1977</td><td>637,9</td><td>17,3</td><td>797,8</td><td>15,9</td><td>933,6</td><td>16,2</td><td>706,6</td><td>16,2</td><td>741,6</td><td>16,5</td></tr><tr><td>1978</td><td>355,6</td><td>17,0</td><td>787,9</td><td>16,0</td><td>929,0</td><td>16,2</td><td>741,3</td><td>16,0</td><td>778,1</td><td>16,2</td></tr><tr><td>1979</td><td>510,1</td><td>17,2</td><td>791,7</td><td>16,7</td><td>829,2</td><td>16,8</td><td>741,9</td><td>16,1</td><td>728,5</td><td>16,4</td></tr><tr><td>1980</td><td>542,0</td><td>16,5</td><td>837,9</td><td>15,7</td><td>1087,0</td><td>15,8</td><td>933,7</td><td>15,4</td><td>890,3</td><td>15,7</td></tr><tr><td>1981</td><td>372,3</td><td>16,9</td><td>843,0</td><td>16,2</td><td>816,6</td><td>16,4</td><td>764,1</td><td>15,9</td><td>718,4</td><td>16,2</td></tr><tr><td>1982</td><td>500,2</td><td>18,1</td><td>897,5</td><td>17,2</td><td>973,7</td><td>17,3</td><td>781,7</td><td>16,9</td><td>819,8</td><td>17,2</td></tr><tr><td>1983</td><td>562,6</td><td>17,6</td><td>777,3</td><td>16,5</td><td>934,7</td><td>16,7</td><td>848,0</td><td>16,4</td><td>794,4</td><td>16,6</td></tr><tr><td>1984</td><td>584,9</td><td>17,0</td><td>991,4</td><td>15,8</td><td>978,4</td><td>16,0</td><td>757,3</td><td>15,9</td><td>805,9</td><td>16,1</td></tr><tr><td>1985</td><td>516,4</td><td>17,3</td><td>959,3</td><td>16,2</td><td>1105,1</td><td>16,3</td><td>850,7</td><td>16,2</td><td>951,0</td><td>16,4</td></tr><tr><td>1986</td><td>414,8</td><td>17,7</td><td>718,9</td><td>16,2</td><td>868,9</td><td>16,5</td><td>747,3</td><td>16,5</td><td>768,9</td><td>16,8</td></tr><tr><td>1987</td><td>734,5</td><td>17,3</td><td>1206,6</td><td>16,5</td><td>1244,1</td><td>16,7</td><td>896,6</td><td>16,3</td><td>959,2</td><td>16,6</td></tr><tr><td>1988</td><td>200,9</td><td>18,6</td><td>663,7</td><td>17,8</td><td>712,5</td><td>18,0</td><td>416,7</td><td>17,7</td><td>575,8</td><td>17,9</td></tr><tr><td>1989</td><td>354,9</td><td>17,9</td><td>660,7</td><td>16,8</td><td>708,9</td><td>17,0</td><td>587,0</td><td>16,7</td><td>568,9</td><td>16,9</td></tr><tr><td>1990</td><td>511,4</td><td>18,2</td><td>553,7</td><td>17,1</td><td>906,7</td><td>17,3</td><td>638,7</td><td>17,1</td><td>634,9</td><td>17,3</td></tr><tr><td>1991</td><td>624,6</td><td>17,6</td><td>1049,5</td><td>16,2</td><td>1127,1</td><td>16,4</td><td>993,4</td><td>16,3</td><td>1009,0</td><td>16,6</td></tr><tr><td>1992</td><td>656,2</td><td>17,3</td><td>977,5</td><td>15,6</td><td>956,0</td><td>15,9</td><td>768,8</td><td>15,7</td><td>775,0</td><td>16,0</td></tr><tr><td>1993</td><td>488,1</td><td>17,8</td><td>648,4</td><td>16,4</td><td>660,8</td><td>16,7</td><td>588,2</td><td>16,5</td><td>605,6</td><td>16,8</td></tr><tr><td>1994</td><td>482,1</td><td>18,5</td><td>654,9</td><td>17,2</td><td>754,4</td><td>17,5</td><td>494,2</td><td>17,3</td><td>590,4</td><td>17,6</td></tr><tr><td>1995</td><td>303,4</td><td>18,4</td><td>651,0</td><td>16,9</td><td>745,5</td><td>17,1</td><td>537,5</td><td>17,1</td><td>610,5</td><td>17,3</td></tr><tr><td>1996</td><td>861,2</td><td>18,0</td><td>1062,8</td><td>16,3</td><td>1197,7</td><td>16,6</td><td>989,7</td><td>16,5</td><td>1200,7</td><td>16,7</td></tr><tr><td>1997</td><td>395,4</td><td>18,3</td><td>627,9</td><td>16,8</td><td>721,5</td><td>17,1</td><td>526,9</td><td>17,0</td><td>589,9</td><td>17,2</td></tr><tr><td>1998</td><td>588,5</td><td>18,4</td><td>1123,6</td><td>16,8</td><td>1258,8</td><td>17,1</td><td>980,0</td><td>17,0</td><td>1114,1</td><td>17,2</td></tr><tr><td>1999</td><td>631,5</td><td>18,2</td><td>1240,8</td><td>17,1</td><td>941,0</td><td>17,4</td><td>777,1</td><td>17,3</td><td>852,1</td><td>17,6</td></tr><tr><td>2000</td><td>569,4</td><td>18,7</td><td>636,7</td><td>17,6</td><td>704,0</td><td>17,9</td><td>512,7</td><td>17,8</td><td>583,7</td><td>18,1</td></tr></table>

The main program: "MainSST.m"

The main program allows reading the needed data for the SST "PTfile.m" and running the SST by calling the "SST.m" routine.

Three variables need to be set by the user:
OptSST: Variable that define the SST to be used. It must be set to 'GDSST' for GDSST, 'Mobile' for sliding-window SST,

'Rand_part' for random bootstrap SST, and '4PDSST' for 4-sub-period DSST.

nsousperiod: number of sub-periods to be generated by GDSST or random bootstrap SST
durationSubP: Duration of the sub-periods expressed in years.

The code give the following outputs:

Echantillon2: a four column array containing all the independent validations exercises. The first column contain the order of calibration period, the second the order of validation period, the third contain the change in temperature between calibration and validation period ΔT and the last contains the relative change in precipitation ΔP. Validations exercises from the first catchment are ranged in the first lines, then the second catchment, etc. The order of catchments is the same as PTfile.m

Combination2: contain the years composing the generated sub-periods. Each line contains one sub-
period. The first nsousperiod lines contain sub-periods from the first catchment, then the second
catchment, etc. The order of catchments is the same as PTfile.m figure.m: Scatter representing the
calibration-validation exercises generated by the selected SST expressed in term of ΔT and ΔP.

The MATLAB® code of the main program "MainSST.m" is presented as follow:

% GENERAL INFORMATION %
%**********************************************************************
% This script Generate sub-periods by General Differential Split Sample
% Test (Dakhlaoui et al., 2019) and three benchmark Split Sample Tests:
% (i) sliding-window SST (Coron et al., 2012);
% (ii) random bootstrap SST (Coron, 2013; Arsenault et al., 2018);
% (iii) 4-sub-period DSST (Dakhlaoui et al., 2017).
%**********************************************************************
% Variable that need to be set
% OptSST         : Variable that define the SST to be used. It must be equal
close all
% to 'GDSST' for GDSST, 'Mobile' for sliding-window SST,
clear
'Rand_part' for random bootstrap SST, and '4PDSST'
Ncatchment      = (size (PT,2)-1)/2;
%**********************************************************************
% Variable that need to be set
% OptSST         = 'GDSST' % 'GDSST', 'Mobile' , 'Rand_part', '4PDSST' % variable to be set by user
% Nsousperiod    = 100      % variable to be set by user
% durationSubP   = 8        % variable to be set by user

for j=1:ncatchment % loops over catchments
    a = int2str(j);
    disp(['Catchment : ', a])
    AnnualPrecip(:,1) = PT(:,1);
    AnnualTemp(:,1)  = PT(:,1);
    AnnualPrecip(:,2) = PT(:,j*2);
    AnnualTemp(:,2)  = PT(:,j*2+1);
    [echantillon,Combination] = SST(OptSST,AnnualPrecip,AnnualTemp,nsousperiod,durationSubP);
    if j==1
        echantillon2 = echantillon;
        Combination2 = Combination;
    else
        echantillon2 = [echantillon2;echantillon];
        Combination2 = [Combination2;Combination];
    end
end % end loops over catchments

% Scatter of the calibration-validation exercises generated by the selected SST expressed in term of DT and DP.
figure = scatter(echantillon2(:,3),echantillon2(:,4)*100);
ylabel('DP (%)','FontSize',12,'FontName','Arial');
xlabel('DT (°C)','FontSize',12,'FontName','Arial');
Implementation details

The MATLAB® codes (MainSST.m and SST.m) and the data file (PTfile.mat) must be put in the same folder. The three variables that need to be set by the user (OptSST, nsousperiod and durationSubP) have to be set directly in the MATLAB® code of the routine (SST.m). The SST could be performed by running "MainSST.m"

Results of the application example

As an illustration, the three benchmark SST techniques and the proposed GDSST were applied to the application example described above. The techniques were compared as regards to the number of validation exercises and to the precipitation-temperature differences they provided. The length of the sub-periods was set to 8 years for the sliding-window SST, the random bootstrap SST and the GDSST. However, for the 4-sub-period DSST, the 30-year reference period was spread over 7–8 years hot/dry, hot/wet, cold/dry and cold/wet sub-periods (since 30 is not a multiple of four). The random bootstrap SST results in a large number of possible sub-periods if fully applied to a 30-year period (around six millions 8-year sub-periods). Due to limited time budget for model calibration and validation, we decided to use only 100 randomly selected sub-periods for each catchment. For sake of fair comparison, the same number of randomly selected sub-periods was set with the GDSST. Note that the number of sub-periods for the two other techniques is already limited by their design: 23 sub-periods for the sliding-window SST and four for the 4-sub-period DSST.

Fig. 3 shows the scatter plots generated by the MATLAB® code with the different sampling techniques according to the differences in mean annual temperature and precipitation between the validation and calibration sub-periods (ΔT and ΔP). The figure allows the spread of the sample provided by each sampling technique to be evaluated in terms of ΔT and ΔP.

Fig. 3a shows the sample offered by the sliding-window SST technique when applied to the five study catchments. It provided 1 495 possible validation exercises for 115 (23 × 5 basins) calibration exercises. The differences between the different sub-periods in mean precipitation ranged from −20% to +25% and the differences in temperature ranged from −1.8 °C to +1.8 °C. When looking at the random bootstrap SST technique (Fig. 3b), it provided 5800 possible validation exercises for a total of 500 calibration (100 × 5 basins) exercises. The differences between the different sub-periods in mean precipitation ranged from −35% to + 50%, and in temperature from −1.4 °C to +1.4 °C. The 4-sub-period DSST (Fig. 3c) provided 60 possible validation exercises from 20 (4 × 5 basins) calibration sub-periods. The differences in mean precipitation obtained ranged from −40% to +60%, and in temperature from −2 °C to +2 °C. Like the random bootstrap SST, the proposed GDSST (Fig. 3d) provided 9 320 possible validation exercises for a total of 500 calibration (100 × 5 basins) exercises. However, the differences in mean precipitation obtained ranged from −45% to +80%, and in temperature from −2 °C to +2 °C. It should be noted the limited redundancy between the different sampling techniques. In fact sub-periods identified by one approach, are not identified by another. This can be explained by the high number of possible sub-periods (around six million sub-period per catchment) compared the limited number of sampled sub-periods generated by each of the four split-sample methods (4–100 sub-periods per catchment).

Although the sliding-window SST technique provided numerous validation exercises, the
differences in P (ΔP) are less contrasted than those offered by the three other techniques. The
sliding-window technique thus appears to depend too much on the historical climate trends to detect
extremely contrasted sub-periods for calibration. Using this method, Coron et al. [2] found well
contrasted precipitation in southeast Australia. However, the authors reported precipitation trends
that contributed to obtain a significant contrast in precipitation characteristics between different
periods. In northern Tunisia, continuous sliding periods were unable to provide sufficiently contrasted
periods because there was no trend in precipitation during the historical study period, as shown by
Dakhlaoui et al. [4]. In addition, the study area presents high inter-annual precipitation variability (see
also [4]). Using continuous sub-periods thus smooths the average precipitation in the sub-periods,
thereby reducing the climate contrast between them. However, this is not the case for temperature,
for which the sliding-window SST technique provided significant differences in T (ΔT) due to the
<figure><img src="image_9.png" /><figcaption>(b) Random bootstrap SST</figcaption></figure>

<figure><img src="image_10.png" /><figcaption>(c) 4-sub-period DSST</figcaption></figure>

<figure><img src="image_11.png" /><figcaption>(d) General DSST</figcaption></figure>

<figure><img src="image_12.png" /></figure>

**Fig. 3.** The validation exercises according to four split-sample methods applied over a 30-year reference period (1970–2000) in the five studied basins: (a) sliding-window SST; (b) random bootstrap SST; (c) 4-sub-period DSST; and (d) the proposed General DSST. ΔT and ΔP represent respectively the differences in mean annual temperature and the relative difference in annual precipitation between the calibration and validation sub-periods.
increasing temperature trends in northern Tunisia over 1970–2000 [4]. The random bootstrap SST technique provided an important number of validation exercises (5 800). However it led to limited differences in *T* (Δ*T*) and a poor distribution of the sample with high concentration in the center of the figure, where there is the least significant contrast to test model parameter transferability. The 4-sub-period DSST provided more contrasted Δ*T* and Δ*P* than the sliding-window and random bootstrap SST. Indeed it is based on a sampling technique generating highly climate-contrasted sub-periods. However, although it explored contrasted climatic conditions in the historical period, the technique provides very few insights into moderate Δ*T* and Δ*P* compared to the other techniques. The oriented bootstrap of the GDSST provided more validation exercises than the random bootstrap SST, although both techniques were based on the same number of calibration exercises (500). This can be explained by the fact that the oriented bootstrap favours the selection of independent sub-periods by reducing overlap between them. In addition, the GDSST provided a better spread of validation periods. Indeed, contrary to the random bootstrap technique in which the validation exercises were concentrated in the zone of Δ*T* and Δ*P* near 0, the sample provided by the GDSST technique was more concentrated at the extremes Δ*t* and Δ*P*, which are the most contrasted sub-periods to test the parameter transferability. Hence, the differences in mean precipitation and temperature between the different sub-periods ranged respectively from -45% to +80%, and from -2 °C to +2 °C, thus providing a more marked climatic contrast between the calibration and validation periods compared with the previous techniques (see Fig. 3).

It should be noted that the random bootstrap technique theoretically includes all the spread of ΔT and ΔP provided by the other techniques tested. In other words, the theoretical limits of the tested combinations (if all possible combinations were sampled) should be as large as the largest limits provided by all the other techniques. However, the problem is that the application of a bootstrap on all combinations would require excessive computation time and would lead to a very large number (~6 million of 8-year sub-periods) of combinations that could obviously not be tested through cross-validation with hydrological models. The proposed GDSST has the advantage to be more effective: with only a limited number of calibration exercises (100), it provides a large number of sub-periods from similar to contrasted conditions in terms of precipitation and temperature, while ensuring that the most climatically contrasted sub-periods are sampled.

Conclusion

We present in this paper the MATLAB® code and an application example of the GDSST proposed by Dakhlaoui et al. [5]. The code allows to user to generate subperiods and corresponding calibration-validation exercises by the new GDSST and three benchmark Split Sample Tests. The code allows the user to visualize the generated subperiods in a scatter plot representing the calibration-validation exercises expressed in term of the differences in mean annual temperature and precipitation between the validation and calibration sub-periods (Δ*T* and Δ*P*). This figure allows evaluating the spread of the sample provided by the sampling technique, before to use it in a DSST exercise. In the provided application example, the GDSST was compared to three other existing techniques to select sub-periods over a 30-year past period on a set of five basins under semi-arid conditions in northern Tunisia. We showed that the GDSST outperformed the other split-sample techniques by providing a large number of sub-periods from similar to contrasted conditions in terms of precipitation and temperature, while ensuring that the most climatically contrasted sub-periods are sampled. This technique thus allows parameter transferability to be tested under wide ranges of climate conditions, which is a key step to assess robustness of hydrological models under past and future climate variability. The users can implement the new and the benchmark split-sample approaches with their own data, by adapting the input file to their data.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
## Acknowledgments

This work is part of the Postdoc research of the first author at the HSM laboratory, Montpellier France, which was supported by an Erasmus Mundus Alyssa mobility scholarship. It was also carried out as part of the ENVI-Med CLIHMag (*Changement cLimatique et Impacts Hydrologiques au Maghreb*) project funded by the program INSU-MISTRALS (2014–2015). The first author have benefited also of Erasmus+ MIC short stay support in 2018. The authors are grateful to INM (*Institut National de la Météorologie*) and to DGRE (*Direction Générale des Ressources en Eau*) in Tunisia for providing the necessary hydro-climatic data for this study.

## Supplementary materials

Supplementary material associated with this article can be found, in the online version, at doi:10.1016/j.mex.2020.101008. A GDSST-package.zip file containing the Matlab script to run the three benchmark SSTs and the proposed GDSST, as well as an application example on five northern Tunisian catchments, can be downloaded from these links: https://fr.mathworks.com/matlabcentral/fileexchange/78856-a-general-differential-split-sample-test-gdsst-package or https://drive.google.com/open?id=17S4QqC8AlBVis58aDaNpRA0Wc283wu4y.

## References

[1] R. Arsenault, F. Brissette, J.-L. Martel, The hazards of split-sample validation in hydrological model calibration, J. Hydrol. 566 (2018) 346–362, doi:10.1016/j.jhydrol.2018.09.027.

[2] L. Coron, V. Andréassian, C. Perrin, J. Lerat, J. Vaze, M. Bourqui, F. Hendrickx, Crash testing hydrological models in contrasted climate conditions: an experiment on 216 Australian catchments, Water Resour. Res. 48 (2012) W05552, doi:10.1029/2011WR011721.

[3] L. Coron, in: Les Modèles Hydrologiques Conceptuels Sont-ils Robustes Face à un Climat en Évolution ?, ISIVE, AgroParisTech, 2013, p. 364.

[4] H. Dakhlaoui, D. Ruelland, Y. Tramblay, Z. Bargaoui, Evaluating robustness of conceptual rainfall-runoff models under climate variability in northern Tunisia, J. Hydrol. 550 (2017) 201–217, doi:10.1016/j.jhydrol.2017.04.032.

[5] H. Dakhlaoui, D. Ruelland, Y. Tramblay, A bootstrap-based differential split-sample test to assess the transferability of conceptual rainfall-runoff models under past and future climate variability, J. Hydrol. Amst. (2019), doi:10.1016/j.jhydrol.2019.05.056.

[6] P.C. Mahalanobis, On the generalised distance in statistics, Proc. Natl. Inst. Sci. India 2 (1) (1936) 49–55.