# Individual and combined effects of design and inter-operator variability on stream ecological assessment

Albin Meyer, Elise Billoir, Virginie Archaimbault, Cédric P Mondy, Philippe Usseglio-Polatera

<figure><img src="image_1.png" /></figure>

## To cite this version:

Albin Meyer, Elise Billoir, Virginie Archaimbault, Cédric P Mondy, Philippe Usseglio-Polatera. Individual and combined effects of design and inter-operator variability on stream ecological assessment. Science of the Total Environment, 2025, 974, pp.179218. 10.1016/j.scitotenv.2025.179218. hal-05011887

HAL Id: hal-05011887

https://hal.univ-lorraine.fr/hal-05011887v1

Submitted on 31 Mar 2025

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_3.png" /></figure>

# Individual and combined effects of design and inter-operator variability on stream ecological assessment

Albin Meyer<sup>a,b,*</sup>, Elise Billoir<sup>b</sup>, Virginie Archaimbault<sup>c</sup>, Cédric P. Mondy<sup>d</sup>, Philippe Usseglio-Polatera<sup>b</sup>

<figure><img src="image_4.png" /></figure>

## HIGHLIGHTS

* We assessed the reliability of a biotic index used for stream ecological assessment

* We studied the effect of individual and combined sources of uncertainty

* Inter-operator variability was the main source of uncertainty

* The uncertainty related to the index construction methodology remained very limited

* A robust index construction is mandatory for the reliability of biotic indices

## GRAPHICAL ABSTRACT

<figure><img src="image_5.png" /></figure>

## ARTICLE INFO

Editor: Ouyang Wei

**Keywords:**

Bioindication

Biomonitoring

Multimetric index

Large scale

Routine survey

## ABSTRACT

In the context of long-term monitoring of stream ecological quality at large spatial extent, it is important to assess the overall reliability of a biotic index and to mitigate its uncertainty to ensure the effectiveness of stakeholder assessments and decisions. Only few studies have focused on the uncertainties associated to both the design of the method used to assess ecological status and the different steps of biological data acquisition. In this study we investigated the uncertainty associated with the design of a multimetric index (i.e., the uncertainty related to index parameter values because of the development dataset characteristics and calculation method), as well as the field data uncertainty because of inter-operator variability (i.e., associated with the sampling, sorting and identification steps of the applied protocol). We examined how the combined sources of uncertainty propagate to the index values and the final ecological assessment of water bodies. For this study, we focused on the I₂M₂, a multimetric index based on benthic macroinvertebrate assemblages and used to assess the ecological quality of French wadeable streams in the context of the European Water Framework Directive. We have shown that inter-operator variability was always the main source of uncertainty. The uncertainty related to the index design remained very limited. This study has highlighted the importance of robust index design combined with standardisation of the field and laboratory steps for the overall reliability of a biotic index, especially for long-term, spatially extensive monitoring of stream ecological health.

<footnote><sup>a</sup> INRAE, EABX, ECOVEA, F-33612 Cestas, France</footnote>

<footnote><sup>b</sup> Université de Lorraine, CNRS, LIEC, F-57000 Metz, France</footnote>

<footnote><sup>c</sup> University of Paris-Saclay, INRAE, UR HYCAR, F-92761 Antony, France</footnote>

<footnote><sup>d</sup> Office Français de la Biodiversité, Direction Régionale d'Ile-de-France, F-94300 Vincennes, France</footnote>

<footnote>* Corresponding author at: INRAE, EABX, ECOVEA, F-33612 Cestas, France.</footnote>

<footnote>E-mail address: albmey@hotmail.com (A. Meyer).</footnote>
# 1. Introduction

Human activities are increasingly impairing water bodies all around the world, leading to water quality degradation and/or habitat alteration (Dudgeon, 2019). These multiple sources of deterioration are a major factor explaining the loss of biodiversity, ecological functions and ecosystem services in aquatic ecosystems (Grizzetti et al., 2019; Teichert et al., 2017; Vaughn, 2010). In response, many countries have enacted laws to identify and address, when feasible and necessary, all these sources of deterioration (e.g., the Water Framework Directive in Europe, and the Clean Water Act in the US). In Europe, the Water Framework Directive (WFD; European Council, 2000) requires the status of surface water to be assessed taking into account both (i) the chemical and (ii) the ecological characteristics of water bodies, by studying: (i) a set of priority substances and (ii) certain hydromorphological (hydrological regime, river continuity, morphological conditions), physico-chemical (e.g., nutrients, salinity, pH, oxygen balance, specific synthetic or non-synthetic pollutants) and biological quality elements (BQEs; e.g., invertebrates, diatoms, macrophytes, phytoplankton or fish), respectively. The finality of the BQE-based assessment of a given water body is to assign it a quality class (high, good, moderate, poor or bad) summarizing its ecological status, based on a selection of robust taxonomic and/or functional metrics. The WFD stipulates that the ecological status assessment of water bodies (for example, by means of biotic indices) must take account of the specific BQE characteristics: (i) the taxonomic composition and taxon abundances of the BQE monitored, sometimes their pollution sensitivity (invertebrates, fish) and/or their taxonomic diversity (invertebrates), and (ii) the reference conditions specific to the type of water body, defined and used as benchmarks. In the WFD framework, the results of monitoring programmes must be provided with estimates of their levels of confidence and precision, given the biotic index used (European Council, 2000; Annex V §1.3). However, while the WFD has triggered the development of many biotic indices for the biological assessment of water bodies (Birk et al., 2012), few studies have examined the various sources of uncertainty associated with the successive steps leading to the final assessment of the water body ecological status. Several studies have focused specifically on the field sampling step (Wiederkehr et al., 2016), the laboratory/identification step (Dromph et al., 2013; Haase et al., 2006; Wiederkehr et al., 2015), the individual metric selection, sensitivity and aggregation (Buffagni et al., 2020; Zucchetta et al., 2020) and/or the definition of reference values (Marzin et al., 2014). They have rarely investigated the combined effects of these different sources of uncertainty (but see Clarke, 2013, and Thackeray et al., 2013, as examples of studies of combinations of sources of uncertainty). This aspect is however essential to ensure both optimal reliability in water body ecological assessment and maximal efficiency in management decisions for aquatic ecosystem conservation and restoration (Bark et al., 2013; Ounanian et al., 2018; Vadas et al., 2022). Reyjol et al. (2014) even identified the evaluation of uncertainty in ecological assessment as one of the top ten research priorities for WFD-compliant tools and indices.

The level of confidence and precision of ecological assessments, when assigning an ecological quality class (from “bad” to “high”) based on the value of a biotic index, is directly linked to the robustness of this index in the face of the various sources of uncertainty that can influence the result of its calculation. These sources of uncertainty can be aggregated in four main categories related to:

1. the sampling method design; i.e., the choice of the BQE to be sampled (Johnson et al., 2006) and the choice of protocols for the field (e.g., sampling method and equipment) and the laboratory steps (e.g., subsampling, sorting, and taxonomic identification) (Clarke et al., 2006; Clarke and Hering, 2006; Kelly et al., 2009; Lorenz and Clarke, 2006; Ramos-Merchante and Prenda, 2017; Vlek et al., 2006);

2. the environmental variability influencing the distribution, abundance and composition of the monitored BQE: natural changes at

different spatial (from mesohabitats – sensu Armitage et al., 1995, and Pardo and Armitage, 1997; i.e., “visually distinct units of habitat within the stream, recognizable from the bank and with an apparent physical uniformity” (e.g., in terms of substrate and current velocity) – to watershed) and temporal (from one-off events to multi-year trends) scales, including hydrological and hydromorphological conditions or habitat availability (e.g., in terms of current velocity and bottom substrate diversity) (Clarke and Hering, 2006; Kelly et al., 2009; Šporka et al., 2006; Wiederkehr et al., 2016);

3. the index design (Hawkins et al., 2010) and calculation; e.g., the learning dataset size and composition, and the values of some parameters. For example, reference values (i.e., index values defining reference conditions), which are often used to standardise biotic index values, can be predicted (Hawkins et al., 2010; Logez et al., 2019), derived from historical data (Krause-Jensen et al., 2005) or directly derived from the learning dataset (Mondy et al., 2012). In the latter case, uncertainties may arise from the way in which reference conditions have been defined (Santos et al., 2021; but see Pardo et al., 2012, and Stoddard et al., 2006, for guidelines);

4. the inter-operator variability during field sampling (e.g., via the choice of sampled mesohabitats), organism identification (via taxon omission, misidentification, miscounting and mislabeling), and even bio-informatics steps (via spelling mistakes or errors in abundance data entry).

Most of the recent studies on the uncertainty of biotic index values have focused mainly on inter-operator variability, as this appeared to be a major source of uncertainty in the bioassessment context (e.g., Gobeyn et al., 2016; Wach et al., 2019; Wiederkehr et al., 2015). Inter-operator variability depends on the experience and skills of the operators (Gobeyn et al., 2016; Loga and Wierzchołowska-Dziedzic, 2017; Metzeling et al., 2003; Wiederkehr et al., 2015). Inter-operator variability can lead to variable rate of taxon misidentification and/or cause variation in taxon abundance/cover estimations (Wiederkehr et al., 2015), which can ultimately change the ecological quality class allocated to water bodies based, for example, on macroinvertebrates or macrophytes (Haase et al., 2006). But inter-operator variability also depends on the monitored BQE and on the robustness of the biotic index (e.g., Loga and Wierzchołowska-Dziedzic, 2017). Mascaró et al. (2013) and Thackeray et al. (2013) concluded that the variability and uncertainty of macrophyte-based and phytoplankton-based index values were more related to spatial variations and variability (between and within waterbodies, respectively for each study) than to inter-operator variability. Wach et al. (2019) also showed that inter-operator variability resulted in significant differences in the species list, richness and abundance of diatom and macrophyte assemblages, but not in the values of related indices and in the final ecological assessment of the sampled streams. In France, several projects were launched to assess uncertainty related to the IBD diatom-based index (Wach et al., 2019), the IBMR macrophyte-based index (Wach et al., 2019), the IPR+ and the ELFI fish-based indices (Logez et al., 2019; Marzin et al., 2014). Several of these studies have focused on how the variability of the reference values used to calculate a WFD-compliant index can influence the ecological assessment.

In this study, we aimed at investigating the individual and combined effects of uncertainty of a WFD-compliant multimetric index values in relation with its design (i.e. parameter variability) and data collection process (i.e. inter-operator variability). We (i) modelled the uncertainties linked to inter-operator variability (when several certified operators, but from different backgrounds, apply the complete protocol on a given site) or linked to the different index parameters, (ii) combined them, and finally (iii) propagate these uncertainties to the index values and final ecological assessment.

For this study we focused on the I₂M₂, an invertebrate-based multi-metric index that complies with the WFD and is used routinely to assess the ecological quality of French wadeable streams (Mondy et al., 2012).
This index is calculated using data from standardised field sampling and standardised sorting, identification and counting of invertebrate samples in the laboratory, so that it can be used consistently across the country. It has been successfully intercalibrated with other European invertebrate-based indices (LIEC and AFB, 2018). The sampling protocol (AFNOR, 2009; revised in 2016) also gives recommendations on hydrological conditions and sampling period to limit temporal variability and clearly describes how to select the mesohabitats sampled to optimize comparisons between sites. The residual intra-site spatial variability can therefore be considered as part of the inter-operator variability (e.g., Wiederkehr et al., 2016).

We modelled the uncertainties linked to inter-operator variability using the results of a previous study in which several operators independently applied the complete protocol (for the sampling, sorting and identification steps of the protocol) to 24 different streams, both in 2009 and 2010 (Archaimbault et al., 2012). Uncertainty related to the index design was modelled taking into account the development dataset characteristics (e.g. size, number of reference sites) in index parameters calculations.

## 2. Material & methods

### 2.1. The I₂M₂ index

The I₂M₂ index combines the values of five different metrics, namely the Shannon-Weaver diversity index (Shannon, 1948), the Average Score Per Taxon (ASPT; Armitage et al., 1983), the total taxonomic richness and the relative frequencies of individuals from polyvoltine (i.e., taxa exhibiting at least two successive generations per year; %Polyvoltinism) and ovoviviparous (%Ovoviviparity) species in the sampled assemblage (Mondy et al., 2012). The ASPT is an index, that can vary from 0 (most tolerant community) to 10 (most sensitive community), indicating the average sensitivity of the invertebrate families observed in the sample to organic pollution (Armitage et al., 1983). This combination of metrics has been selected based on its ability to efficiently discriminate Impaired River Reaches (IRRs) from Least Impaired River Reaches (LIRRs) over 17 stressors, including 10 stressors linked to water quality (e.g., organic pollution and nutrient inputs) and seven stressors related to habitat degradation (e.g., stream physical alterations such as straightening and damming) (Mondy et al., 2012). The final I₂M₂ value, expressed as an Ecological Quality Ratio (EQR; normalized between 0 and 1), is the average value of 17 sub-indices (i₂m₂), each calculated for a given stressor. The I₂M₂ value is used to assign one of five ecological quality class (i.e., high, good, moderate, poor and bad) to stream reaches, based on five different sets of boundary values adapted to the common EU intercalibration types that have been defined in line with the typological requirements of the Annex II of the WFD (European Council, 2000). Each common EU intercalibration type comprises several stream types from the French typology, defined on the basis of stream size and hydro-ecorégion (Wasson et al., 2002; Billoir et al., 2015; also see §2.2.4).

Each i₂m₂ sub-index is equal to the weighted average of the EQR values of the constituent metrics. For a given stressor, the weights (WGHT) are equal to the discrimination efficiency of the metrics for this stressor (DE values in Mondy et al., 2012). Thus, the formula used to calculate the i₂m₂ sub-index for the stressor “s” (i.e. i₂m₂s) is:

$$i_2 m_2^s = \frac{\sum_m (WGHT_m^s \times EQR_m)}{\sum_m (WGHT_m^s)} \quad (1)$$

with WGHT_m^s the discrimination efficiency (sensu Ofenböck et al., 2004) of the metric “m” for the stressor “s”, and EQR_m the value of the metric “m” expressed as an EQR. Metrics can either be considered as “increasing” (i.e., taxonomic diversity, ASPT and total richness) or “decreasing” (i.e., %Polyvoltinism and %Ovoviviparity) with an increase in ecological quality.

EQR values are calculated based on the following formula:

$$EQR_m = (\text{OBS}_m - \text{WORST}_m)/(\text{BEST}_{m}^{\text{typ}} - \text{WORST}_m) \quad (2)$$

with: (i) OBS_m, the raw value of the metric “m”, (ii) WORST_m, the 5th (increasing metrics) or the 95th (decreasing metrics) percentile of the distribution of the metric “m” in the IRRs at the national scale, and (iii) BEST_m^typ, the 95th (increasing metrics) or the 5th (decreasing metrics) percentile of the distribution of the metric “m” in the LIRRs corresponding to the stream type “typ” of the tested site (Mondy et al., 2012).

The WGHT, BEST and WORST values currently in use (I₂M₂ version 1.0.6; https://seee.eaufrance.fr/) were defined on a learning dataset including the faunal lists of 10,074 stream sampling events (SSEs), performed during the 2004–2013 period (Usseglio-Polatera et al., 2016). This dataset was completed by a second one providing the level of impairment (LIRR or IRR) of all the available SSEs. This impairment status was defined following the procedure described by Mondy et al. (2012). LIRRs were selected first based on the evaluation of their water quality and habitat degradation level using available abiotic data (183 environmental parameters taken into account). This selection was further validated based on a reduced set of biological metrics to ensure that sampled communities were not impaired by other stressors than the 17 investigated. All the other river reaches were thus considered as impaired (IRRs) at least for one stressor.

### 2.2. Uncertainty modelling

We evaluated the uncertainties on OBS, BEST, WORST and WGHT, using either Monte-Carlo simulations for OBS, or a bootstrapping approach for BEST, WORST and WGHT (see Fig. 1 and the following subsections). Monte-Carlo simulations were used for OBS, as they allowed us to model uncertainties for metric values based on observed inter-operator variability (cf. §2.2.1.), whereas a bootstrapping approach was preferred for BEST, WORST and WGHT, as the variability of these values was linked to the dataset used during index design (Mondy et al., 2012; cf. §2.2.2.). We investigated the impact of those uncertainties on the final ecological assessment of the 10,074 SSEs before-mentioned, considering all the possible combinations (from one to four) of these sources of uncertainty:

(i) Single sources: OBS, BEST, WORST and WGHT;

<figure><img src="image_6.png" /><figcaption>Fig. 1. Summary of the workflow used to estimate and propagate uncertainties from all the studied sources (i.e., OBS, BEST, WORST and WGHT; see text for more information on acronyms) for the invertebrate-based I₂M₂ index (Mondy et al., 2012). m = metric, s = stressor, typ = stream type.</figcaption></figure>
(ii) Dual sources: OBS+BEST, OBS+WORST, OBS+WGHT, BEST+WORST, BEST+WGHT and WORST+WGHT;

(iii) Triple sources: OBS+BEST+WORST, OBS+BEST+WGHT, OBS+WORST+WGHT and BEST+WORST+WGHT;

(iv) All four sources: OBS+BEST+WORST+WGHT.

### 2.2.1. Inter-operator uncertainties associated with raw metric values (OBS)

The results of a dedicated field survey (Fig. 2; Archaimbault et al., 2012) were used to assess the inter-operator variability of the raw values of the five individual metrics (OBS<sub>m</sub>) of the I<sub>2</sub>M<sub>2</sub> index (Fig. 1). Two to four different operators applied the standardised protocols of the current methodology used for obtaining the raw faunal data needed for the I<sub>2</sub>M<sub>2</sub>

calculation (norms T90–333 for field sampling (AFNOR, 2009; revised in 2016) and T90–388 for invertebrate sorting and identification (AFNOR, 2010; revised in 2020)) on 24 streams in both 2009 and 2010 (Table 1). One site was sampled per stream. Each operator, applying the standardised field protocol, performed eight samples in the dominant substrate types (i.e., with an individual share of at least 5 % coverage at the site scale) and four samples in the marginal substrate types (i.e., with an individual share of <5 % coverage), using a standardised Surber net (sampling area = 0.05 m², mesh size = 500 µm). After sorting, the invertebrates were identified to genus level, with the exception of Oligo-gochaeta (identified as such), Diptera (family) and certain Trichoptera and Coleoptera (at subfamily or tribe level). The field and laboratory steps are described in more details in the standards, which makes it

<figure><img src="image_7.png" /><figcaption>**Fig. 2.** Location of the river reaches surveyed during the field assessment of the effect of inter-operator variability on the raw values of the I₂M₂ and its individual metrics (Archaimbault et al., 2012). Black delineations indicate the main French hydro-ecoregions (Wasson et al., 2002), with the background colors corresponding to the three European geographical intercalibration groups (in light blue: Central/Baltic, in light green: Alpine, and in light orange: Mediterranean; van de Bund, 2009). Details about the field sampling of river reaches are in Table 1.</figcaption></figure>
possible to standardise the sampling and sorting effort.

Operators were from different backgrounds, either from the French Biodiversity Agency (OFB), from a Research Agency (INRAE), or from private expert companies, steadily contracted by Water Agencies for the routine monitoring of French wadeable streams. All the operators exhibited a similar skill level and they were certified for the application of these standards (NF EN ISO/CEI 1702 by COFRAC; https://www.cof.rac.fr/en/customer-area/). A typical group of four operators (76.6 % of the "site × date" sampling events of this dedicated field survey) included two hydrobiologists from the INRAE and two hydrobiologists from the OFB or a private expert company. For a given SSE, each operator carried out the two steps necessary for the calculation of the I₂M₂, i.e., the field step (i.e. habitat description and sampling), and the laboratory step (i.e., invertebrate sorting, taxonomic identification and counting).

One hundred and seventy-two faunal lists (i.e., taxa abundance and composition; respectively, 86 in 2009 and 86 in 2010) were provided by the sampling design, with one list per combination of "site × date × operator". For each "site × date", we calculated the mean value of each I₂M₂ constitutive metric, based on the values obtained by the different operators: four values for 76.6 % of the "site × date" sampling events, only two or three values for the remaining 23.4 % of the sampling events (see Table 1). Then, we calculated the deviation between the inter-operator mean and the individual values provided by operators (Fig. 1). Because we did not observe any significant difference between inter-operator deviations in 2009 vs. 2010 (Mann-Whitney test; *p* > 0.05 for each of the five metrics), we fitted a Gaussian distribution (with a mean set to zero) on the 172 calculated deviations to (i) model inter-operator variability and (ii) estimate the corresponding standard deviation (sd<sub>m</sub>), for each of the five I₂M₂ metrics (Fig. 1).

The results (sd<sub>m</sub>) of this inter-operator study were then used to feed the models used for simulating inter-operator variability (i.e., the uncertainty associated with OBS values). For each of the 10,074 SSEs, random drawings in the Gaussian distributions of the standard deviations of each metric were run following a Monte-Carlo approach, and then added to the raw metric values of the 10,074 SSEs to simulate inter-operator variability (Fig. 1). We repeated this step 10,000 times for each metric and each SSE (Fig. 1).

### 2.2.2. Uncertainties associated with BEST, WORST and WGHT values

Three hundred and thirty BEST values were calculated (i.e., for each combination of metric (n = 5) and type of stream (n = 66)), using only sampling events performed on the LIRRs of each stream type (Fig. 1). The number of SSEs performed on LIRRs ranged from 6 to 135 per stream type, with an average of 35.4. When the uncertainty associated with the BEST values was to be accounted for, bootstrap resampling with replacement in the learning dataset was used to generate 10,000 new datasets for each stream type (Fig. 1). BEST values were recalculated for each new dataset. When the uncertainty associated with the WORST values was to be accounted for, a similar approach was used, with the difference that WORST values were defined using data from all the IRRs (n = 7738), independently of stream type.

Weight values (WGHT) characterize each metric (n = 5) for each stressor (n = 17). Each of these 85 WGHT values has been calculated on a dataset including all the LIRRs (n = 2336) and the IRRs impacted by the corresponding stressor, independently of stream type and the intensity of other stressors. When the uncertainty associated to WGHT values was to be accounted for, bootstrapping simulations with replacement were used to generate new datasets for each stressor, using – each time – the set of IRRs impaired by this stressor. WGHT values were recalculated on each new dataset.

### 2.2.3. Uncertainty propagation and risk of misclassification

The final uncertainty was assessed by calculating, for each of the 10,074 SSEs and each combination of uncertainty sources, 10,000 simulated I₂M₂ values. Depending on the combination of uncertainty sources considered, each of the simulated I₂M₂ values corresponds to Monte-Carlo simulated OBS values for each metric (§2.2.1) and/or one (sub-)set of bootstrap simulated values for BEST, WORST and/or WGHT (§2.2.2) (Fig. 1).

For each combination of uncertainty sources, we calculated the probability for each SSE to be in a given ecological class as the proportion of the 10,000 simulated values allocated to this ecological class (cf. Clarke, 2013; Logez et al., 2019; Marzin et al., 2014). We also calculated for each SSE the sum of the probabilities of this SSE to be in another ecological quality class than the observed one (i.e., without taking into account any sources of uncertainty); for example, the cumulated proportion of the simulated values allocated to either "high",

<caption>**Table 1**</caption>

Characteristics of the streams surveyed in 2009 and 2010 during the field sampling campaigns regarding the assessment of inter-operator variability of raw metric values (Archaimbault et al., 2012).

<table><thead><tr><th rowspan="2">Code</th><th rowspan="2">Stream</th><th rowspan="2">Station</th><th colspan="2">Number of operators</th><th colspan="2">Coordinates</th></tr><tr><th>2009</th><th>2010</th><th>Longitude</th><th>Latitude</th></tr></thead><tbody><tr><td>ALL</td><td>ALLIER</td><td>LANGOGNE</td><td>4</td><td>2</td><td>3.8657243</td><td>44.7258211</td></tr><tr><td>ANC</td><td>ANCEDUNORD</td><td>BEAUZAC</td><td>0</td><td>4</td><td>4.1141145</td><td>45.2855933</td></tr><tr><td>AUS</td><td>AUSTREBERTHE</td><td>VILLERS-ECALLES 1</td><td>4</td><td>4</td><td>0.9387338</td><td>49.5322570</td></tr><tr><td>BEU</td><td>BEUVE</td><td>BAZAS</td><td>2</td><td>3</td><td>-0.1610835</td><td>44.4743108</td></tr><tr><td>CHE</td><td>CHER</td><td>CHAMBONCHARD</td><td>2</td><td>4</td><td>2.5662830</td><td>46.1442486</td></tr><tr><td>CLE</td><td>CLEURIE</td><td>CLEURIE</td><td>4</td><td>4</td><td>6.7033492</td><td>48.0578953</td></tr><tr><td>COL</td><td>COLAGNE</td><td>MARJEVOLS</td><td>4</td><td>4</td><td>3.2456655</td><td>44.4914266</td></tr><tr><td>DES</td><td>DESGES</td><td>DESGES</td><td>4</td><td>4</td><td>3.4587526</td><td>45.0156597</td></tr><tr><td>DIV</td><td>DIVES</td><td>OMMOY</td><td>4</td><td>4</td><td>-0.0165291</td><td>48.8566306</td></tr><tr><td>DOU</td><td>DOUX</td><td>LABATIE D ANDAURE</td><td>4</td><td>4</td><td>4.4932249</td><td>45.0285417</td></tr><tr><td>EAU</td><td>EAULNE</td><td>ENVERMEU</td><td>4</td><td>2</td><td>1.2782757</td><td>49.8865361</td></tr><tr><td>ESC</td><td>ESCOURCE</td><td>SAINT-PAUL-EN-BORN</td><td>4</td><td>4</td><td>-1.1802806</td><td>44.2304281</td></tr><tr><td>GLA</td><td>GLAND</td><td>SAINT-MICHEL 2</td><td>4</td><td>4</td><td>4.1589290</td><td>49.9219204</td></tr><tr><td>HER</td><td>HERAULT</td><td>ASPIRAN</td><td>4</td><td>2</td><td>3.4712857</td><td>43.5680507</td></tr><tr><td>JOY</td><td>JOYEUSE</td><td>PREVERANGES</td><td>3</td><td>3</td><td>2.2872416</td><td>46.4712387</td></tr><tr><td>LER</td><td>LERGUE</td><td>BRIGNAC</td><td>4</td><td>4</td><td>3.4773550</td><td>43.6281932</td></tr><tr><td>LEY</td><td>LEYSSE</td><td>BOURGET DU LAC</td><td>4</td><td>4</td><td>5.8621465</td><td>45.6498339</td></tr><tr><td>LOT</td><td>LOT</td><td>MENDE</td><td>4</td><td>4</td><td>3.4564895</td><td>44.4807792</td></tr><tr><td>OIS</td><td>OISE</td><td>ERLOY</td><td>4</td><td>4</td><td>3.8283576</td><td>49.9102858</td></tr><tr><td>REM</td><td>REMEL</td><td>NEUNKIRCHEN-LES-BOUZONVILLE</td><td>4</td><td>4</td><td>6.5671221</td><td>49.3418197</td></tr><tr><td>ROU</td><td>ROUVRE</td><td>FAVEROLLES</td><td>4</td><td>4</td><td>-0.3081024</td><td>48.6706836</td></tr><tr><td>SAU</td><td>SAULDRE</td><td>MENETREOL-SUR-SAULDRE</td><td>3</td><td>3</td><td>2.3047559</td><td>47.4466963</td></tr><tr><td>SAY</td><td>SAYE</td><td>GRIMARD</td><td>4</td><td>4</td><td>-0.2918917</td><td>45.0141781</td></tr><tr><td>SEG</td><td>SEGRE</td><td>BOURG MADAME</td><td>4</td><td>4</td><td>1.9409139</td><td>42.4295145</td></tr></tbody></table>
"good", "poor" and "bad" classes, when the observed class was "moderate". Hereafter, we call this relative frequency of quality class mismatches the "Mismatch Frequency" (MF).

### 2.2.4. Modelling confidence of class allocation

Following the method described by Kelly et al. (2009) and used by Wach et al. (2019), we modelled the confidence of being in each ecological quality class for any SSE. Our objective was to obtain a general model, which provides the confidence of the ecological quality class allocated to a SSE based on its I₂M₂ value. To summarize the approach, a polynomial function was fitted to model the relationship between the observed I₂M₂ value of each SSE and the standard deviation of the corresponding simulated values (accounting for all the sources of uncertainty) (Fig. S1 in the Supplementary Material). This fitted polynomial function was then used to model the standard deviation of the five different sets of I₂M₂ boundary values currently being applied in France. The five sets of boundary values correspond to the five groups of European intercalibration common stream types found in France and at the scale of which European biotic indices have been intercalibrated (i. e., 12 types belonging either to the Alpine, Mediterranean or Central-Baltic types; see van de Bund, 2009). Finally, the modelled standard deviations were used to calculate the probability of being in each ecological quality class for any SSE, following the method detailed in Wach et al. (2019).

All statistical analyses were performed with the software R (R version 3.6.1; R Core Team, 2020). The polynomial regressions were done with the function *nls*, available in the package *nls2* (version 0.2; Grothendieck, 2013). The Gaussian distribution fittings were done with the function *fitdist* from the package *fitdistrplus* (version 1.0–14; Delignette-Muller et al., 2020). The scripts and data are available online (https://github.com/Albin-Meyer/Uncertainty-I2M2).

## 3. Results

The results of the field survey have shown that inter-operator variability was different for the five metrics involved in the calculation of the I₂M₂ value (Fig. S2 in the Supplementary Material). The average coefficients of variation of the I₂M₂ individual metrics and index values obtained during this survey were equal to 0.034 (ASPT), 0.053 (%Pol-yvoltinism), 0.088 (total taxonomic richness), 0.116 (Shannon-Weaver index), 0.175 (%Ovoviviparity) and 0.080 (I₂M₂).

Fig. 3 provides the average Mismatch Frequency (MF) for all the possible combinations of sources of uncertainties we accounted for. When considered separately, the uncertainty linked to operators (OBS) was the highest (average MF = 0.120; Fig. 3). In decreasing order of importance, the next sources of uncertainty were related to the BEST values (average MF = 0.042), the WORST values (0.008) and finally the WGHT values (0.002). The average MF of a given combination of sources of uncertainty was mostly defined by the average MF of the highest source of uncertainty, in general the variability related to operators (Fig. 3). Combining all the tested sources of uncertainty (OBS, BEST, WORST and WGHT) led to an average MF equal to 0.129 (Fig. 3).

The BEST values have been defined by stream type. We investigated whether their uncertainty was related to the size (number of LIRR SSEs and number of LIRR sites) of the stream-type specific learning dataset used to calculate the BEST values. Fig. 4 represents, for each stream type, the average MF when accounting for the uncertainty associated with the BEST values against the number of LIRR SSEs (Fig. 4A) and against the number of LIRR sites (Fig. 4B). The average MF exhibited high variation for stream types with less than about 50 SSEs, with average MF ranging from 0.0003 to 0.2098 (Fig. 4A). Average MF stabilized for stream types with more than about 50 SSEs (Fig. 4A). The average MF decreased along an increasing gradient of the number of sites considered as LIRR per stream type (Fig. 4B).

The MF depended also on the ecological status of the SSE (Kruskal-Wallis rank sum test; *p* < 0.001). The MF significantly decreased from

<figure><img src="image_8.png" /><figcaption>Fig. 3. Average Mismatch Frequency (MF) for each possible combination of tested uncertainty sources (OBS, BEST, WORST and/or WGHT), as indicated below the x axis by black points (potentially related by a segment).</figcaption></figure>

the "moderate" status, to the "poor", "good", "bad", and then the "high" status (multiple comparison Conover's test after Kruskal-Wallis, with Holm's adjustment method for *p*; all *p* < 0.01) (Fig. 5).

Fig. 6 provides, for all the SSEs, the probability to be in each of the five ecological quality classes against the I₂M₂ value, taking into account the between quality-class boundary values of this index. The five sub-figures (Fig. 6A to 6E) correspond respectively to the five sets of I₂M₂ boundary values applied in France; i.e., one set per European intercalibration type including some French stream types (Fig. 6F; also see Fig. 2). In general, the confidence is higher for the widest classes (in terms of I₂M₂ value range) than for the narrowest ones (e.g., "good" vs "moderate" in group 5; Fig. 6E).

## 4. Discussion

In the context of long-term monitoring of stream ecological quality at large spatial extent (such as within the European WFD framework), it is important to assess the overall reliability of a biotic index and to mitigate its uncertainty. In this study our objective was to assess (i) the uncertainty associated with the design of a multimetric, WFD-compliant index (the I₂M₂ index) and the uncertainty associated with inter-operator variability during data collection, and (ii) how these combined sources of uncertainty may affect the final ecological assessment.

Hawkins et al. (2010) showed that the method selected for both the definition of reference values and the index development can lead to a proportion of streams classified as altered potentially varying by a factor of two according to indices. In this study, we did not investigate the potential impact on uncertainty related to all the alternatives that could have been selected during the I₂M₂ development. Instead, we looked at how the uncertainties associated with the selected design might affect the index values and the resulting ecological assessment. We specifically tested four sources of uncertainty, linked either to inter-operator variability or to the learning dataset used for the construction of the index (Mondy et al., 2012; Usseglio-Polatera et al., 2016). The overall uncertainty resulting from a given combination of sources of uncertainty appeared to be primarily driven by the highest individual source of variation.
<figure><img src="image_9.png" /><figcaption>**Fig. 4.** A. Relationship between the number of SSEs in LIRRs per stream type and the average Mismatch Frequency (MF) when accounting for the uncertainty associated with the BEST values. B. Relationship between the number of sites considered as LIRRs per stream type and the average MF when accounting for the uncertainty associated with the BEST values. For both figures, dashed red lines are exponential regressions. *N* = 66.</figcaption></figure>

<figure><img src="image_10.png" /><figcaption>**Fig. 5.** Distribution (minimum/Q25/median/Q75/maximum) of the mismatch frequencies for each ecological quality class, when accounting for all the tested uncertainty sources (OBS, BEST, WORST and WGHT). Letters indicate significant differences, based on multiple comparison Conover's test after Kruskal-Wallis rank sum test, with Holm's adjustment method for *p*; all *p* < 0.01.</figcaption></figure>
<figure><img src="image_11.png" /><figcaption>**Fig. 6.** Probability of a stream sampling event (SSE) belonging to one of the five ecological quality classes, taking into account all the sources of uncertainty tested (OBS, BEST, WORST and WGHT). For a given SSE and its associated I₂M₂ value (on the x-axis), there are five vertically aligned points, coloured differently according to the corresponding quality class (blue for “high”, green for “good”, yellow for “moderate”, orange for “poor” and red for “bad”). They provide the probabilities that the SSE belongs to each of these five ecological quality classes. For example, in the first sub-fig. A, SSEs located in the Central-Baltic region, with an I₂M₂ value between 0.443 and 0.665, have (i) a high to very high probability (> 0.5) of belonging to the “good” quality class, (ii) a lower probability (< 0.5) of belonging to the “moderate” or “high” quality class, and (iii) a zero probability of belonging to the “poor” and “bad” quality classes. The black curves correspond to the modelled relationships between the I₂M₂ index values and the degree of confidence in the ecological quality classes assigned (see §2.2.4). The SSEs were divided into five groups (sub-figs. A-E), according to their geographical location in France, taking into account the European intercalibration types of streams, with which different quality class limits (in F) are associated (van de Bund, 2009).</figcaption></figure>

## 4.1. Inter-operator variability

Several studies, including this one, have shown that inter-operator variability was one of the main sources of uncertainty in freshwater

ecological assessment (Clarke, 2009; Gobeyn et al., 2016; Thackeray et al., 2013; Wiederkehr et al., 2015). However, Metzeling et al. (2003) observed that a small number of taxon misidentifications and omissions did not impair the ability of sampled invertebrate assemblages to
discriminate between reference and disturbed sites. Nevertheless, Metzeling et al. (2003) identified taxa at the family level, while the I₂M₂ protocol requires identification at the genus level for most taxa (AFNOR, 2010). Such a difference may help to explain why they observed a limited effect of inter-operator variability.

One of the factors influencing inter-operator variability and partly explaining its major contribution to uncertainty is the natural spatial variability of invertebrate assemblages at mesohabitat scale and its assessment by operators (e.g., Barnes et al., 2013; Beisel et al., 1998, 2000; Boyero, 2003; Buffagni et al., 2020; Wiederkehr et al., 2016). For example, when inter-operator studies involve simultaneous sampling of a given river reach by different operators, they must sample different patches of a given habitat with potentially significant differences in the structure and composition of the macroinvertebrate assemblage (Wiederkehr et al., 2016), because of unconsidered – yet important – factors of distribution such as water depth (Effenberger et al., 2006; Lencioni and Rossaro, 2005), near-bed hydraulic conditions (Alvarez-Cabria et al., 2011; Effenberger et al., 2006; Statzner et al., 1988), substrate clogging (Descloux et al., 2014; Rabeni and Minshall, 1977), algal biomass (Effenberger et al., 2006) or substrate type of the neighbouring patches as potential providers of specific colonizers (Beisel et al., 1998, 2000).

The importance of inter-operator variability on stream ecological assessment may also depend on the monitored BQE (and its associated sampling protocol) and the robustness of the corresponding biotic index (e.g., Loga and Wierzcholowska-Dziedzic, 2017). For example, depth variations and hydromorphological alterations could have higher effects on the values of metrics based on macrophyte or phytoplankton assemblages (Mascaró et al., 2013; Staniszewski et al., 2006; Thackeray et al., 2013) and led to higher risk of water body misclassification than inter-operator variability. Nevertheless, Wach et al. (2019) showed that although inter-operator variability resulted in significant differences in species list, richness and abundance of diatom and macrophyte assemblages, the impact on the values of related indices and on the final ecological assessment was low, as they did not observe significant inter-operator differences in the distribution of index values. Similarly, Clarke (2009) concluded that the inter-operator effect on a macroinvertebrate-based assessment (BMWP; Armitage et al., 1983) was negligible compared to other sources of uncertainty, such as the intrinsic variability of the fauna within a given site.

In this study, we showed that the level of inter-operator variability differed among the five metrics combined in the I₂M₂. The lowest inter-operator variability, obtained for ASPT, could be related to the fact that this metric does not take taxon abundances into account and is calculated on a list of taxa standardised at the family level (Armitage et al., 1983). The highest inter-operator variability was observed for the relative abundance of ovoviviparous organisms in the assemblage. This metric was mainly driven by the abundance of gammarids in invertebrate assemblages, which can greatly vary among habitats and among patches within a given habitat (Dahl and Greenberg, 1996; Dedecker et al., 2005). These results were in accordance with those of Wiederkehr et al. (2016). It should also be noted that the standard (T90–388; AFNOR, 2010) allows for sub-sampling to estimate the abundance of the most abundant taxa in samples. This estimation may increase inter-operator variability on abundance data.

The importance of inter-operator variability on I₂M₂ values was intermediate between that of inter-operator variability of its least variable metrics (ASPT, %Polyvoltinism) and that of its most variable metrics (Shannon-Weaver index, %Ovoviviparity). The variability, and related uncertainty, of a multimetric index depends on the variability of its individual metrics. Therefore we can argue that a multimetric index should include different types of metrics (e.g., taking into account, or not, taxon abundances) to limit its overall susceptibility to inter-operator variability (Loga and Wierzcholowska-Dziedzic, 2017; Marzin et al., 2014). Moreover, the metrics combined in the I₂M₂ exhibit two different response patterns along a gradient of increasing ecological

quality: they increase (taxonomic diversity, ASPT and total richness) or decrease (%Polyvoltinism and %Ovoviviparity) with increasing ecological quality. Combining metrics with different patterns of responses along a gradient of ecological quality can also mitigate inter-operator variability, which is more likely to be additive if all the metrics exhibit the same response pattern (e.g., for a hypothetical multimetric index that would be only composed by metrics positively related to the richness and/or the abundance of pollution-sensitive taxa).

## 4.2. Reference values

The uncertainty associated with reference values, which are necessary to adapt individual metric values to the national stream typology before combining them into a single multimetric index value (e.g., Blocksom, 2003) or to calculate the EQR recommended by the WFD, appeared to be the second source of uncertainty, although it was about three times lower than that related to inter-operator variability. The importance of this source of uncertainty appeared to be related to the size of the dataset used for the definition of I₂M₂ metric reference values. The uncertainty related to the WORST values was low because WORST values were defined on a large data set including all the sampling events performed in Impaired River Reaches (7738 SSEs), whatever the stream type (Usseglio-Polatera et al., 2016). We can also hypothesize that the uncertainty associated with the WORST values was low simply because the invertebrate assemblages found in the heavily impacted river reaches may exhibit similar characteristics on a national scale (e.g., the absence of pollution-sensitive and rare taxa, and an assemblage dominated by common and ubiquitous taxa). In contrast, the 66 BEST values (per metric) were defined on subsets specific to a stream type, sometimes small (between 6 and 135 SEEs, depending on the stream type) from a total dataset of “only” 2336 SSEs performed in Least Impaired River Reaches (Usseglio-Polatera et al., 2016). As a result, the uncertainty related to the BEST values was higher than that related to the WORST values. In a more practical point of view, these results have suggested that reference values (i.e., BEST values in our study) should be defined from subsets of data including at least 50 SSEs from at least 20 different sites. If such criteria are not met for a given stream type, type-specific reference values can be modelled based on the characteristics of this stream type (e.g., on characteristics less or non-affected by human activities, such as location, altitude, slope, geochemical background; Clarke et al., 2003).

The uncertainty related to the variations of the WGHT values of individual metrics was very low, close to zero. Similarly to the BEST values, each WGHT was defined on a stressor-specific subset of data. However, for increasing the robustness of these WGHT values, each of them was calculated as an average WGHT over 100 random subsets of the corresponding stressor-specific dataset (Mondy et al., 2012). Although the robustness of the WGHT values generated in this study for the assessment of the I₂M₂ uncertainty was not optimized by a secondary bootstrapping step on the generated subsets of data, the uncertainty related to WGHT remained very low. This implies that this overall low uncertainty may not be explained only by the robustness of the WGHT definition, but also by the use of WGHT values as weights for the calculation of the final I₂M₂ value (which is the arithmetic mean of the 17 i₂m₂ sub-index scores, cf. Eq.1). Although each individual WGHT value could have exhibited some amount of variability in our models, the high number of WGHT involved in the index calculation (17 stressors × 5 metrics = 85 different WGHT values) has probably compensated part of the effect of individual WGHT variabilities on the final I₂M₂ value variability. The low uncertainty associated with WGHT can therefore be explained by the number of metrics included in the multimetric index and by the number and variety of stressors taken into account by the index: the smaller the number of metrics and stressors, the more the uncertainty associated with the WGHT can contribute to the uncertainty of the index.
### 4.3. Implications for stakeholders

Whatever the studied index, BQE and/or source of uncertainty, the highest uncertainty and mismatch frequency are actually observed when the index values are close to the ecological class boundaries, with a 50 % probability for both neighbouring classes when the index value is exactly equal to a given boundary value (Moe et al., 2015; Wiederkehr et al., 2015, and this study).

Moreover, uncertainties appeared to be higher for intermediate ecological classes ("good", "moderate" and "poor") than for extreme classes ("high" and "bad") for diatoms and macrophytes (Wach et al., 2019; Wiederkehr et al., 2015), fish (Logez et al., 2019; Marzin et al., 2014) or invertebrates (this study). Such higher uncertainty in the ecological assessment of intermediate quality classes could be related to mathematical reasons: the uncertainty in the quality class allocation increases when the index values are closer to the limit of an ecological class, reaching up to 50 % when equal to the boundary value. Thus, the narrower ecological classes, such as the intermediate classes in this study, exhibit greater uncertainty than the broader classes, i.e., extreme ones here. The higher uncertainty in the ecological assessment for sampling events assigned to intermediate quality classes could also be explained by ecological reasons related to the strength of the stressor/ effect relationship, which is potentially stronger for extreme stressor levels than for intermediate levels where local natural constraints or environmental characteristics (e.g., habitat diversity or stability, avail- ability of refugia) can partly blur assemblage responses to the investi- gated stressors (Townsend, 1989; Townsend and Hildrew, 1994), or lead to the coexistence of both sensitive and tolerant taxa (Kelly et al., 2009; Wach et al., 2019).

The relationship between the mismatch frequency and the quality class properties (identity and width) is of interest for stakeholders, especially when focusing on the boundary "good-moderate", because the WFD states that European water bodies should at least have good ecological status. Therefore, sites with index values close to this limit represent (i) sites that are the most plausible candidates for restoration actions that could allow them to reach – "rather easily" – good ecological status (if their I₂M₂ value is slightly below the good-moderate bound- ary), or (ii) sites whose ecological status is fragile and which require special attention to achieve a more robust ecological status (if their I₂M₂ value is slightly above this boundary). The assessment of these sites needs to be improved, for example by analysing data combined over several years, as suggested by Clarke et al. (2002) and Clarke (2009), and as effectively applied for reporting on the ecological status of water bodies of European member states in the WFD framework. This approach will limit the effect of between-years variability and could be combined with other strategies (cf. next paragraph) to further enhance the robustness of the ecological status assessment.

The results of this study confirmed that a robust biotic index design is an important way to mitigate the uncertainty associated with the final stream ecological assessment (Marzin et al., 2014; Reyjol et al., 2014; Stoddard et al., 2008; Vadas et al., 2022). Given that inter-operator variability represents the greatest source of uncertainty, combining the construction of robust indices and the assessment of their uncer- tainty with quality management actions (e.g., field and laboratory step standardisation and operator training) should make it possible to improve ecological assessment (Reyjol et al., 2014).

### CRediT authorship contribution statement

**Albin Meyer:** Writing – review & editing, Writing – original draft, Visualization, Validation, Methodology, Investigation, Formal analysis, Data curation, Conceptualization.
**Elise Billoir:** Writing – review & editing, Writing – original draft, Visualization, Validation, Supervision, Project administration, Methodology, Investigation, Funding acquisition, Formal analysis, Data curation, Conceptualization.
**Virginie Archaimbault:** Writing – review & editing, Writing – original draft,

Visualization, Validation, Supervision, Resources, Project administra- tion, Methodology, Investigation, Funding acquisition, Formal analysis, Data curation, Conceptualization.
**Cédric P. Mondy:** Writing – review & editing, Writing – original draft, Visualization, Validation, Supervision, Methodology, Investigation, Formal analysis, Data curation, Conceptu- alization.
**Philippe Usseglio-Polatera:** Writing – review & editing, Writing – original draft, Visualization, Validation, Supervision, Re- sources, Project administration, Methodology, Investigation, Funding acquisition, Formal analysis, Data curation, Conceptualization.

### Declaration of competing interest

The authors declare the following financial interests/personal re- lationships which may be considered as potential competing interests: Virginie Archaimbault reports financial support was provided by Onema. If there are other authors, they declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

### Acknowledgments

This study was supported by grants from the French Biodiversity Agency (OFB; ex-ONEMA; Framework agreement ONEMA-CEMAGREF 2009-2012; ONEMA-2016-02869). We also wish to thank the numerous staff from the OFB (ex-DREALs Aquitaine, Auvergne, Basse Normandie, Centre, Haute Normandie, Languedoc-Roussillon, Lorraine, Picardie and Rhône-Alpes) and the following engineering offices (AQUABIO, AQUASCOP, ASCONIT, GREBE and HYDROBIO) for their participation in the field study, and Marie-Claude Roger and Aurélie Josset (INRAE) for their help with the sorting and the identification of macroinvertebrates.

## Appendix A. Supplementary data

Supplementary data to this article can be found online at https://doi.org/10.1016/j.scitotenv.2025.179218.

### Data availability

The scripts and data are available online (https://github.com/Albin-Meyer/Uncertainty-I2M2).

### References

* FNOR, 2009. Qualité de l'eau – Prélèvement des macro-invertébrés aquatiques en rivières peu profondes. T90-333.

* AFNOR, 2010. Qualité de l'eau – Traitement au laboratoire d'échantillons contenant des macro-invertébrés de cours d'eau. T90-388.

* Álvarez-Cabria, M., Barquín, J., Juanes, J.A., 2011. Microdistribution patterns of macroinvertebrate communities upstream and downstream of organic effluents. Water Res. 45 (3), 1501–1511. https://doi.org/10.1016/j.watres.2010.11.028.

* Archaimbault, V., Josset, A., & Roger, M.-C., 2012. Évaluation de l'incertitude liée à l'application du protocole d'échantillonnage des macro-invertébrés benthiques en cours d'eau peu profonds selon la norme XP T90-333. Bilan d'avancement de l'année 2011. Action I-A-04 : Consolidation et mise au point de méthodes de bio-indication et transfert aux opérateurs. CEMAGREF report.

* Armitage, P.D., Moss, D., Wright, J., Furse, M., 1983. The performance of a new biological water quality score system based on macroinvertebrates over a wide range of unpolluted running-water sites. Water Res. 17 (3), 333–347. https://doi.org/10.1016/0043-1354(83)90188-4.

* Armitage, P.D., Pardo, I., Brown, A., 1995. Temporal constancy of faunal assemblages in 'mesohabitats' – application to management? Arch. Hydrobiol. 367–387. https://doi.org/10.1127/archiv-hydrobiol/133/1995/367.

* Bark, R.H., Peeters, L.J.M., Lester, R.E., Pollino, C.A., Crossman, N.D., Kandulu, J.M., 2013. Understanding the sources of uncertainty to reduce the risks of undesirable outcomes in large-scale freshwater ecosystem restoration projects: an example from the Murray-Darling basin, Australia. Environ. Sci. Pol. 33, 97–108. https://doi.org/10.1016/j.envsci.2013.04.010.

* Barnes, J.B., Vaughan, I.P., Ormerod, S.J., 2013. Reappraisuing the effects of habitat structure on river macroinvertebrates. Freshw. Biol. 58 (10), 2154–2167. https://doi.org/10.1111/fwb.12198.
Beisel, J.-N., Usseglio-Polatera, P., Moreteau, J.-C., 2000. The spatial heterogeneity of a river bottom: A key factor determining macroinvertebrate communities. In: Jungwirth, M., Muhar, S., Schmutz, S. (Eds.), Assessing the Ecological Integrity of Running Waters. Springer, Netherlands, pp. 163–171. https://doi.org/10.1007/978-94-011-4164-2_13.

Beisel, J.-N., Usseglio-Polatera, P., Thomas, S., Moreteau, J.-C., 1998. Stream community structure in relation to spatial variation: the influence of mesohabitat characteristics. Hydrobiologia 389 (1–3), 73–88. https://doi.org/10.1023/A:100351942997.

Billoir, E., Coulaud, R., Larras, F., Usseglio-Polatera, P., 2015. Finalisation I₂M₂, chantier Diagnostic multi-EQB et développement d'un indice pour les grands cours d'eau. Scientific Report., University of Lorraine, Metz, France.

Birk, S., Bonne, W., Borja, A., Brucet, S., Courrat, A., Poikane, S., Solimini, A., van de Bund, W., Zampoukas, N., Hering, D., 2012. Three hundred ways to assess Europe's surface waters: an almost complete overview of biological methods to implement the water framework directive. Ecol. Indic. 18, 3–41. https://doi.org/10.1016/j.ecolind.2011.10.009.

Blocksom, K.A., 2003. A performance comparison of metric scoring methods for a multimetric index for mid-Atlantic highlands streams. Environ. Manag. 31 (5), 670–682. https://doi.org/10.1007/s00267-002-2949-3.

Boyer, L., 2003. Multiscale patterns of spatial variation in stream macroinvertebrate communities. Ecol. Res. 18 (4), 365–379. https://doi.org/10.1046/j.1440-1703.2003.00562.x.

Buffagni, A., Erba, S., Cazzola, M., Barca, E., Belfiore, C., 2020. The ratio of lentic to lotic habitat features strongly affects macroinvertebrate metrics used in southern Europe for ecological status classification. Ecol. Indic. 117, 106563. https://doi.org/10.1016/j.ecolind.2020.106563.

Clarke, R.T., 2009. Uncertainty in WFD Assessments for Rivers Based on Macroinvertebrates and RIVPACS. Environment Agency.

Clarke, R.T., 2013. Estimating confidence of European WFD ecological status class and WISER bioassessment uncertainty guidance software (WISERBUGS). Hydrobiologia 704 (1), 39–56. https://doi.org/10.1007/s10750-012-1245-3.

Clarke, R.T., Davy-Bowker, J., Sandin, L., Friberg, N., Johnson, R.K., Bis, B., 2006. Estimates and comparisons of the effects of sampling variation using 'national' macroinvertebrate sampling protocols on the precision of metrics used to assess ecological status. Hydrobiologia 566 (1), 477–503. https://doi.org/10.1007/s10750-006-0076-5.

Clarke, R.T., Furse, M.T., Gunn, R.J.M., Winder, J.M., Wright, J.F., 2002. Sampling variation in macroinvertebrate data and implications for river quality indices. Freshw. Biol. 47 (9), 1735–1751. https://doi.org/10.1046/j.1365-2427.2002.00885.x.

Clarke, R.T., Hering, D., 2006. Errors and uncertainty in bioassessment methods – major results and conclusions from the STAR project and their application using STARBUGS. Hydrobiologia 566 (1), 433–439. https://doi.org/10.1007/s10750-006-0079-2.

Clarke, R.T., Wright, J.F., Furse, M.T., 2003. RIVPACS models for predicting the expected macroinvertebrate fauna and assessing the ecological quality of rivers. Ecol. Model. 160 (3), 219–233. https://doi.org/10.1016/j.s0304-3800(02)00255-7.

Dahl, J., Greenberg, L., 1996. Effects of habitat structure on habitat use by *Gammarus pulex* in artificial streams. Freshw. Biol. 36 (3), 487–495. https://doi.org/10.1046/j.1365-2427.1996.00096.x.

Dedecker, A.P., Goethals, P.L.M., D'heygere, T., Gevrey, M., Lek, S., & De Pauw, N., 2005. Application of artificial neural network models to analyse the relationships between *Gammarus pulex* L. (Crustacea, Amphipoda) and river characteristics. Environ. Monit. Assess., 111(1), 223–241. doi:https://doi.org/10.1007/s10661-005-8221-6.

Delignette-Muller, M.-L., Dutang, C., Pouillot, R., Denis, J.-B., Siberchicot, A., 2020. Help to fit of a parametric distribution to non-censored or censored data. Package R « fitdistrplus». https://bbbe.univ-lyon1.fr/fitdistrplus.html.

Descloux, S., Datry, T., Usseglio-Polatera, P., 2014. Trait-based structure of invertebrates along a gradient of sediment colmation: benthos versus hypheos responses. Sci. Total Environ. 466–467, 265–276. https://doi.org/10.1016/j.scitotenv.2013.06.082.

Dromph, K.M., Agusti, S., Basset, A., Franco, J., Henriksen, P., Icely, J., Lehtinen, S., Moncheva, S., Revilla, M., Roselli, L., Sørensen, K., 2013. Sources of uncertainty in assessment of marine phytoplankton communities. Hydrobiologia 704 (1), 253–264. https://doi.org/10.1007/s10750-012-1353-0.

Dudgeon, D., 2019. Multiple threats imperil freshwater biodiversity in the Anthropocene. Curr. Biol. 29 (19), R960–R967. https://doi.org/10.1016/j.cub.2019.08.002.

Effenberger, M., Sailer, G., Townsend, C.R., Matthaei, C.D., 2006. Local disturbance history and habitat parameters influence the microdistribution of stream invertebrates. Freshw. Biol. 51 (2), 312–332. https://doi.org/10.1111/j.1365-2427.2005.01502.x.

European Council, 2000. Directive 2000/60/EC. Establishing a framework for community action in the field of water policy. European Commission PE-CONS 3639/1/100 rev 1, Luxemburg.

Gobeyn, S., Bennetsen, E., Van Echelpoel, W., Everaert, G., Goethals, P.L.M., 2016. Impact of abundance data errors on the uncertainty of an ecological water quality assessment index. Ecol. Indic. 60, 746–753. https://doi.org/10.1016/j.ecolind.2015.07.031.

Grizzetti, B., Liqueste, C., Pistocchi, A., Vigiak, O., Zulian, G., Bouraoui, F., De Roo, A., Cardoso, A.C., 2019. Relationship between ecological condition and ecosystem services in European rivers, lakes and coastal waters. Sci. Total Environ. 671, 452–465. https://doi.org/10.1016/j.scitotenv.2019.03.155.

Grothendieck, G., 2013. Non-linear regression with brute force. Package R nls2. http://nl2.googlecode.com.

Haase, P., Murray-Blight, J., Lohse, S., Pauls, S., Sundermann, A., Gunn, R., Clarke, R.T., 2006. Assessing the impact of errors in sorting and identifying macroinvertebrate

samples. Hydrobiologia 566 (1), 505–521. https://doi.org/10.1007/s10750-006-0075-6.

Hawkins, C.P., Cao, Y., Roper, B., 2010. Method of predicting reference condition biota affects the performance and interpretation of ecological indices. Freshw. Biol. 55 (5), 1066–1085. https://doi.org/10.1111/j.1365-2427.2009.02357.x.

Johnson, R.K., Hering, D., Furse, M.T., Clarke, R.T., 2006. Detection of ecological change using multiple organism groups: metrics and uncertainty. Hydrobiologia 566 (1), 115–137. https://doi.org/10.1007/s10750-006-0101-8.

Kelly, M., Bennion, H., Burgess, A., Ellis, J., Juggins, S., Guthrie, R., Jamieson, J., Adriaensens, V., Yallop, M., 2009. Uncertainty in ecological status assessments of lakes and rivers using diatoms. Hydrobiologia 633 (1), 5–15. https://doi.org/10.1007/s10750-009-9872-z.

Krause-Jensen, D., Greve, T.M., Nielsen, K., 2005. Eelgrass as a bioindicator under the European Water Framework Directive. Water Resour. Manag. 19 (1), 63–75. https://doi.org/10.1007/s11269-005-0293-0.

Lencioni, V., Rossaro, B., 2005. Microdistribution of chironomids (Diptera: Chironomidae) in alpine streams: an autoecological perspective. Hydrobiologia 533 (1), 61–76. https://doi.org/10.1007/s10750-004-2393-x.

LIEC (Laboratoire Interdisciplinaire des Environnements Continentaux), & AFB (Agence Française pour la Biodiversité). 2018. Proposal of a new method: the multimetric index based on macroinvertebrate fauna for the ecological assessment of French wadeable rivers (I₂M₂).

Loga, M., Wierzcholowska-Dziedzic, A., 2017. Probability of misclassifying biological elements in surface waters. Environ. Monit. Assess. 189 (12), 647. https://doi.org/10.1007/s10661-017-6368-6.

Logez, M., Maire, A., Argillier, C., 2019. Monte-Carlo methods to assess the uncertainty related to the use of predictive multimetric indices. Ecol. Indic. 96, 52–58. https://doi.org/10.1016/j.ecolind.2018.08.051.

Lorenz, A., Clarke, R.T., 2006. Sample coherence – A field study approach to assess similarity of macroinvertebrate samples. In: Furse, M.T., Hering, D., Brabec, K., Buffagni, A., Sandin, L., Verdonschot, P.F.M. (Eds.), The Ecological Status of European Rivers: Evaluation and Intercalation of Assessment Methods. Springer, Netherlands, pp. 461–476. https://doi.org/10.1007/978-1-4020-5493-8_32.

Marzin, A., Delaigue, O., Logez, M., Belliard, J., Pont, D., 2014. Uncertainty associated with river health assessment in a varying environment: the case of a predictive fish-based index in France. Ecol. Indic. 43, 195–204. https://doi.org/10.1016/j.ecolind.2014.02.011.

Mascaró, O., Alcoverro, T., Dencheva, K., Díez, I., Gorostiaga, J.M., Krause-Jensen, D., Balsby, T.J.S., Marbà, N., Muxika, I., Neto, J.M., Nikolić, V., Orfanidis, S., Pedersen, A., Pérez, M., Romero, J., 2013. Exploring the robustness of macrophyte-based classification methods to assess the ecological status of coastal and transitional ecosystems under the Water Framework Directive. Hydrobiologia 704 (1), 279–291. https://doi.org/10.1007/s10750-012-1426-0.

Metzeling, L., Chessman, B., Hardwick, R., Wong, V., 2003. Rapid assessment of rivers using macroinvertebrates: the role of experience, and comparisons with quantitative methods. Hydrobiologia 510 (1), 39–52. https://doi.org/10.1023/B:HYDR.000008500.34301.a0.

Moe, S.J., Lyche Solheim, A., Soszka, H., Gołub, M., Hutorowicz, A., Kolada, A., Picińska-Faltynowicz, J., Białokoz, W., 2015. Integrated assessment of ecological status and misclassification of lakes: the role of uncertainty and index combination rules. Ecol. Indic. 48, 605–615. https://doi.org/10.1016/j.ecolind.2014.08.018.

Mondy, C.P., Villeneuve, B., Archaimbault, V., Usseglio-Polatera, P., 2012. A new macroinvertebrate-based multimetric index (I₂M₂) to evaluate ecological quality of French wadeable streams fulfilling the WFD demands: a taxonomic and trait approach. Ecol. Indic. 18, 452–467. https://doi.org/10.1016/j.ecolind.2011.12.013.

Ofenböck, T., Moog, O., Gerritsen, J., Barbour, M., 2004. A stressor specific multimetric approach for monitoring running waters in Austria using benthic macro-invertebrates. In: Hering, D., Verdonschot, P.F.M., Moog, O., Sandin, L. (Eds.), Integrated Assessment of Running Waters in Europe. Springer, Netherlands, pp. 251–268. https://doi.org/10.1007/978-94-007-0993-5_15.

Ounanian, K., Carballo-Cárdenas, E., van Tatenhove, J.P.M., Delaney, A., Papadopoulou, K.N., Smith, C.J., 2018. Governing marine ecosystem restoration: the role of discourses and uncertainties. Mar. Policy 96, 136–144. https://doi.org/10.1016/j.marpol.2018.08.014.

Pardo, I., Armitage, P.D., 1997. Species assemblages as descriptors of mesohabitats. Hydrobiologia 344, 111–128. https://doi.org/10.1023/A:1002958412237.

Pardo, I., Gómez-Rodríguez, C., Wasson, J.-G., Owen, R., van de Bund, W., Kelly, M., Bennett, C., Birk, S., Buffagni, A., Erba, S., Mengin, N., Murray-Blight, J., Ofenböck, G., 2012. The European reference condition concept: a scientific and technical approach to identify minimally-impacted river ecosystems. Sci. Total Environ. 420, 33–42. https://doi.org/10.1016/j.scitotenv.2012.01.026.

R Core Team, 2020. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria. https://www.R-project.org/.

Rabeni, C.F., Minshall, G.W., 1977. Factors affecting microdistribution of stream benthic insects. Oikos 29 (1), 33–43. https://doi.org/10.2307/3543290.

Ramos-Merchante, A., Prenda, J., 2017. Macroinvertebrate taxa richness uncertainty and kick sampling in the establishment of Mediterranean rivers ecological status. Ecol. Indic. 72, 1–12. https://doi.org/10.1016/j.ecolind.2016.07.047.

Reyjol, Y., Argillier, C., Bonne, W., Borja, A., Buijse, A.D., Cardoso, A.C., Daufresne, M., Kernan, M., Ferreira, M.T., Poikane, S., Prat, N., Solheim, A.-L., Stoffek, S., Usseglio-Polatera, P., Villeneuve, B., van de Bund, W., 2014. Assessing the ecological status in the context of the European Water Framework Directive: where do we go now? Sci. Total Environ. 497–498, 332–344. https://doi.org/10.1016/j.scitotenv.2014.07.119.

Santos, J.I., Vidal, T., Gonçalves, F.J.M., Castro, B.B., Pereira, J.L., 2021. Challenges to water quality assessment in Europe – is there scope for improvement of the current
water framework directive bioassessment scheme in rivers? Ecol. Indic. 121, 107030. https://doi.org/10.1016/j.ecolind.2020.107030.

Shannon, C.E., 1948. A mathematical theory of communication. The Bell Syst. Tech. J., 27, 379–423 and 623–656.

Šporka, F., Vlek, H.E., Bulánková, E., Krno, I., 2006. Influence of seasonal variation on bioassessment of streams using macroinvertebrates. In: Furse, M.T., Hering, D., Brabec, K., Buffagni, A., Sandin, L., Verdonschot, P.F.M. (Eds.), The Ecological Status of European Rivers: Evaluation and Intercalibration of Assessment Methods. Springer, Netherlands, pp. 543–555. https://doi.org/10.1007/978-1-4020-5493-8_36.

Staniszewski, R., Szoszkiewicz, K., Zbierska, J., Lesny, J., Jusik, S., Clarke, R.T., 2006. Assessment of sources of uncertainty in macrophyte surveys and the consequences for river classification. Hydrobiologia 566 (1), 235–246. https://doi.org/10.1007/s10750-006-0093-4.

Statzner, B., Gore, J.A., Resh, V.H., 1988. Hydraulic stream ecology: observed patterns and potential applications. J. N. Am. Benthol. Soc. 7 (4), 307–360. https://doi.org/10.2307/1467296.

Stoddard, J.L., Herlihy, A.T., Peck, D.V., Hughes, R.M., Whittier, T.R., Tarquinio, E., 2008. A process for creating multimetric indices for large-scale aquatic surveys. J. N. Am. Benthol. Soc. 27 (4), 878–891. https://doi.org/10.1899/08-053.1.

Stoddard, J.L., Larsen, D.P., Hawkins, C.P., Johnson, R.K., Norris, R.H., 2006. Setting expectations for the ecological condition of streams: the concept of reference condition. Ecol. Appl. 16 (4), 1267–1276. https://doi.org/10.1890/1051-0761 (2006)016[1267:SEFTEC]2.0.CO;2.

Teichert, N., Lepage, M., Sagouis, A., Borja, A., Chust, G., Ferreira, M.T., Pasquaud, S., Schinegger, R., Segurado, P., Argillier, C., 2017. Functional redundancy and sensitivity of fish assemblages in European rivers, lakes and estuarine ecosystems. Sci. Rep. 7 (1), 17611. https://doi.org/10.1038/s41598-017-17975-x.

Thackeray, S.J., Nöges, P., Dunbar, M.J., Dudley, B.J., Skjelbred, B., Morabito, G., Carvalho, L., Phillips, G., Mischke, U., Catalan, J., de Hoyos, C., Laplace-Treyture, C., Austoni, M., Padedda, B.M., Maileht, K., Pasztaleniec, A., Järvinen, M., Solheim, A. L., Clarke, R.T., 2013. Quantifying uncertainties in biologically-based water quality assessment: a pan-European analysis of lake phytoplankton community metrics. Ecol. Indic. 29, 34–47. https://doi.org/10.1016/j.ecolind.2012.12.010.

Townsend, C.R., 1989. The patch dynamics concept of stream community ecology. J. N. Am. Benthol. Soc. 8, 36–50. https://doi.org/10.2307/1467400.

Townsend, C.R., Hildrew, A.G., 1994. Species traits in relation to a habitat templet for river systems. Freshw. Biol. 31, 265–275. https://doi.org/10.1111/j.1365-2427.1994.tb01740.x.

Usseglio-Polatera, P., Larras, F., Coulaud, R., Billoir, E., 2016. Finalisation $I_2M_2$, chantier diagnostic multi-EQB et développement d'un indice pour les grands cours d'eau. Lorraine University report.

Vadas, R.L., Hughes, R.M., Bae, Y.J., Baek, M.J., Gonzáles, O.C.B., Callisto, M., Reis de Carvalho, D., Chen, K., Ferreira, M.T., Fierro, P., Harding, J.S., Infante, D.M., Kleynhans, C.J., Macedo, D.R., Martins, I., Silva, N.M., Moya, N., Nichols, S.J., Pompeu, P.S., Yoder, C.O., 2022. Assemblage-based biomonitoring of freshwater ecosystem health via multimetric indices: a critical review and suggestions for improving their applicability. Water Biology and Security 1 (3), 100054. https://doi.org/10.1016/j.watbs.2022.100054.

van de Bund, W., 2009. Water Framework Directive Intercalibration: Technical report. Part 1, Rivers. Publications Office. https://doi.org/10.2788/23384.

Vaughn, C.C., 2010. Biodiversity losses and ecosystem function in freshwaters: emerging conclusions and research directions. BioScience 60 (1), 25–35. https://doi.org/10.1525/bio.2010.60.1.7.

Vlek, H.E., Šporka, F., Krno, I., 2006. Influence of macroinvertebrate sample size on bioassessment of streams. In: Furse, M.T., Hering, D., Brabec, K., Buffagni, A., Sandin, L., Verdonschot, P.F.M. (Eds.), The Ecological Status of European Rivers: Evaluation and Intercalibration of Assessment Methods. Springer, Netherlands, pp. 523–542. https://doi.org/10.1007/978-1-4020-5493-8_35.

Wach, M., Guéguen, J., Chauvin, C., Delmas, F., Dagens, N., Feret, T., Loriot, S., Tison-Rosebery, J., 2019. Probability of misclassifying river ecological status: a large-scale approach to assign uncertainty in macrophyte and diatom-based biomonitoring. Ecol. Indic. 101, 285–295. https://doi.org/10.1016/j.ecolind.2019.01.028.

Wasson, J.-G., Chandesris, A., Pella, H., & Blanc, L., 2002. Les hydro-écorégions de France métropolitaine – Approche régionale de la typologie des eaux courantes et éléments pour la définition des peuplements de référence d'invertébrés. CEMAGREF report.

Wiederkehr, J., Grac, C., Fabrègue, M., Fontan, B., Labat, F., Le Ber, F., Trémolières, M., 2015. Experimental study of uncertainties on the macrophyte index (IBMR) based on species identification and cover. Ecol. Indic. 50, 242–250. https://doi.org/10.1016/j.ecolind.2014.10.021.

Wiederkehr, J., Grac, C., Fontan, B., Labat, F., Le Ber, F., Trémolières, M., 2016. Experimental study of the uncertainty of the intrasubstrate variability on two French index metrics based on macroinvertebrates. Hydrobiologia 779 (1), 59–73. https://doi.org/10.1007/s10750-016-2799-2.

Zucchetta, M., Scapin, L., Franco, A., Franzoi, P., 2020. Uncertainty in developing fish based multi-metric indices. Ecol. Indic. 108, 105768. https://doi.org/10.1016/j.ecolind.2019.105768.