FRM4SM WORK PACKAGE 5 TECHNICAL NOTE 1
SO-TN-CB-GS-0104 REQ-16: R&D CASE STUDY 1
SMOS VALIDATION AND COMMITTED AREAS

François Gibon, Arnaud Mialon, Philippe Richaume, Yann H. Kerr, Ali Mahmoodi, Nemesio Rodríguez-Fernández, Roberto Sabia, Alexander Boresch

► To cite this version:

François Gibon, Arnaud Mialon, Philippe Richaume, Yann H. Kerr, Ali Mahmoodi, et al. FRM4SM WORK PACKAGE 5 TECHNICAL NOTE 1 SO-TN-CB-GS-0104 REQ-16: R&D CASE STUDY 1 SMOS VALIDATION AND COMMITTED AREAS. SO-TN-CB-GS-0104, CESBIO. 2024. hal-04840013

HAL Id: hal-04840013

https://hal.science/hal-04840013v1

Submitted on 16 Dec 2024

**HAL** is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Abstract

The aim of this document is to investigate the validation of the satellite SMOS soil moisture product considering the ISMN *in-situ* network in order to define committed areas of accuracy. A validation method is presented and detailed here. The first benchmark result shows a global agreement between the two datasets. Then a sensitivity study of the scores regarding the probes configuration shows better results considering only the one installed at the surface level. Concerning the sensitivity of the scores to the SMOS footprint content, the scores show a performance improvement with a minimization of forest, topography, water, and ice in the footprint. Concerning the soil parameters, the scores improve when the SMOS footprint content is sandier, has less clay, and has a high bulk-density. From those results, maps of expected accuracy and recommended areas for validation have been derived. The SMOS accuracy of 0.04 m³.m⁻³ specified in the Mission Requirement Document (MRD) is also investigated with the identification of surface conditions constraints to reach it.

**Keywords:** SMOS validation, committed areas accuracy, metrics, in-situ representativeness
## Document approval

<table><thead><tr><th></th><th>Role</th><th>Name</th><th>Date</th><th>Signature</th></tr></thead><tbody><tr><td rowspan="5">Written by</td><td rowspan="5">Project engineer</td><td>F. Gibon</td><td>18/03/22</td><td rowspan="5"></td></tr><tr><td>A. Mialon</td><td>18/03/22</td></tr><tr><td>Y. Kerr</td><td>18/03/22</td></tr><tr><td>N. Rodriguez F.</td><td>18/03/22</td></tr><tr><td>P. Richaume</td><td>18/03/22</td></tr><tr><td>Approved by</td><td>Lead investigator</td><td>A. Mialon</td><td>18/03/22</td><td></td></tr><tr><td>Approved by</td><td>Technical officer</td><td>R. Sabia</td><td>/ /22</td><td></td></tr></tbody></table>

## Distribution list

<table><thead><tr><th>Name</th><th>Affil.</th><th>Signature</th></tr></thead><tbody><tr><td></td><td></td><td></td></tr><tr><td></td><td></td><td></td></tr><tr><td></td><td></td><td></td></tr><tr><td></td><td></td><td></td></tr><tr><td></td><td></td><td></td></tr><tr><td></td><td></td><td></td></tr><tr><td></td><td></td><td></td></tr><tr><td></td><td></td><td></td></tr><tr><td></td><td></td><td></td></tr></tbody></table>

<table><thead><tr><th>Name</th><th>Affil.</th><th>Signature</th></tr></thead><tbody><tr><td></td><td></td><td></td></tr><tr><td></td><td></td><td></td></tr><tr><td></td><td></td><td></td></tr><tr><td></td><td></td><td></td></tr><tr><td></td><td></td><td></td></tr><tr><td></td><td></td><td></td></tr><tr><td></td><td></td><td></td></tr><tr><td></td><td></td><td></td></tr><tr><td></td><td></td><td></td></tr><tr><td></td><td></td><td></td></tr></tbody></table>

## Auxiliary documents related to this project

<table><tr><td>ATBD L2</td><td>Kerr, Y.H. and Richaume, P., and Waldteufel, P. and Ferrazzoli, P. and Wigneron, J.P. and Schwank, M. and Rautiainen, K., Algorithm theoretical basis document (ATBD) for the SMOS level 2 soil moisture processor, Technical Report TN-ESL-SM-GS-0001-4b SM-ESL (CBSA), 145p, 2020</td></tr><tr><td>FRM4SM detailed proposal</td><td>FRM4SM - Fiducial Reference Measurements for Soil Moisture Detailed Proposal, Vienna, Austria, AWST TUWIEN CESBIO, January 2021</td></tr><tr><td>MRD</td><td>EEOM-SMOS-MRD, Mission Objectives and Scientific Requirements of the Soil Moisture and Ocean Salinity (SMOS) Mission, Version 5, issue 1.0 040701, https://esamultimedia.esa.int/docs/SMOS_MRD_V5.pdf</td></tr></table>
# Purpose and scope of document

Fiducial Reference Measurement for Soil Moisture (FRM4SM) is a two-year ESA project (May 2021-April 2023) with the main goal to provide confidence information in satellite-based soil moisture retrievals all along the mission duration towards a maximum Return On Investment (ROI).

## ESA FRM4SM PROJECT CONTRACT N°4000135204/21//I-BG

The present document corresponds to the REQ-16 of the FRM4SM detailed proposal, related to the Case Study 1. It answers, in part, to the different objectives of the Task 5 led by the Working Group 5 which are:

* OBJ-11 – Three SMOS R&D case studies

* OBJ-12 – Sub-footprint variability and vertical stratification influence

* OBJ-13 – Temporal aliasing

* OBJ-14 – In-situ-driven strategy versus satellite-driven strategy

* OBJ-15 – Committed areas

# Document status

<table><thead><tr><td>Issue</td><td>Date</td><td>Details</td><td>Editor</td></tr></thead><tbody><tr><td>0.0</td><td>Early November 2021</td><td>Creation of draft and structure within document</td><td>François Gibon<br>Arnaud Mialon</td></tr><tr><td>0.1</td><td>January 21, 2022</td><td>Add maps of uncertainties</td><td>François Gibon<br>Arnaud Mialon</td></tr><tr><td>0.2</td><td>Feb-March, 2022</td><td>Final draft</td><td>François Gibon<br>Arnaud Mialon<br>Yann Kerr<br>Philippe Richaume</td></tr><tr><td>0.3</td><td>March 23, 2022</td><td>Submitted to ESA</td><td>François Gibon<br>Arnaud Mialon</td></tr></tbody></table>

If further corrections are required please contact François Gibon or Arnaud Mialon (francois.gibon@univ-tlse3.fr, arnaud.mialon@univ-tlse3.fr).
# Contents

## 1 Introduction 6

1.1 Microwave satellite validation issues 6

1.2 Objectives 6

## 2 Data 7

2.1 In-situ data: the International Soil Moisture Network 7

2.1.1 Presentation 7

2.1.2 Sites description 7

2.1.3 The ISMN metadata 9

2.2 Satellite data: Soil Moisture and Ocean Salinity 9

2.2.1 Presentation 9

2.2.2 SMOS grid, Weighted Function, Working Area and Fraction 10

2.2.3 The Radio Frequency Interference issues 11

2.3 Geophysical auxiliary database 13

## 3 Validation procedure description 14

3.1 Validation chain description 14

3.2 Uncertainty sources investigation 16

3.3 Metrics 18

## 4 Analysis of the scores and the uncertainties 19

4.1 Overview of validation SMOS vs. ISMN 19

4.2 Score sensitivity to the in-situ probes configuration 21

4.2.1 To the probe depth 21

4.2.2 To the probes technologies 24

4.2.3 To the soil characteristics 25

4.2.4 To the ISMN land cover metadata (ESA CCI) 26

4.2.5 ISMN Quality Flag analysis 27

4.2.6 To the ISMN climate class metadata (Köppen climates classification) 27

4.2.7 To the ISMN network ranking 29

4.3 Score sensitivity to the SMOS footprint content 30

4.3.1 RFI filtering impact on the scores 30

4.3.2 SMOS quality index (DQX, CHI2P) 31

4.3.3 Investigation of the SMOS geophysical footprint surface conditions 32

4.4 Score sensitivity to the number of samples 37

## 5 Committed and recommended areas for SMOS validation 38

5.1 The committed areas in the SMOS MRD 38

5.2 Toward a new MRD surface conditions 41

5.3 Maps of committed areas accuracy regarding the sensitivity study SMOS vs. ISMN 43

5.4 Maps of recommended areas for SMOS validation 47

## 6 Conclusion, limitations, and perspective of improvements 48

## References 50

## Appendices: Gibon et al. 2022 in prep. 52
## Acronyms

<table><tr><td>- AGB</td><td>Above Ground Biomass</td></tr><tr><td>- BT</td><td>Brightness Temperature</td></tr><tr><td>- CATDS</td><td>Centre Aval de Traitement des Données SMOS</td></tr><tr><td>- ECMWF</td><td>European Center for Medium-Range Weather Forecasts</td></tr><tr><td>- ESA</td><td>European Space Agency</td></tr><tr><td>- ESL</td><td>Expert Support Laboratory</td></tr><tr><td>- FRM4SM</td><td>Fiducial References Measurements for Soil Moisture</td></tr><tr><td>- HWSD</td><td>Harmonized World Soil Database</td></tr><tr><td>- ISMN</td><td>International Soil Moisture Network</td></tr><tr><td>- MEANWEF</td><td>MEANWEF weighted average function of all the WEFs</td></tr><tr><td>- MRD</td><td>Mission Requirements Document</td></tr><tr><td>- QA4SM</td><td>Quality Assurance for Soil Moisture</td></tr><tr><td>- RFI</td><td>Radio Frequency Interference</td></tr><tr><td>- RTM</td><td>Radiative Transfer Model</td></tr><tr><td>- SM</td><td>Soil Moisture</td></tr><tr><td>- SMOS</td><td>Soil Moisture and Ocean Salinity</td></tr><tr><td>- VOD</td><td>Vegetation Optical Depth</td></tr><tr><td>- VWC</td><td>Vegetation Water Content</td></tr><tr><td>- WA</td><td>Working Area</td></tr><tr><td>- WEF</td><td>WEighted Function</td></tr></table>

## Terminology

* **Accuracy**: Accuracy refers to the agreement between a measurement and the true or correct value [1].

* **Error**: Error refers to the disagreement between a measurement and the true or accepted value [1].

* **Heterogeneity**: Heterogeneity refers to the uniformity of the structure of a particular substance.

* **Precision**: Precision refers to the repeatability of measurement [1].

* **Uncertainty**: Uncertainty of a measured value is an interval around that value such that any repetition of the measurement will produce a new result that lies within this interval. There is two categories of uncertainties: systematic (i.e. consistently result on a too small or too large value) and random (i.e. variation without predictable pattern) [1].

* **Uniformity**: Uniformity is the quality of being uniform of something.
# 1 Introduction

All passive microwave satellite sensors (SMOS, SMAP, AMSR-E, AMSR2, SSMI, SMMR ...) are characterized by their low spatial resolution of several tens of square kilometers, typically ~43x43 km for SMOS [2]. The validation of the derived geophysical variables is then an open issue as the reference measured at ground level cannot fully represent what is actually monitored by the sensors. This technical note presents the particular case of SMOS soil moisture product and aims to better assess its accuracy using the International Soil Moisture Network (ISMN) as reference.

## 1.1 Microwave satellite validation issues

One of the objective of Earth Observation data from satellite is to determine the best possible representation of an "actual" but unknown theoretical "true" value. This potential of representation depends on the estimation accuracy. In order to assess the accuracy of satellite data, comparison with ground measurements are often performed, called "validation".

In the case of the soil moisture, satellite missions have been used since 1970's in order to estimate, at the global scale, soil water quantity at the surface level. Different technologies have been conventionally used: passive and active microwave radiometry with different wavelengths, infrared sensors... SMOS is the first dedicated satellite mission, with an optimized frequency of observation of 1.4 GHz (L-Band) and multiple incidence angle observations helping in separating ground contributions and vegetation contributions (a very unique feature).

A challenge common to these missions is to evaluate the quality of their derived geophysical variables, which is complex due to the diverse nature of uncertainties: from the satellite side (RFI, auxiliary data uncertainties...) and the *in-situ* side (calibration, technology representativeness...). Moreover, the spatial and temporal scale differences between a probe's measurement (~cm and ~min) and a satellite observation (~km and ~days) [3] is an issue to assess.

## 1.2 Objectives

In this context, the ESA's project Fiducial Reference Measurement for Soil Moisture (FRM4SM) is investigating the validation strategy of SMOS soil moisture as well as the uncertainty assessment. The objectives of the present study are the following: 1) to assess a global validation of SMOS using the ISMN in order to improve the SMOS uncertainty representation, 2) to evaluate the SMOS accuracy specification of 0.04 m³.m⁻³ in the committed areas and 3) to characterise uncertainties elsewhere and to relate them to the geophysical surface conditions within the SMOS footprint.

The assessment steps proposed here are summarized in Figure 1: (i) define and discuss a relevant validation chain, (ii) investigate the validation scores of SMOS on the whole ISMN database, (iii) study the sensitivity of the scores on the probes configuration and SMOS footprint content, (iv) determine statistical models between scores and geophysical descriptors and (v) attempt a first mapping of SMOS SM uncertainties at global scale.

<figure><img src="image_2.png" /><figcaption>Figure 1: Technical Note 1 flowchart</figcaption></figure>
# 2 Data

This section introduces the data used in this study, namely: *in-situ* soil moisture measurements from the ISMN, satellite soil moisture estimates from the SMOS L2 soil moisture product and the different geophysical auxiliary data.

## 2.1 *In-situ* data: the International Soil Moisture Network

### 2.1.1 Presentation

Initiated in 2009 by the SMOS project and the SMOS Cal Val team, the International Soil Moisture Network (ISMN) [4] is a global database of harmonized soil moisture *in-situ* measurements supported by the Vienna University of Technology (TUWIEN), the European Space Agency (ESA), the Committed on Earth Observation Satellites (CEOS), the Global Energy and Water Exchange Project (GEWEX), the Global Climate Observing System (GCOS), the Global Terrestrial Network on Hydrology (GTN-H), the Group on Earth Observations (GEO) and the Rutgers University. The database (available on the web portal: https://ismn.geo.tuwien.ac.at/en/) is built from a set of 72 networks representing more than 2800 stations over the world which are providing soil moisture measurements since 1950 for the oldest ones. These time series are measured by various probe brands (i.e. technology, calibration), at various depth... For time consistency with the SMOS mission, only the data acquired within the time window 2010-2020 are used here (Figure 2). There are considered as references in this study despite uncertainties associated with the measurements such as identified in section 3.2.

### 2.1.2 Sites description

<figure><img src="image_3.png" /><figcaption>Figure 2: ISMN sites location in this study</figcaption></figure>

In Figure 2, one can notice the heterogeneity in terms of spatial distribution of the probes. The distribution covers only a limited number of climate and surface conditions, which implies that some conditions can not be characterized.
<table><caption>Table 2: Data from https://ismn.geo.tuwien.ac.at/en/networks/</caption><thead><tr><td>Name</td><td>Country</td><td>n stations</td></tr></thead><tbody><tr><td>AACES</td><td>Australia</td><td>49</td></tr><tr><td>AMMA-CATCH</td><td>Bénin, Niger, Mali</td><td>7</td></tr><tr><td>ARM</td><td>USA</td><td>35</td></tr><tr><td>AWDN</td><td>USA</td><td>50</td></tr><tr><td>BIEBRZA</td><td>Poland</td><td>30</td></tr><tr><td>BNZ-LTER</td><td>Alaska</td><td>12</td></tr><tr><td>CALABRIA</td><td>Italy</td><td>5</td></tr><tr><td>CAMPANIA</td><td>Italy</td><td>2</td></tr><tr><td>COSMOS</td><td>USA</td><td>109</td></tr><tr><td>CTP_SMTMN</td><td>China</td><td>57</td></tr><tr><td>DAHRA</td><td>Sénégal</td><td>1</td></tr><tr><td>FLUXNET_AMERIFLUX</td><td>USA</td><td>8</td></tr><tr><td>FMI</td><td>Finland</td><td>27</td></tr><tr><td>FR_Aqui</td><td>France</td><td>5</td></tr><tr><td>GROW</td><td>UK</td><td>150</td></tr><tr><td>GTK</td><td>Finland</td><td>7</td></tr><tr><td>HOBE</td><td>Denmark</td><td>32</td></tr><tr><td>HYDROL_NET_PERUGIA</td><td>Italy</td><td>2</td></tr><tr><td>ICN</td><td>USA</td><td>19</td></tr><tr><td>IIT_KANPUR</td><td>India</td><td>1</td></tr><tr><td>IMA_CAN1</td><td>Italy</td><td>12</td></tr><tr><td>IPE</td><td>Spain</td><td>2</td></tr><tr><td>Lab_NET</td><td>China</td><td>4</td></tr><tr><td>MOL_RAO</td><td>Germany</td><td>2</td></tr><tr><td>MySMNet</td><td>Malaysia</td><td>7</td></tr><tr><td>ORACLE</td><td>France</td><td>6</td></tr><tr><td>OZNET</td><td>Australia</td><td>38</td></tr><tr><td>PBO_H2O</td><td>USA</td><td>159</td></tr><tr><td>PTSMN</td><td>New Zealand</td><td>20</td></tr><tr><td>REMEDHUS</td><td>Spain</td><td>24</td></tr><tr><td>RISMA</td><td>Canada</td><td>24</td></tr><tr><td>RSMN</td><td>Romania</td><td>20</td></tr><tr><td>SCAN</td><td>USA</td><td>239</td></tr><tr><td>SMOSMANIA</td><td>France</td><td>23</td></tr><tr><td>SNOTEL</td><td>USA</td><td>441</td></tr><tr><td>SOILSCAPE</td><td>USA</td><td>171</td></tr><tr><td>SWEX_POLAND</td><td>Poland</td><td>6</td></tr><tr><td>TERENO</td><td>Germany</td><td>5</td></tr><tr><td>UDC_SMOS</td><td>Germany</td><td>11</td></tr><tr><td>UMBRIA</td><td>Italy</td><td>13</td></tr><tr><td>USCRN</td><td>USA</td><td>115</td></tr><tr><td>VAS</td><td>Spain</td><td>3</td></tr><tr><td>VAD</td><td>USA</td><td>4</td></tr><tr><td>WEGERNET</td><td>Austria</td><td>12</td></tr><tr><td>WSMN</td><td>UK</td><td>8</td></tr><tr><td>iRON</td><td>USA</td><td>10</td></tr></tbody></table>
### 2.1.3 The ISMN metadata

The ISMN data contains metadata that describes the conditions at the site such as soil parameters, latitude/longitude, depth, land cover, Köppen classes and more, that are detailed in 2.3.

## 2.2 Satellite data: Soil Moisture and Ocean Salinity

### 2.2.1 Presentation

The Soil Moisture and Ocean Salinity (SMOS) is an ESA mission (with CNES and CDTI) of the Earth Explorer Mission satellite program. SMOS was launched in November 2009 [2] in order to provide surface soil moisture and sea salinity fields at global scale. These variables are keys to a better understanding of the water and energy balances at global scale [5].

The SMOS concept overview is presented in Figure 3. Based on synthetic aperture radiometer at L-band (1.417 GHz), SMOS measures fully polarised multi-angular Brightness Temperatures (BT), as represented by the blue curve in the 'Parameters retrieval' panel Figure 3. In order to retrieve the geophysical variables such as the soil moisture, the Radiative Transfer Model (RTM) L-MEB (L-band Microwave Emission of the Biosphere) [6], is then used to model the angular brightness temperature profile as measured by SMOS [7] [8]. This model generates the green curve in the 'Parameters retrieval' panel Figure 3. This simulation is run with auxiliary data presented in the next section (2.3), which describes as accurately as possible the SMOS footprint content in terms of land cover, soil properties, vegetation coverage, ground and vegetation temperatures... Next, SMOS and modelled TB are compared and the modelled brightness temperatures profile is fitted on the SMOS measurements by a cost function minimization. In most cases the two retrieved parameters are the Soil Moisture (SM) and the Vegetation Optical Depth (VOD), or more rarely, SM and roughness HR in some particular cases, such as in dry areas. More details can be found in the level 2 ATBD document.

<figure><img src="image_4.png" /><figcaption>Figure 3: Principle of the soil moisture estimation by SMOS</figcaption></figure>
### 2.2.2 SMOS grid, Weighted Function, Working Area and Fraction

SMOS products are projected on two main grids depending on the level of processing. SMOS Level 2 products are processed for each Discrete Global Grid (DGG) nodes of the isea4H9 grid (~15 km grid [9]), depicted by the red dots in Figure 4.

A retrieval at a given DGG node accounts for the spatially-distributed information participating to the SMOS-observed BT, as weighted components by BT synthetic Antenna Footprint Patterns (AFP) projected onto the Earth. Depending on the BT observation geometry, these AFPs define various Earth ellipsoids.

The Earth surface spatial information was gathered from several auxiliary data files (ADF). This information includes temperatures (ground, vegetation ...), soil texture (sand, clay, organic matter content), vegetation (LAI, LAI max), land-cover classes and their fractions, provided on the almost equal area Discrete Flexible Fine Grid (DFFG) at a higher resolution with ~4x4 km cells.

<figure><img src="image_5.png" /><figcaption>Figure 4: Summary of the grids in the Level 2 products</figcaption></figure>
The retrievals account for this spatially-distributed information in a Working Area (WA) of ~123x123 km centered on the DGG [10] that corresponds to all the antenna pattern footprint configurations. WAs are always composed of 35x35 DFFG cells as illustrated in Figure 4 in blue, showing the DFFG grid cells limited to the WA positioned around its central red dot DGG. Such a WA size allows to capture more than 99% of the microwave flux integrated by SMOS through its synthetic antenna lobes, even for those leading to the largest Earth surface and/or most elongated AFPs considered in the retrieval.

All quantities in the retrieval algorithm are considered as radiometric quantities, defined as the convolution of Earth surface spatially-distributed values with the normalized antenna gain patterns projected at the Earth surface. We consider them as either WEF or MEANWEF weighting functions. WEF are attached to particular BTs and thus depend on the BT observation geometry, while MEANWEF is an averaged weighting function representing the average of all the WEF sizes and shapes that SMOS geometries can produce. Conceptually the MEANWEF represents a nadir pointing antenna centrosymmetric pattern, with a -3 dB half-power defining a circular average resolution of ~40 km as depicted by the red disc on Figure 4. The densest MEANWEF central part corresponds to the higher weights, the larger antenna gain.

### 2.2.3 The Radio Frequency Interference issues

SMOS is measuring the surface emission in the protected band range 1.4-1.427 GHz, part of the L-band spectrum, dedicated to the Earth Exploration Satellite Service (EESS) [11] [12]. As any microwave radiometer ([13, 14]), the SMOS receiver is very sensitive as the surface emission at L-band is very low. However, since the SMOS launch in 2009, Radio Frequency interference (RFI) have been observed all around the world. Those RFI are mainly due to 1) unauthorized emissions within the protected passive band coming from active sources and 2) unwanted emissions from active services operating in neighboring bands. The consequence of these radio interference phenomena is the impossibility in some regions to measure the natural emission of the surface, the latter being overwhelmed by RFI, or, even worse, contaminated by levels similar to the Earth thermal emission. The map in Figure 5 shows the areas which are the most affected over a period of one year: i) in green the data are not impacted, ii) from green to yellow green, the data are certainly impacted at "unknown" level, iii) in yellowish the impact on the retrieval is visible and iv) in red the retrieval is failing. The impact on the validation process is significant, and so it has to be taken into account by filtering the contaminated SMOS data, as presented in section 4.3.1.

The map in Figure 5 is an annual average of the RFI (04/2020-04/2021). Large interference areas can be identified, e.g. in the Middle East, China or Japan. More local interference can also be identified with the "red dots", in South America for example. Nevertheless, the data displayed on the map are the average, meaning that RFI can pop up suddenly in the clean areas (green in Figure 5).
<figure><img src="image_6.png" /><figcaption>Figure 5: Map of averaged RFI for the period 04/2020 - 04/2021</figcaption></figure>
## 2.3 Geophysical auxiliary database

The databases presented here are used to describe the geophysical conditions within a SMOS footprint. Some of them are used in the operational retrieval algorithm to process the SMOS level 2 soil moisture (*), others are included in the metadata of the ISMN database (**) to provide information about the probe environment. Finally, a last group of data is used for the purpose of the analysis conducted in this study and additional perspectives (***).

* Soil texture & composition :

- FAO HWSD*,**: Global map of soil texture and composition. It is a merging of more than 16000 regional and national soil information maps (SOTER, ESD, Soil Map of China, WISE) and the FAO-UNESCO Soil Map of the World (FAO, 19711981), with a spatial resolution of 30 arc-second (~1x1 km) [15].

- Soilgrid*** V2.0 (2020): SoilGrids is a system for global digital soil mapping that uses over 230 000 soil profiles observations from the WoSIS database and a series of environmental covariates. Covariates were selected from a pool of over 400 environmental layers from Earth observation derived products and other environmental information including climate, land cover and terrain morphology [16].

* Land cover classification:

- IGBP*: International Geosphere-Biosphere Programme (IGBP) is a land cover database derived from satellite observation, as Advanced Very High Resolution Radiometer (AVHRR) and other remote sensing data [17].

- ESA CCI Landcover**: Derived from a multi-year and multi-sensor strategy [18].

* Vegetation :

- Vegetation index LAI*: MODIS [19]

- Above Ground Biomass***: The AGB used here is from the GlobBiomass database [20]. It was derived from growing stock volume obtained by radar and optic remote sensing.

* Fixed or initial surface condition ($T_{surf}$, $T_{deep}$, $\theta_{surf}$)*: ECMWF

* Climatic classification***: The Köppen Climate Classification divides the Earth's climate into five main climate groups: A (tropical), B (dry), C (temperate), D (continental), E (polar). These are subdivided by seasonal precipitation and heat. It was first published by the Russian-German climatologist Wladimir Köppen in 1884, with several later modifications by Köppen and others, notably Rudolf Geiger, hence the system is sometimes also called the Köppen-Geiger climate classification system [21].

* SNOW***: ERA5-Land is a reanalysis data set providing a consistent view of the evolution of land variables over several decades at an enhanced resolution compared to ERA5. ERA5-Land has been produced by replaying the land component of the ECMWF ERA5 climate reanalyses. Reanalysis combines model data with observations from across the world into a globally complete and consistent data set using the laws of physics. Reanalysis produces data that span several decades back in time, providing an accurate description of the climate of the past [22].

<figure><img src="image_7.png" /></figure>

This section introduced the data used in this study, namely the ISMN data (considered as reference), the satellite based SMOS data, and the auxiliary data used to describe the surface condition.
# 3 Validation procedure description

First, this section introduces the validation process to compare SMOS retrieved soil moisture and the measured soil moisture by ISMN. Second, we attempt to list the sources of uncertainties. Finally the statistical metrics defined.

## 3.1 Validation chain description

The validation chain aims at comparing two data sets. Several steps are needed to insure the good quality of the data and to collocate the data sets in space and time. All steps are presented in a flow chart in Figure 6, inspired by the recommendation of Gruber et. al [23].

<figure><img src="image_8.png" /></figure>

Figure 6: Validation chain used in this project. The bottom left figure ① represents SMOS grid nodes (red dots) and *in-situ* sites from the ISMN (blue crosses). The arrows indicate the spatial collocation of the *in-situ* site and SMOS nodes. The bottom right figure ② is a hypothetical SMOS SM time series (blue) and an *in-situ* measurements (red) to illustrate the difference in time sampling.

A detailed description of this chain is presented in the following:

* **ISMN and SMOS database**

The *in-situ* and SMOS time series are provided with their respective time of acquisition, as represented in Figure 7.

* **Spatial sampling**

The aim is to spatially collocate the two data. Several strategies exist, the Nearest Neighbor method is the one used in our analysis. For each *in-situ* probes (blue crosses bottom left Figure 6), the nearest SMOS DGG node (red points bottom left Figure 6) is attributed. The maximum distance between a probe location and its attributed DGG is thus less than ~7.5 km.
<figure><img src="image_9.png" /><figcaption>Figure 7: Top: SM in-situ time series from 0.05 cm probes, Bottom: SMOS time series, for AMMA-Catch site of Belefoungou in Bénin.</figcaption></figure>

### Probe selection

Different types of measurements are available on the ISMN sites (soil moisture, air temperature, precipitation...) but only the soil moisture data is selected for our purpose.

### RFI Filter

In order to filter the Radio Frequency Interference, the SMOS L2 contains 2 metadata fields that are the 15 days probability *RFI_P* and the daily probability *RFI_P_DAY*:

$$RFI_P_DAY = \frac{N\_RFI\_X + N\_RFI\_Y}{M\_AV\_A0} \qquad (1)$$

The degree of RFI contamination, and the relevance of the retrieval, can then be evaluated. A threshold of 0.1 is recommended, meaning that less than 10% of SMOS TB were considered as affected by RFI. This threshold ensures the good quality of the derived SM. An undetected lower level of RFI contamination, below the detection thresholds, is always possible and may impact the retrieved soil moisture.

### Filter extreme SM values

To avoid non-physical values, the data have to be within the considered range: $0 < \theta < 0.8 \text{ m}^3\text{m}^{-3}$.

### Temporal sampling

The time collocation is also calculated with the Nearest Neighbor method, as was done for the spatial sampling. For every SMOS observation, the closest *in-situ* SM in time is retained, within a limit of $|\Delta t| \le 30 \text{ min}$ to ensure that the SMOS observation and the *in-situ* measurement are temporally consistent.

### Scores

For each comparison between *in-situ* and satellite data, the Pearson correlation coefficient R, the RMSE (Root Mean Square Error), the ubRMSE (unbiased Root Mean Square Error) and the Bias are computed to evaluate time consistency, global accuracy, random uncertainties and static uncertainties, respectively. Section 3.3 provides the details of these statistics.
## 3.2 Uncertainty sources investigation

All measurements bear uncertainties. This section aims at identifying the possible sources of these uncertainties for the two cases of interest, namely SMOS and in-situ. This part is a first attempt to identify them and their origin, a summary of which can be found in Table 3. Figure 8 is a conceptual representation of uncertainty propagation, from the theoretical soil moisture $\theta^{TRUE}$, sampled by the two sensors, to the validation.

<figure><img src="image_10.png" /><figcaption>Figure 8: Sources of uncertainties, the lists below are not exhaustive</figcaption></figure>

* **u(Geophysical)**

First, the theoretical soil moisture $\theta^{TRUE}$ is sampled with two sensors that are characterized with two different spatio-temporal scales. The result is two soil moisture, $\theta_{SMOS}^{TRUE}$ which is the target soil moisture estimated by SMOS, and $\theta_{INSITU}^{TRUE}$ which is the "true" value measured by the probes. The difference between these two "true" values is due to the difference of scales of the geophysical processes these two systems are access to when observing $\theta^{TRUE}$, represented in Figure 8 by u(Geophysical). In fact, SMOS integrates geophysical process by a spatial resolution of ~43×43 km when the in-situ is much more influenced by local process of few centimeters around the probes. So the heterogeneity and uniformity distribution of the land cover or soil properties are strongly impacting the difference between $\theta_{SMOS}^{TRUE}$ and $\theta_{INSITU}^{TRUE}$. Moreover the geographical localization is leading to different water cycle intensity. An intense precipitation or a strong evapo-transpiration regime can increase the consequences of the temporal sampling difference of SMOS (~days) and the probes (~minutes).

* **u(SMOS)**

$\theta_{SMOS}^{EST}$ is the result of accessing $\theta_{SMOS}^{TRUE}$ through the whole SMOS observation system characteristics an imperfections. It includes the radiometer characteristics and limitations, like radiometric noise, image reconstruction errors, scene dependent biases ... propagating into the retrieval system. The retrieval system has also its own characteristics and limitations like models assumptions, approximations, non error free fixed parameters ... All these characteristics and imperfection are bound together trough the cost function reaching finally the retrieved parameters as the soil moisture is. So, u(SMOS) is representing the uncertainty chain, summarized in the Table 3.

* **u(INSITU)**

$\theta_{INSITU}^{EST}$ is the result of accessing $\theta_{INSITU}^{TRUE}$ through the in-situ measurement system characteristics and imperfections. u(INSITU) represents the uncertainties due to the device/sensor, which includes the measurements error, the conversion from the raw measurements to the SM and the soil conditions. These are uncertainties are developed in the Table 3.
<table><caption>Table 3: Uncertainty budget (S for systematic, R for random and ? for open question)</caption><thead><tr><th></th><th>Uncertainties</th><th>Description</th></tr></thead><tbody><tr><td rowspan="7">u(In-situ)</td><td>u/installation)<sup>S</sup></td><td>uncertainties due to the probe installation, rod contact, orientation, soil compacting... The contact surface sensor/soil is fundamental to insure an accurate measurement with a good quality</td></tr><tr><td>u(calibration)<sup>S</sup></td><td>uncertainties of the calibration law (cf θ = f(ε)) Strongly impacts the in-situ soil moisture value, more information by asking to the providers which calibration was used.</td></tr><tr><td>u(technology)<sup>S</sup><sub>R</sub></td><td>uncertainties due to the spatial representativeness of the sensor technology; for instance COSMOS and TDR technologies are not based on the same physics. Some ideas: 1) look empirically which technology gives the better results 2) discuss the results with appropriate references on the technologies</td></tr><tr><td>u(deterioration)<sup>S</sup><sub>R</sub></td><td>uncertainties due to the probe's deterioration Hard to determine, could be reduced by a regular re-calibration.</td></tr><tr><td>u(depth)<sup>S</sup><sub>R</sub></td><td>uncertainties on the probe depth Important regarding to the radiometry penetration depth. Could be reduced by asking to the provider.</td></tr><tr><td>u(soildensity)<sup>S</sup><sub>R</sub></td><td>uncertainties on the bulk density (compacting, plowing)</td></tr><tr><td>u(soilproperties)<sup>S</sup><sub>R</sub></td><td>uncertainties on the soil compositions %sand, %clay, %silt, %organic... could be reduced by using soil database (HWSD, SoilGrid) or by asking to the provider.</td></tr><tr><td rowspan="7">u(SMOS)</td><td>u(land cover)<sup>S</sup><sub>R</sub></td><td>uncertainties on the probe land cover site Probes location on field ? Under trees ? Could be reduced by the using land cover database (CCI-Landcover), or by asking to the provider.</td></tr><tr><td>u(sensor)<sup>S</sup><sub>R</sub></td><td>uncertainties on sensor Antenna non-linearity, noise, ... Can be characterized by analysing the DQX distribution</td></tr><tr><td>u(resolution)<sub>R</sub></td><td>uncertainties on the radiometric resolution Antenna gain, aliasing, ...</td></tr><tr><td>u(RFI)<sup>S</sup><sub>R</sub></td><td>uncertainties on the RFI RFI detection algorithm provide information on the RFI probability</td></tr><tr><td>u(data_aux)<sup>S</sup><sub>R</sub></td><td>uncertainties on the model database Model relies on data base of soil texture, vegetation, initial values...</td></tr><tr><td>u(equation)<sup>S</sup></td><td>uncertainties on the Radiative Transfer Model (RTM) equations Non-linearity, dielectric model, scale issues, ... Can be investigated by breadbording exercise</td></tr><tr><td>u(inversion)<sup>S</sup><sub>R</sub></td><td>uncertainties on the retrieval quality Dependent of u(equation), u(data_aux)... Can be investigated through the CHI2 metadata</td></tr><tr><td rowspan="2">u(Geophysical)</td><td>u(spatialscale)?</td><td>uncertainties on the spatial scale mismatch Heterogeneity and uniformity of the geophysical process : Can be reduced by determine areas where the probes representativeness is increased and by modelling</td></tr><tr><td>u(temporalscale)?</td><td>uncertainties on the temporal scale mismatch: due to the temporal dynamic of the geophysical process: Difference of sampling frequency between SMOS (~days) vs. in-situ (~min)</td></tr></tbody></table>
## 3.3 Metrics

Our analysis is based on statistic metrics that are commonly used by the community to describe and compare data. They are reported down below in the Table 5.

<table><caption>Table 4: Summary of the metrics used for validation. $\theta$ represents the SM.</caption><thead><tr><th>Name</th><th>Description &amp; Utilization</th><th>Formula</th></tr></thead><tbody><tr><td>MEAN</td><td>Represents the mean value of a soil moisture time series.</td><td>$\theta_{mean} = \frac{1}{n} \sum_{i=1}^{n} \theta_{i}$</td></tr><tr><td>STD</td><td>Represent the dynamic of a soil moisture time series.</td><td>$\sigma = \sqrt{\frac{1}{n-1} \sum_{i=1}^{n} (\theta_{i} - \theta_{mean})^{2}}$</td></tr><tr><td>BIAS</td><td>Represent the mean difference of SMOS and ISMN (static uncertainty)</td><td>$Bias = \frac{1}{n} \sum_{i=1}^{n} (\theta_{i}^{SMOS} - \theta_{i}^{ISMN})$</td></tr><tr><td>MAE</td><td>Represent the absolute mean difference of SMOS and ISMN</td><td>$MAE = \frac{1}{n} \sum_{i=1}^{n} |\theta_{i}^{SMOS} - \theta_{i}^{ISMN}|$</td></tr><tr><td>MSE</td><td>Represent the mean square difference of SMOS to ISMN</td><td>$MSE = \frac{1}{n} \sum_{i=1}^{n} (\theta_{i}^{SMOS} - \theta_{i}^{ISMN})^{2}$</td></tr><tr><td>RMSE<br>RMSD<br>SEE</td><td>Represent the global accuracy of SMOS to estimate ISMN</td><td>$RMSE = \sqrt{\frac{1}{n} \sum_{i=1}^{n} (\theta_{i}^{SMOS} - \theta_{i}^{ISMN})^{2}}$</td></tr><tr><td>ubRMSE<br>ubRMSD<br>SDD</td><td>Represent the unbiased accuracy of SMOS to estimate ISMN (random uncertainty)</td><td>$ubRMSE = \sqrt{RMSE^{2} - Bias^{2}}$</td></tr><tr><td>R Pearson</td><td>Represent the SMOS ability to represent the temporal dynamic of the in-situ time series</td><td>$R_{p} = \frac{\sum_{i=1}^{n} (\theta_{i}^{SMOS} - \theta_{mean}^{SMOS}) (\theta_{i}^{ISMN} - \theta_{mean}^{ISMN})}{\sqrt{\sum_{i=1}^{n} (\theta_{i}^{SMOS} - \theta_{mean}^{SMOS})^{2} (\theta_{i}^{ISMN} - \theta_{mean}^{ISMN})^{2}}}$</td></tr></tbody></table>

* Graphical representation

To represent the scores, we use mostly histograms and box plots (displaying the median (red line), 25th and 75th percentile).

* Correlation average:

Because R Pearson coefficients are not additive, it is not recommended to average them directly. Alexander et al. [24] to compute the average as follows:

1. Transform R to Fisher z-score:

$$z_{i} = 0.5 \frac{\log 1 + R_{i}}{1 - R_{i}} = \tanh^{-1}(R_{i})$$

2. Compute $\tilde{z}$:

$$\tilde{z} = \frac{\sum_{i=1}^{n} (n_i - 3) z_i}{\sum_{i=1}^{n} (n_i - 3k)}$$

with $n_i$ the number of sample used to compute $R_i$, and $k$ the number of $R_i$ to average

3. Transform $\tilde{z}$ to $\bar{R}$:

$$\bar{R} = \tanh \tilde{z}$$

* Bootstrap Confidence Interval (CI):

In order to compute the 95% confidence interval and to compare the scores, a bootstrap method is used for each score computation which can be averaged. To compute this CI, the function 'bootci' with 'bca: Bias corrected and accelerated percentile' [25] of MATLAB is used.

<figure><img src="image_11.png" /></figure>

The validation chain was described, the uncertainty chain discussed and the metrics presented. In the follow a global benchmark using this process to validate SMOS on the ISMN is investigated.
# 4 Analysis of the scores and the uncertainties

This section contains the results of the comparison between SMOS and the *in-situ* measurements. First, a global validation is processed considering all the available probes, whatever their depths and technology. Then the scores are analysed through the probe configuration point of view. Finally the scores are studied regarding the SMOS footprint content.

## 4.1 Overview of validation SMOS vs. ISMN

An overview of the scores obtained from the global comparison SMOS vs. ISMN using the validation chain, considering all the probes at every depths (i.e. from the surface down to 1 m.), is displayed in Figure 9.

<figure><img src="image_12.png" /><figcaption>Figure 9: Histograms of the scores when comparing SMOS to all the probe time series, including all possible depths. Number of samples: 5674 *in-situ* time series and 1018 SMOS time series.</figcaption></figure>

<table><caption>Table 5: Score summary with confidence (CI) intervals in brackets.</caption><thead><tr><th>Probe depth</th><th>R*</th><th>RMSE(m<sup>3</sup>.m<sup>-3</sup>)</th><th>ubRMSE(m<sup>3</sup>.m<sup>-3</sup>)</th><th>Bias(m<sup>3</sup>.m<sup>-3</sup>)</th></tr></thead><tbody><tr><td>&le; 1m<sub>5674/5674probes.</sub></td><td>0.462<sub>[0.349;0.492]</sub></td><td>0.142<sub>[0.134;0.150]</sub></td><td>0.087<sub>[0.079;0.093]</sub></td><td>-0.069<sub>[-0.060;-0.079]</sub></td></tr></tbody></table>
Figure 9 contains the histograms of the various scores, such as:

* The Pearson correlation coefficient (top left), considering a *p*-value 0.05. Globally the correlation is positive with a mode of ~0.5 and a gap around zero due to the non-respect of the *p*-value threshold. This conveys the globally positive ability of SMOS to represent the temporal variation measured by the *in-situ* with an average of 0.462, as shown in the Table 5.

* In Figure 9 (top right), the histogram shows the RMSE distribution. The mode is less than 0.1 m³.m⁻³ and the average is 0.142 m³.m⁻³.

* In Figure 9 (bottom left) the histogram shows the ubRMSE distribution. The mode is less than 0.1 m³.m⁻³ and the average is 0.087 m³.m⁻³, which is more than twice the MRD (0.04 m³.m⁻³).

* In Figure 9 bottom right corner the histogram is showing the Bias (SMOS-insitu) distribution. The mode is slightly negative, showing a global underestimation of soil moisture regarding to *in-situ*.

All these scores are computed without any assumption on the probes depth, the probes technology, the soil characteristics, ... However all this parameters have a strong influence on the uncertainties presented in Table 3.

Figure 10 shows the bias as a function of the ubRMSE for every site. The color scale represents the R coefficient. The red dots (high R) corresponds to the lower ubRMSE and a bias within the range [-0.2 0.2].

<figure><img src="image_13.png" /></figure>

Figure 10: SMOS performances using the ISMN database as the reference. Each dot represents an *in-situ* site. The ubRMSE is reported along the x-axis, the bias as the y-axis, whereas the color scale represents the Pearson correlation coefficient.

<figure><img src="image_14.png" /></figure>

There is a global agreement between SMOS and the whole ISMN database, and the method seems consistent with respect to the score distribution shapes. Further investigation/analysis is needed to relate the performance of SMOS with conditions such as the probe's depth or the presence of vegetation in the SMOS footprint, for example.
## 4.2 Score sensitivity to the *in-situ* probes configuration

Currently, various technologies exist to measure soil water content (capacitive, TDR, ...), with specific features such as pin length, measurement frequency, probes position - vertical, horizontal, slanted and so on and so forth. Such a diversity of features makes cross-comparison between sites difficult. We suggest in this section to study/analyse SMOS SM performances with respect to these features.

### 4.2.1 To the probe depth

Soil moisture depends strongly on soil depth due to several physical processes (infiltration, evaporation, roots suction...). Moreover, even if it is admitted that the penetration depth at 1.4 GHz is about 5 cm, it depends strongly on the soil moisture content and texture (i.e. clayey expected shallower, sandier expected deeper) [26] [27]. More recent studies assessing the penetration depth confirm what was found [28] [29] [30] [31]. Considering this context, we conducted an evaluation of SMOS SM performances according to various sampling depths. An example of absolute soil moisture values measured by the probes are shown as colored lines in Figure 11, and the SMOS observation are the black dots.

<figure><img src="image_15.png" /><figcaption>Figure 11: SMOS data in black point and *in-situ* time series for different depth in colored line.</figcaption></figure>

Four groups have been defined regarding the total distribution: $d \le 0.1\text{m}$ (43%), $0 < d \le 0.25\text{m}$ (27%), $0 < d \le 0.5\text{m}$ (26%) and $0 < d \le 1\text{m}$ (4%), results are presented in Figure 12 with the scores reported in Table 6:

* In Figure 12 top left histograms show the correlation ($p<0.05$) for different depths. The global correlation is better when only considering the probes near to the surface. In fact the average correlation is increasing from 0.462 considering all the probes to 0.562 considering only the probes between 0 and 0.1m.

* In Figure 12 top right histograms show the distribution of scores of RMSE with different depth. Modes are going to a better RMSE as the probes reach the surface, in average from $0.142 \text{ m}^3 \cdot \text{m}^{-3}$ to $0.128 \text{ m}^3 \cdot \text{m}^{-3}$.

* In Figure 12 top right histograms show the distribution of scores of ubRMSE for various depths. Modes are going to a better ubRMSE as the probes reach the surface, in average from 0.087 $\text{m}^3 \cdot \text{m}^{-3}$ to 0.082 $\text{m}^3 \cdot \text{m}^{-3}$.

* In Figure 12 bottom right histograms show the distribution of scores of Bias (SMOS-insitu) for various depths. Modes from deep to surface are going to a lower Bias, in average from -0.069 $\text{m}^3 \cdot \text{m}^{-3}$ to -0.051 $\text{m}^3 \cdot \text{m}^{-3}$, but it still negative, showing a global underestimation of soil moisture regarding to *in-situ*.
<figure><img src="image_16.png" /><figcaption>Figure 12: Distribution of scores per range of probes depth</figcaption></figure>

<table><caption>Table 6: Table of scores regarding to the probes depth</caption><thead><tr><td>Probe depth</td><td>R*</td><td>RMSE(m<sup>3</sup>.m<sup>-3</sup>)</td><td>ubRMSE(m<sup>3</sup>.m<sup>-3</sup>)</td><td>Bias(m<sup>3</sup>.m<sup>-3</sup>)</td></tr></thead><tbody><tr><td>≤ 1m<sub>5674/5674prob.</sub></td><td>0.462[0.349;0.492]</td><td>0.142[0.134;0.150]</td><td>0.087[0.079;0.093]</td><td>-0.069[-0.060;-0.079]</td></tr><tr><td>≤ 0.5m<sub>4661/5674prob.</sub></td><td>0.473[0.36;0.502]</td><td>0.138[0.130;0.146]</td><td>0.085[0.076;0.091]</td><td>-0.064[-0.054;-0.074]</td></tr><tr><td>≤ 0.25m<sub>3935/5674prob.</sub></td><td>0.523[0.414;0.550]</td><td>0.133[0.130;0.146]</td><td>0.084[0.076;0.090]</td><td>-0.059[-0.050;-0.068]</td></tr><tr><td>≤ 0.1m<sub>2576/5674prob.</sub></td><td>0.562[0.450;0.583]</td><td>0.128[0.130;0.146]</td><td>0.082[0.074;0.088]</td><td>-0.051[-0.042;-0.060]</td></tr></tbody></table>
Another representation of the score dependence to the probe depth is proposed in Figure 13. The correlation coefficient R shows a significant improvement of 0.25 when considering the probes at the surface level, i.e. 0-0.1 m, comparing to the 0.5-1 m range. The theoretical L-band radiometry penetration depth is ~5 cm [32] but it can vary with the soil texture and soil moisture content. It can be less [28] or a bit more in case of sandy dry soil.

<figure><img src="image_17.png" /><figcaption>Figure 13: Boxplots displaying the median (red line), 25th and 75th percentile (bottom and top edges of the box), of the scores R, ubRMSE, Bias, RMSE by range of depths (top y axis is the surface level)</figcaption></figure>

Note that a limit of 0.1 m is preferred at the expected 0.05 m because: i) measuring the soil moisture at 0.05 m is quite complex due to the uncertainties on the volume monitored by the device, ii) it adds 700 probes to the data set and so it increases the robustness of the statistics and iii) the performances are similar in both cases ($\Delta R = 0.014$, $\Delta \text{BIAIS} = 0.0003 \text{ m}^3 \cdot \text{m}^{-3}$, $\Delta \text{RMSE} = 0.002 \text{ m}^3 \cdot \text{m}^{-3}$ and $\Delta \text{ubRMSE} = 0.001 \text{ m}^3 \cdot \text{m}^{-3}$).

<figure><img src="image_18.png" /></figure>

Validation scores improve with shallow top surface layer. The next analyses consider only the probes within the first 10cm.
### 4.2.2 To the probes technologies

The various devices used to measure the soil moisture in the ISMN are developed according to several technologies. In fact, different measurement principles exist like gravimetry, tensiometry, neutron, capacitive, TDR, cosmic-ray and dielectric methods [33]. All of these technologies do not have the same representativeness (soil volume sampling), which has an impact on the scores, as shown in Figure 14.

<figure><img src="image_19.png" /><figcaption>Figure 14: Distribution of scores across the probes references</figcaption></figure>

However, it could be misleading to analyse directly the scores presented in the Figure 14 because of the surface condition variability of each sites in terms of land cover, soil texture, climate, ... Further investigations are needed to evaluate the technologies by considering identical geophysical characteristics for a fair comparison.

<figure><img src="image_20.png" /></figure>

Need to investigate the effect of technology by considering equivalent surface conditions.
### 4.2.3 To the soil characteristics

Most of the probes technology provides an indirect measure of SM that is derived through the measure of either the dielectric constant, pressure of suction, time... This means that the soil characteristics (texture, salt concentration and property) strongly impact the soil moisture measurement. Probes calibration laws have then to be defined in order to convert the probes' measurements into SM information. These laws are a function of the soil characteristics and subject to large uncertainties.

In order to investigate the scores sensitivity to this soil characteristics, a study is performed regarding the HWSD texture database. Soil parameters and corresponding validation scores for each probe are shown in Figure 15.

<figure><img src="image_21.png" /><figcaption>Figure 15: Scores sensitivity to probes soil textures (HWSD)</figcaption></figure>

* **Sand content:** An increasing of the sand content seems to lead to a negative bowl-shape of the correlation coefficient (negative bowl-shape) and a global improvement of the Bias, the RMSE and the ubRMSE.

* **Clay content:** An increasing of the clay content seems to lead to a slight worsening of the RMSE and ubRMSE.

* **Bulk density content:** An increasing of the bulk density seems to lead to improved Bias, RMSE and ubRMSE, but does not change clearly the correlation.

<figure><img src="image_22.png" /></figure>

The validation scores show better results when the probes are located in sandy soil with a high bulk density. However, those trends depend on the database accuracy and on the probes configuration (technology, calibration...).
### 4.2.4 To the ISMN land cover metadata (ESA CCI)

The ISMN metadata that contain information concerning the probes location land cover are provided. The database used here is the ESA CCI Land cover (index in Table 7). The aim here is to investigate the scores variability per land cover classes.

<figure><img src="image_23.png" /><figcaption>Figure 16: Distribution of scores per ESA CCI Land cover reported as index and explained in Table 7.</figcaption></figure>

These results do not show any clear trend. SMOS target of 0.04 m³.m⁻³ is reached for some land cover classes (indexed 10, 12, 70, 100, 120, 130, 180, 190, 200 and 201), see Table 7. However, these results are to be interpreted with caution as they are prone to be biased by probes features.

<figure><img src="image_24.png" /></figure>

As for the probe technology study, more investigation is needed to identify a land cover descriptor.
<table><caption>Table 7: Land cover index</caption><thead><tr><td>idx</td><td>Land cover</td></tr></thead><tbody><tr><td>0</td><td>UNDEFINED</td></tr><tr><td>10</td><td>Bare areas</td></tr><tr><td>11</td><td>Consolidated bare areas</td></tr><tr><td>12</td><td>Cropland, irrigated or post-flooding</td></tr><tr><td>20</td><td>Cropland, rainfed</td></tr><tr><td>30</td><td>Cropland, rainfed / Herbaceous cover</td></tr><tr><td>40</td><td>Cropland, rainfed / Tree or shrub cover</td></tr><tr><td>50</td><td>Grassland</td></tr><tr><td>60</td><td>Lichens and mosses</td></tr><tr><td>61</td><td>Mosaic cropland (&gt;50%) / natural vegetation (tree, shrub, herbaceous</td></tr><tr><td>62</td><td>Mosaic herbaceous cover (&gt;50%) / tree and shrub (&lt;50%)</td></tr><tr><td>70</td><td>Mosaic natural vegetation (tree, shrub, herbaceous cover) (&gt;50%) / cropland (&lt;50%)</td></tr><tr><td>90</td><td>Mosaic tree and shrub (&gt;50%) / herbaceous cover (&lt;50%)</td></tr><tr><td>100</td><td>Shrub or herbaceous cover, flooded, fresh/saline/brackish water</td></tr><tr><td>110</td><td>Shrub-land</td></tr><tr><td>120</td><td>Sparse vegetation (tree, shrub, herbaceous cover) (&lt;15%)</td></tr><tr><td>130</td><td>Tree cover, broad-leaved, deciduous, Closed (&gt;40%)</td></tr><tr><td>140</td><td>Tree cover, broad-leaved, deciduous, Closed to open (&gt;15%)</td></tr><tr><td>150</td><td>Tree cover, broad-leaved, deciduous, Open (15-40%)</td></tr><tr><td>160</td><td>Tree cover, broad-leaved, evergreen, Closed to open (&gt;15%)</td></tr><tr><td>180</td><td>Tree cover, flooded, fresh or brackish water</td></tr><tr><td>190</td><td>Tree cover, mixed leaf type (broad-leaved and needle-leaved)</td></tr><tr><td>200</td><td>Tree cover, needle-leaved, evergreen, closed to open (&gt;15%)</td></tr><tr><td>201</td><td>Urban areas</td></tr><tr><td>210</td><td>Water</td></tr></tbody></table>

### 4.2.5 ISMN Quality Flag analysis

Due to the ongoing development of the ISMN quality flag by the TUWien, this sensitivity study will be investigated later during the FRM4SM project.

### 4.2.6 To the ISMN climate class metadata (Köppen climates classification)

The objective of this part is to study how the climate location of the probes is impacting the scores, because of seasonality, water cycle intensity,... In order to investigate the climatic impact on the scores, a classification is done regarding the Köppen climate classes.

A Tropical (f: Rain forest, m: Monsoon, w: savanna with dry winter, s: savanna with dry summer)

B Arid (w: Desert, s: Steppe, h: Hot, k: Cold)

C Temperate (w: Dry winter, f: No dry season, s: Dry summer, a: Hot summer, b: Warm summer, c: Cold summer)

D Continental (w: Dry winter, f: No dry season, s: Dry summer, a: Hot summer, b: Warm summer, c: Cold summer, d: Very cold winter)

E Polar (T: Tundra, F: Eternal frost)

<figure><img src="image_25.png" /></figure>

As for the land cover analysis dependence, the variability showed here is impacted by the probes technology differences and need an independent analysis.
<figure><img src="image_26.png" /><figcaption>Figure 17: On the top, a map representing the spatial distribution of the ISMN sites and on the bottom the validation scores regarding to Köppen climate classification</figcaption></figure>
### 4.2.7 To the ISMN network ranking

This section introduces the scores per ISMN network. As presented on the map in Figure 2, the ISMN are spread over the globe. However, they do not cover all surface and climate conditions, so the performances reported here are only reliable for the specific conditions covered by the sites. Looking at the scores per network can give an idea of the expected performance:

<figure><img src="image_27.png" /><figcaption>Figure 18: Distribution of scores per ISMN network</figcaption></figure>

<figure><img src="image_28.png" /></figure>

The score distribution per ISMN network allows an analysis per site. However ranking them is biased by the different probe configurations and surface conditions between the sites.
## 4.3 Score sensitivity to the SMOS footprint content

SMOS measures the emission coming from a wide area of ~43 km which is characterized by its surface conditions, that may be different from the *in-situ* site but include it. This section is dedicated to relating SMOS performance with footprint conditions.

### 4.3.1 RFI filtering impact on the scores

SMOS brightness temperatures are artificially increased by RFI sources, which leads to suspicious SM retrieval values. To avoid such bias, the BTs need to be filtered out to remove the contaminated data. The RFI filtering has two effects: 1) removing the contaminated data and 2) reducing the number of samples available for the validation.

#### 1. Effect on the scores

In the Figure 19 delta scores are represented *AFTER FILTERING - BEFORE FILTERING*. The scores are improving when the SMOS SM retrievals associated to high RFI are discarded. However, it may happen in some cases that filtering SMOS SM worsens the scores. One possible explanation is the decrease of the number of samples as represented in Figure 5 and where the effect is presented in section 4.4.

<figure><img src="image_29.png" /><figcaption>Figure 19: RFI filtering impact on the scores per country</figcaption></figure>
## 2. Effect on the number of sample

On the Figure 20 the reduction of samples due to filtering is presented. Some places look very impacted by the filtering, as Austria, Germany, France, Romania... when others do not as Canada, Senegal or Australia. Those results are consistent with the map Figure 5.

<figure><img src="image_30.png" /></figure>

Figure 20: RFI filtering impacts on the number of samples used to compute validation, per site on the left and by country on the right.

<figure><img src="image_31.png" /></figure>

This part shows the improvement of the scores due to RFI filtering, however there is a risk of removing too many samples to make a fair validation.

## 4.3.2 SMOS quality index (DQX, CHI2P)

CHI2P and DQX (representing the soil moisture retrieval quality) are presented in the SMOS product, and it would be of interest to use these fields to estimate the SM uncertainty. However, the relationship between the CHI2P and the SM uncertainty is not straightforward and needs to be further investigated. One suggestion would be to define a statistical descriptor (e.g. mode, std...) for each CHI2P distribution, and to link them with specific validation scores. This task is of interest for users and needs more investigation to link the CHI2P (or DQX) to SMOS SM measurement performance.

<figure><img src="image_32.png" /></figure>

Figure 21: The link between the SMOS quality and the validation scores have to be investigated.
### 4.3.3 Investigation of the SMOS geophysical footprint surface conditions

In this part a score sensitivity study is done regarding the SMOS footprint surface conditions. On each DGG point, the MEANWEF was applied on the WA fraction content (see section 2.2.2).

<figure><img src="image_33.png" /><figcaption>Figure 22: Correlation per fraction of land cover in the SMOS footprint</figcaption></figure>

In the Figure 22, the results show that the correlation is worsening when the fraction of forest is increasing in the footprint (FNO/FFO). Presence of topography (FTS and FTM) and ice (FEI) is also degrading the correlation.
<figure><img src="image_34.png" /><figcaption>Figure 23: Bias per fraction of land cover in the SMOS footprint</figcaption></figure>

The relation between the bias and the FNO (resp. FFO) fraction decreases (resp. increases) slightly linearly for FNO < 50% (FFO > 50%). There is a linear trend from negative (<60%) to positive bias with sand, a negative bowl-shape of bias with clay and LAI. The presence of water leads to a strong dispersion of bias. However, there is a strong bias improvement with increasing bulk-density.
<figure><img src="image_35.png" /><figcaption>Figure 24: abs(Bias) per fraction of land cover in the SMOS footprint</figcaption></figure>

In the Figure 24, results show that the absolute bias is worsening when the fraction of forest is above 90 %. Presence of water (FWS an FWP) increases the statistical dispersion of the score. One can notice that the bulk density has a strong impact on the absolute bias improvement. There is no clear trend for the other fractions.
<figure><img src="image_36.png" /><figcaption>Figure 25: RMSE per fraction of land cover in the SMOS footprint</figcaption></figure>

In the Figure 25, results shows that, as for the correlation, the RMSE is worsening when the fraction of forest is increasing in the footprint (FNO/FFO). The presence of topography, ice and vegetation is also strongly degrading the RMSE. The RMSE is improving when the bulk density is increasing as observed for the bias.
<figure><img src="image_37.png" /><figcaption>Figure 26: ubRMSE per fraction of land cover in the SMOS footprint</figcaption></figure>

In the Figure 26, results shows that as for the correlation and the RMSE, the ubRMSE is worsens when the fraction of forest increases in the footprint (FNO/FFO). The presence of topography, ice and vegetation is also strongly worsening the ubRMSE. Bias and ubRMSE significantly improve with increasing bulk density.

<figure><img src="image_38.png" /></figure>

Validation scores improved when the SMOS footprint contains: less vegetation, no topography, no ice, no water, a high bulk density, less clay and more sand.
## 4.4 Score sensitivity to the number of samples

This part concerns one validation issue due to SMOS under-sampling (due to the revisit time) of the soil moisture dynamic as it can be monitored by the *in-situ*. In fact, in order to capture the full dynamic of the *in-situ* signal, a minimum of SMOS samples is needed to avoid too large Standard Error (SE) of the statistics estimates of our metrics. One exercise to illustrate this issue is to estimate their SE by bootstrap. It was done and is reported in Figure 27 where the *x-axis* represents the number of samples used for a validation. Let's consider a SMOS time series of *n* observations. We randomly select *m* SMOS samples (varying from 0 to *n*) and compute the scores. This is repeated 1000 times per *m* value, in order to represent the score variations due to the potential under-sampling. From Figure 27 it can be estimated that under ~500 samples, in this case, the scores have a strong variability leading to a potential small robustness score.

<figure><img src="image_39.png" /><figcaption>Figure 27: Effect of the number of samples</figcaption></figure>

<figure><img src="image_40.png" /></figure>

The first results show that if the number of sample used is too low, the confidence interval of the score increases significantly. Assuming the development of a potential filter for a minimum of sample in the validation chain.
# 5 Committed and recommended areas for SMOS validation

In this part, we investigate the question of SMOS accuracy, as per the Mission Requirements Document (MRD). Then, based on the results of Section 4, we use the relation between the surface description and the score performance to map the committed areas with its range of accuracy. Finally, using a more qualitative approach, recommendations of regions identified as the most relevant for validation are provided.

## 5.1 The committed areas in the SMOS MRD

Reported in the SMOS Mission Requirements Document (MRD) (Version 5) document, specifications regarding the SMOS accuracy are such as:

* "In summary the requirements for SM are: Soil moisture accuracy [0.04 m m (i.e. 4% volumetric soil moisture) or better]. For bare soils, for which the influence of w S on surface water fluxes is strong, Chanzy et al. (1995) have shown that a random error of 0.04 m 3 m -3 allows a good estimation of the evaporation and soil transfer parameters. Moreover this value corresponds to the typical rms dispersion of in situ w S observations."

* "After more than 20 years of research on the use of microwave radiometry for soil moisture sensing, the basic capabilities are well understood. Due to the large dielectric contrast between dry soil and water, the soil emissivity at a microwave frequency F depends upon moisture content. Over bare fields (Wang et al., 1983), is almost linearly related to the moisture content of a soil layer whose thickness depends upon F (3-5cm at 1.4 GHz, 1-2cm at 5 GHz). The vegetation cover attenuates soil emission and adds a contribution to the radiation temperature T B. However, at L-band, this attenuation is moderate; TB is sensitive to soil moisture for vegetated areas with biomass ≤ 5 kg.m⁻² (circa 65% of the Earth's land surface)."

From the MRD document, we consider that the ubRMSE is a candidate estimate for the SMOS SM accuracy; an ubRMSE ≤ 0.04 m³.m⁻³ is expected at L-Band considering a flat surface, free of snow ice and water bodies and with low vegetation. In the SMOS context, this accuracy is expected for footprints with a biomass ≤ 5 kg.m⁻² (AGB map [20]) and homogeneous land cover (FNO ≥ 95%).

* **VEGETATION:**
  Initially MRD is defined based on VWC. However, it is not measured at global scale. So for SMOS and SMAP, only a proxy is used. Initially, the VWC proxy was computed with LAI. Now that SMOS provides a VOD (Vegetation Optical Depth) that encompasses the biomass quantity and its VWC [34]. For our study we have three options to consider the MRD conditions (VWC) : i) use the LAI proxy ii) use the derived VOD iii) use the AGB. LAI and VOD are dynamic data, so it is difficult to set a threshold considering that they vary significantly during the seasons. AGB is a static estimate of dry biomass and recent studies ([35], [36]) showed its link to VOD. So, we first use an Above Ground Biomass (AGB) map, from [20] and highlight in green in Figure 28 the areas with an AGB ≥ 5 kg.m⁻². Future analyses will focus on temporal performances of SMOS SM and so will include the LAI and the VOD.

* **HOMOGENEITY**
  Concerning the homogeneity, the WA is assumed homogeneous if the fraction of NO is ≥ 95%. It represents a large part of the land surface as seen Figure 29.
<figure><img src="image_41.png" /><figcaption>Figure 28: In green the areas considered with a too high vegetation contribution $AGB \ge 5 \text{ kg.m}^{-2}$</figcaption></figure>

<figure><img src="image_42.png" /><figcaption>Figure 29: In blue the areas considered as homogeneous regarding the FNO fraction $\ge 95\%$</figcaption></figure>
From the two previous maps reported in Figure 28 and Figure 29, areas respecting the MRD surface conditions are represented in yellow in Figure 30, where the accuracy of SMOS SM is expected not to exceed 0.04 m³.m⁻³ regarding the MRD document. Those areas are considered as low vegetation and relatively homogeneous regarding the IGPB land classification and ESA AGB database. On this maps, ISMN probes are represented in blue when the probe is within the MRD area whereas it is in red when its outside. The spatial distribution of the probes with a depth limit of 0.1m, is 70% inside and 30% outside the MRD areas. In order to analyse the scores regarding probes inside/outside the areas, a classification is represented in Figure 31.

<figure><img src="image_43.png" /><figcaption>Figure 30: Maps of areas respecting the MRD surface conditions: homogeneity (FNO ≥ 95%) and low vegetation (AGB≤ 5 kg.m⁻²)</figcaption></figure>

In Figure 31, the box plot in blue shows better results than the red one. However, looking at the ubRMSE, even if the box plot shows better results than the red one, it is not always ≤ 0.04 m³.m⁻³.

<figure><img src="image_44.png" /></figure>

The scores of validation with probes inside/outside the MRD areas were analysed. These results show better scores in the MRD area, even though the results are not always fulfilling the MRD requirement of ubRMSE ≤ 0.04 m³.m⁻³.
<figure><img src="image_45.png" /><figcaption>Figure 31: Scores for the probes in (blue)/out (red) the surface conditions specified in the MRD</figcaption></figure>

## 5.2 Toward a new MRD surface conditions

In order to look from the MRD perspective, this part proposes to identify the conditions needed to reach the $ubRMSE \le 0.04 \ m^3.m^{-3}$ conditions. In the map Figure 32, probes reaching this condition were identified in red and their corresponding surface conditions are analysed Figure 33.

<figure><img src="image_46.png" /><figcaption>Figure 32: Maps where the probes are reaching the $0.04 \ m^3.m^{-3}$ (red circles)</figcaption></figure>

In the map Figure 32, the locations reaching the $0.04 \ m^3.m^{-3}$ are mainly located in Western US, West Africa and Australia. From these locations, the SMOS footprints were analysed by grouping scores. The results are presented in Figure 33. It is shown that globally all the points reaching the MRD of $ubRMSE \le 0.04 \ m^3.m^{-3}$ have more than 80% of low vegetation (FNO fraction), less than 20% of forest (FFO fraction), less than 15% of medium topography (FTM fraction), less than 22% of clay, more than 22% of sand, more than $1.3 \ g.cm^{-3}$ of soil bulk density and a LAI below 4 in the SMOS footprint.
<figure><img src="image_47.png" /><figcaption>Figure 33: Surface conditions of the probes reaching the ubRMSE: on left $\le 0.04 \text{ m}^3 \cdot \text{m}^{-3}$, on right $> 0.04 \text{ m}^3 \cdot \text{m}^{-3}$</figcaption></figure>

In order to represent the areas specified previously, a map respecting the surface condition thresholds was derived, in yellow, Figure 34. This map is quite similar to the map of surface condition regarding the MRD document in Figure 30. However wide areas in the Middle East and Central Asia were removed from this version.

<figure><img src="image_48.png" /></figure>

Validation points satisfying the $\le 0.04 \text{ m}^3 \cdot \text{m}^{-3}$ accuracy specification of the MRD were identified when all the following surface conditions hold: FNO $\ge 80\%$, FFO $\le 20\%$, TOPO $\le 15\%$, CLAY $\le 22\%$, SAND $\ge 22\%$, LAI $\le 4$ and BulkD $\ge 1.3 \text{ g.cm}^{-3}$. A map of areas respecting those thresholds was derived.
<figure><img src="image_49.png" /><figcaption>Figure 34: Map of areas respecting the surface identified as potential candidate for an ubRMSE ≤ 0.04 m³.m⁻³</figcaption></figure>

## 5.3 Maps of committed areas accuracy regarding the sensitivity study SMOS vs. ISMN

We provide here a first attempt to extrapolate the relations between the scores and the SMOS footprint content, represented in Figure 26, at global scale. Before defining a global relation, a fit between the score and each contributors were defined using a linear regression. The result of the slope and the intercept is summarized for each contributor equation 2. Those maps are displayed in Figure 35. Then, all the maps are averaged (equation 2) to derive a map of expected accuracy (top Figure 36), and the associated standard deviation (equation 3 and bottom Figure 36).

$$ CA_{ubRMSE} = mean(\underbrace{[-0.00052 \times FNO + 0.12]}_{a)low vegetation}; \underbrace{[0.00051 \times FFO + 0.070]}_{b)dense forest}; \underbrace{[0.0094 \times LAI + 0.056]}_{c)LAI}) \\ \underbrace{; [0.00033 \times FTM + 0.079]}_{d)low topography}; \underbrace{[0.0016 \times FTS + 0.080]}_{e)strong topography}; \underbrace{[0.00070 \times CLAY + 0.070]}_{f)clay content}; \\ \underbrace{; [0.0026 \times FWP + 0.078]}_{g)pure water}; \underbrace{[-0.000063 \times SAND + 0.085]}_{h)sand content}; \underbrace{[-0.083 \times BulkDensity + 0.19]}_{i)bulk density}) \quad (2) $$
and the associated standard deviation:

$$
\begin{equation}
\begin{aligned}
\text{std}(CA_{ubRMSE}) = & \text{std}([\underbrace{-0.00052 \times FNO + 0.12}_{\text{a)low vegetation}}]; [\underbrace{0.00051 \times FFO + 0.070}_{\text{b)dense forest}}]; [\underbrace{0.0094 \times LAI + 0.056}_{\text{c)LAI}]) \\
& \quad ; [\underbrace{0.00033 \times FTM + 0.079}_{\text{d)low topography}}]; [\underbrace{0.0016 \times FTS + 0.080}_{\text{e)strong topography}}]; [\underbrace{0.00070 \times CLAY + 0.070}_{\text{f)clay content}}] \\
& \quad ; [\underbrace{0.0026 \times FWP + 0.078}_{\text{g)pure water}}]; [\underbrace{-0.000063 \times SAND + 0.085}_{\text{h)sand content}}]; [\underbrace{-0.083 \times BulkDensity + 0.19}_{\text{i)bulk density}}]
\end{aligned}
\tag{3}
\end{equation}
$$

On the global maps Figure 36, areas such as the US West coast, Australia's center, or South Africa, show an expected accuracy of 0.07-0.08 m³.m⁻³ with a standard deviation of ~0.01 m³.m⁻³. One can note patterns related to topography, soil texture and vegetation, where the accuracy worsens.

**Caution:** More complex statistical approaches, such as neural network or principal component analysis, are needed to improve those maps. In fact, there is a certain number of over-representation (i.e. FNO and FFO), non-independence, uncertainties and non linearity which are not taken into account with a simple linear regression.

<figure><img src="image_50.png" /></figure>

Regression fits are used to define relationships between scores and SMOS footprint content. A global map is derived from the average of those relation and another with the standard deviation, providing a range of uncertainties. However the method has still to be improved with a deeper investigation to assess areas with a minimization of scale representativeness between SMOS and the in-situ.
<figure><img src="image_51.png" /><figcaption>Figure 35: Global maps of expected ubRMSE from the different SMOS footprint surface conditions from the auxiliary data. Each maps correspond to a part of the equation 2.</figcaption></figure>
<figure><img src="image_52.png" /><figcaption>Figure 36: Global committed areas maps, on top the average of the maps Figure 35 and on the bottom the standard deviation.</figcaption></figure>
## 5.4 Maps of recommended areas for SMOS validation

Another suggestion is to define an index, from 0 to 1, representing the level of confidence one may have for validation, index based upon the result of the sensitivity study between the ubRMSE and the SMOS footprint content, Figure 26. This index represents the level of confidence expected for the validation. This index is built on the average of all the normalized SMOS footprint content in order to include all the parameters in a qualitative index:

$$GEOIDX = \frac{[LAI]_0^1 + [LowVeg.]_1^0 + [For.]_0^1 + [Topo.]_0^1 + [Sand]_1^0 + [Clay]_0^1 + [BulkD.]_1^0 + [Water]_0^1}{8} \quad (4)$$

where for a given descriptor, desc, $[desc]_a^b$ is the linear regression of desc values from $[min desc]_a^b$ to $[a b]$. e.g. $[LAI]_0^1$ is going toward 0 by minimizing LAI values while, $[Sand]_1^0$ is going to toward 0 by maximizing Sand values.

On the map Figure 37, the index in green is close to zero, meaning that the conditions are favorable with a small amount of vegetation (LAI), topography (Topo), clay (Clay), open water (Water) and forest fraction (For), and a high amount of sand (Sand), bulk density (BulkD) and fraction of low vegetation (LowVeg) in the SMOS footprint. Then the color will go to yellow to moderate and red for non recommended areas for validation due to the surface conditions. The blue dots are areas highly contaminated by RFI (at the date of this document and corresponding to the RFI average of 2020 over 0.1).

<figure><img src="image_53.png" /><figcaption>Figure 37: Map of recommended areas for validation, the blue dots correspond to an high risk of RFI contamination (> 0.1 on 2020 annual average).</figcaption></figure>
# 6 Conclusion, limitations, and perspective of improvements

The SMOS MRD specifies the expected accuracy of SMOS SM (ubRMSE ≤ 0.04 m³.m⁻³) within specific surface conditions (low-vegetation and homogeneous areas). The main objective of the present study is to take advantage of the SMOS long time series observations to verify and characterize the SMOS SM accuracy. To do so, the three following items were studied:

## 1. Assess the SMOS uncertainty using the ISMN as reference

First of all, a validation chain is defined (masking step, with the different filters, and the spatio-temporal collocation). Then, a part concerning the different sources of uncertainty and the metrics are presented and discussed. This validation chain is finally used to evaluate SMOS on the whole ISMN network. The results show a reasonable agreement between SMOS and the in-situ data (R=0.462, ubRMSE=0.087 m³.m⁻³ and bias=-0.069 m³.m⁻³), which confirms the method consistency. However, these performances were computed assuming all probes were perfectly accurate and precise (probe depth, technology, or location). The analysis of the score through the probe-depth point of view shows a performance worsens as depth increases, which is to be expected. Considering only the probes within the first 10 cm of the soil, the results are improved (R=0.562, ubRMSE=0.082 m³.m⁻³ and bias=-0.051 m³.m⁻³). As a consequence, the rest of the analysis only considers the probes within the first 10 cm (~45% of the whole probes data set).

## 2. Evaluate the SMOS accuracy specification of 0.04 m³.m⁻³

In order to evaluate the expected accuracy of 0.04 m³.m⁻³ in specific surface conditions, maps were derived considering: 1) low vegetation AGB ≤ 5 kg.m⁻² and 2) homogeneity FNO ≥ 95% as prescribed by the SMOS MRD. Then the probes were classified as providing results within or outside the MRD domain of validity. The results show globally better scores for the probes inside the "MRD validity range". However, as the accuracy of ubRMSE ≤ 0.04 m³.m⁻³ is not reached everywhere, surface conditions were updated using the specific probes reaching this accuracy threshold: FNO ≥ 80%, FFO ≤ 20%, TOPO ≤ 15%, CLAY ≤ 22%, SAND ≥ 22%, LAI ≤ 4 and BulkDensity ≥ 1.3 g.cm⁻³.

## 3. Define committed areas as geophysical surface conditions with expected uncertainty

A fit function linking the accuracy with the land cover and soil properties of the SMOS footprint was established. From these relationships, maps of expected uncertainties were derived. Globally, the scores show a performance improvement with a minimization of forest, topography, water, and ice in the footprint. Concerning the soil parameters, the scores improved when the footprint was sandier, had less clay, and had a high bulk-density.

This first study shed light on a list of limitations and perspectives of improvement that we need to develop further:

* **Dependence on geophysical database uncertainties**

The quality of the results presented here depends on the auxiliary data accuracy, metadata... For example, we plan to use the SoilGrid database instead of HWSD which is expected to give a better representation of the soil parameters.

* **Dependence on SMOS uncertainties**

This study was based on the SMOS data, the results could be different with another sensor. However, this strategy can be applied to other satellite missions such as SMAP or AMSR-E.

* **Dependence on ISMN uncertainties**

The global SMOS accuracy computed in the present study is inferred by the in-situ uncertainty. It means that when SMOS is not reaching the 0.04 m³.m⁻³ threshold, one should keep in mind that it may be due to a high in-situ uncertainty.
* **Including a Neural Network approach to assess the spatio-temporal scale difference between SMOS and *in-situ***
The work done in [37] should be extended.

* **Recommendation on the CHI2 and DQX quality index**

The link between the SMOS CHI2 or DQX (in the SML2 product) and the soil moisture uncertainties should be assessed to better filter SMOS data for further applications.

* **In-situ spatial sampling distribution issue (ISMN)**

Due to the spatial distribution of the ISMN network, only some surface conditions are represented, some of them may be over-represented and some of them missing. One solution is to analyse the land-cover class for each probes and to compare them with the global land-cover classification (ESA CCI Land-cover).

* **Assessing a number of sample needed for a fair validation**

We plan to evaluate the number of SMOS-*in-situ* sample to compute significant metrics.

* **Improving the statistical method used to derive committed areas maps**

The simple regression fit which links the SMOS footprint content characteristics and the accuracy range has to be improved. We plan to first apply a principal component analysis.

* **Spatial sampling strategy: *in-situ* driven or satellite driven**

During this first evaluation we used an *in-situ* driven strategy. An upscaling method will be developed (Voronoi triangle, Thiessen polygons...).

* **Spatiotemporal scale mismatch investigation**

We plan to study the SMOS footprint heterogeneity.

* **Probe technology and calibration law dependence**

Probe representativeness is dependent on the technology used. Decomposing the sensitivity of the probes technology, the land cover, and the climate classes is needed to clearly analyse the signal.

* **Potential snow cover risk**

We plan to include a dynamic snow mask.

* **In-situ quality flag**

The QCflag are still in development by TUWien, so this analysis is postponed until ISMN fixes the flag content.

* **Robust statistics**

In addition to the usual moment-based statistics estimates we have used, robust statistics (non parametric, ranked) versions will be considered. Indeed, we have often to deal with outliers, small collocated sample size, non-normal distributions, non-comparable distributions, that can significantly impact the usual moment-based statistics and their interpretations. Considering rather median-, MAD-, quantile-based versions, or rank Spearman, Kendall correlation... would help to have a better view of the overall picture by minimizing the impact of the marginals, the violation to implicit statistical assumptions. Moreover, it would facilitate the identification of marginals or distribution change that may be representative or linked to effects of interest e.g. scales.
References

[1] Bellevue College. B. accuracy vs. precision, and error vs. uncertainty. 2014.

[2] Yann H Kerr, Philippe Waldteufel, J-P Wigneron, JAMJ Martinuzzi, Jordi Font, and Michael Berger. Soil moisture retrieval from space: The soil moisture and ocean salinity (smos) mission. *IEEE transactions on Geoscience and remote sensing*, 39(8):1729-1735, 2001.

[3] Alexander Loew, William Bell, Luca Brocca, Claire E Bulgin, Jörg Burdanowitz, Xavier Calbet, Reik V Donner, Darren Ghent, Alexander Gruber, Thomas Kaminski, et al. Validation practices for satellite based earth observation data across communities. *Reviews of Geophysics*, 2017.

[4] W Dorigo, W Wagner, R Hohensinn, S Hahn, C Paulik, A Xaver, A Gruber, M Drush, S Mecklenburg, P van Oevelen, A Robock, and T Jackson. The International Soil Moisture Network: a data hosting facility for global in situ soil moisture measurements. *Hydrology and Earth System Sciences*, May 2011.

[5] Sonia I. Seneviratne, Thierry Corti, Edouard L. Davin, Martin Hirschi, Eric B. Jaeger, Irene Lehner, Boris Orlowsky, and Adriaan J. Teuling. Investigating soil moisture-climate interactions in a changing climate: A review. *Earth-Science Reviews*, 99(3-4):125-161, May 2010.

[6] J-P Wigneron, Laurent Laguerre, and Yann H Kerr. A simple parameterization of the l-band microwave emission from rough agricultural soils. *IEEE Transactions on Geoscience and Remote Sensing*, 39(8):1697-1707, 2001.

[7] Y. H. Kerr, P. Waldteufel, P. Richaume, J. P. Wigneron, P. Ferrazzoli, A. Mahmoodi, A. Al Bitar, F. Cabot, C. Gruhier, S. E. Juglea, D. Leroux, A. Mialon, and S. Delwart. The SMOS Soil Moisture Retrieval Algorithm. *Geoscience and Remote Sensing, IEEE Transactions on*, 50(5):1384-1403, 2012.

[8] J-P Wigneron, Y H Kerr, P Waldteufel, K Saleh, M-J Escorihuela, P Richaume, P Ferrazzoli, P de Rosnay, R Gurney, J-C Calvet, J P Grant, M Guglielmetti, B Hornbuckle, C Mätzler, T Pellarin, and M Schwank. L-band Microwave Emission of the Biosphere (L-MEB) Model: Description and calibration against experimental data sets over crop fields. *Rem. Sens. Environ.*, 107:639-655, 2007.

[9] M. Talone, M. Portabella, J. Martínez, and V. González-Gambau. About the optimal grid for smos level 1c and level 2 products. *IEEE Geoscience and Remote Sensing Letters*, 12(8):1630-1634, 2015.

[10] Y.H. Kerr, P. Richaume, P. Waldteufel, P. Ferrazzoli, J.P. Wigneron, M. Schwank, and K. Rautiainen. Algorithm theoretical basis document (ATBD) for the smos level 2 soil moisture processor. *Technical Report TN-ESL-SM-GS-0001-4b SM-ESL (CBSA)*, page 145p, 2020.

[11] Roger Oliva, Elena Daganzo, Yann H Kerr, Susanne Mecklenburg, Sara Nieto, Philippe Richaume, and Claire Gruhier. Smos radio frequency interference scenario: Status and actions taken to improve the rfi environment in the 1400-1427-mhz passive band. *IEEE Transactions on Geoscience and Remote Sensing*, 50(5):1427-1439, 2012.

[12] Elena Daganzo-Eusebio, Roger Oliva, Yann H Kerr, Sara Nieto, Philippe Richaume, and Susanne Martha Mecklenburg. Smos radiometer in the 1400-1427-mhz passive band: Impact of the rfi environment and approach to its mitigation and cancellation. *IEEE Transactions on Geoscience and Remote Sensing*, 51(10):4999-5007, 2013.
[13] Philippe Richaume, Yan Soldo, Eric Anterrieu, Ali Khazaal, Simone Bircher, Arnaud Mialon, Ahmad Al Bitar, Nemesio Rodriguez-Fernandez, François Cabot, Yann Kerr, and Ali Mahmoodi. Rfi in smos measurements: Update on detection, localization, mitigation techniques and preliminary quantified impacts on soil moisture products. In *2014 IEEE Geoscience and Remote Sensing Symposium*, pages 223–226, 2014.

[14] M. Martín-Neira, R. Oliva, I. Corbella, F. Torres, N. Duffo, I. Durán, J. Kainulainen, J. Closa, A. Zurita, F. Cabot, A. Khazaal, E. Anterrieu, J. Barbosa, G. Lopes, J. Tenerelli, R. Déz-García, J. Fauste, F. Martín-Porqueras, V. González-Gambau, A. Turiel, S. Delwart, R. Crapolicchio, and M. Suess. Smos instrument performance and calibration after six years in orbit. *Remote Sensing of Environment*, 180:19–39, 2016. Special Issue: ESA’s Soil Moisture and Ocean Salinity Mission - Achievements and Applications.

[15] Fischer G. Global agro-ecological zones assessment for agriculture (gaez 2008). iiasa, laxenburg, austria and fao, rome, italy. 2008.

[16] L. Poggio, L. M. de Sousa, N. H. Batjes, G. B. M. Heuvelink, B. Kempen, E. Ribeiro, and D. Rossiter. Soilgrids 2.0: producing soil information for the globe with quantified spatial uncertainty. *SOIL*, 7(1):217–240, 2021.

[17] Thomas Loveland, Zhiliang Zhu, D. Ohlen, Jesslyn Brown, B. Redd, and Limin Yang. An analysis of igbp global land-cover characterization process. *Photogrammetric Engineering and Remote Sensing*, 65:1069–1074, 09 1999.

[18] ESA. Land cover cci product user guide version 2. tech. rep. (2017). available at: maps.elie.ucl.ac.be/cci/viewer/download/esacci-lc-ph2-pugv2-2.0.pdf.

[19] Y. Knyazikhin R. Myneni and T. Park. Myd15a2h modis/aqua leaf area index/fpar 8-day 14 global 500m sin grid v006. nasa eosdis land processes daac., 2015.

[20] Maurizio Santoro. Globbiomass - global datasets of forest biomass, 2018.

[21] Markus Kottek, Jürgen Grieser, Christoph Beck, Bruno Rudolf, and Franz Rubel. World map of the köppen-geiger climate classification updated. *Meteorologische Zeitschrift*, 15(3):259–263, July 2006.

[22] Copernicus Climate Change Service. Era5-land monthly averaged data from 2001 to present, 2019.

[23] Alexander Gruber, Gabrielle De Lannoy, Clement Albergel, Amen Al-Yaari, Luca Brocca, J-C Calvet, Andreas Colliander, Michael Cosh, Wade Crow, Wouter Dorigo, et al. Validation practices for satellite soil moisture retrievals: What are (the) errors? *Remote sensing of environment*, 244:111806, 2020.

[24] R. Alexander. A note on averaging correlations. *Bulletin of the Psychonomic Society volume*, 1990.

[25] Thomas J. DiCiccio and Bradley Efron. Bootstrap confidence intervals. 11(3), sep 1996.

[26] Thomas J. Schmugge. Remote sensing of soil moisture: Recent advances. *IEEE Transactions on Geoscience and Remote Sensing*, GE-21(3):336–344, 1983.

[27] Eni G Njoku and Dara Entekhabi. Passive microwave remote sensing of soil moisture. *Journal of hydrology*, 184(1-2):101–129, 1996.
[28] M.J. Escorihuela, A. Chanzy, J.P. Wigneron, and Y. H. Kerr. Effective soil moisture sampling depth of l-band radiometry: A case study. *Remote Sensing of Environment*, 114(5):995–1001, 2010.

[29] Marion Leduc-Leballeur, Ghislain Picard, Arnaud Mialon, Laurent Arnaud, Eric Lefebvre, Philippe Possenti, and Yann Kerr. Modeling l-band brightness temperature at dome c in antarctica and comparison with smos observations. *IEEE Transactions on Geoscience and Remote Sensing*, 53(7):4022–4032, 2015.

[30] Shaoning Lv, Yijian Zeng, Jun Wen, Hong Zhao, and Zhongbo Su. Estimation of penetration depth from soil effective temperature in microwave radiometry. *Remote Sensing*, 10(4), 2018.

[31] Xiaoji Shen, Jeffrey P. Walker, Nan Ye, Xiaoling Wu, Nithyapriya Boopathi, In-Young Yeo, Linlin Zhang, and Liujun Zhu. Soil moisture retrieval depth of p- and l-band radiometry: Predictions and observations. *IEEE Transactions on Geoscience and Remote Sensing*, 59(8):6814–6822, August 2021.

[32] F. T. Ulaby. *Microwave Remote Sensing: Active and Passive, Radar Remote Sensing and Surface Scattering and Emission Theory*, volume 2. Addison-Wesley, Reading, Mass, 1982.

[33] Limin Yu, Wanlin Gao, Redmond R. Shamshiri, Sha Tao, Yanzhao Ren, Yanjun Zhang, Guilian Su, and and. Review of research progress on soil moisture sensor technology. *International Journal of Agricultural and Biological Engineering*, 14(3):32–42, 2021.

[34] A. G. Konings, N. M. Holtzman, K. Rao, L. Xu, and S. S. Saatchi. Interannual variations of vegetation optical depth are due to both water stress and biomass changes. *Geophysical Research Letters*, 48(16):e2021GL095267, 2021.

[35] F. Tian, J-P. Wigneron, P. Ciais, J. Chave, J. Ogée, J. Peñuelas, A. Raebild, J-C. Domec, X. Tong, M. Brandt, A. Mialon, N. Rodríguez-Fernández, T. Tagesson, A. Al-Yaari, Y. Kerr, C Chen, R.B. Myneni, W. Zhang, J Ardö, and R. Fensholt. Coupling of ecosystem-scale plant water storage and leaf phenology observed by satellite. *Nature Ecology and Evolution*, 2:1428–1435, 2018.

[36] N. J. Rodríguez-Fernández, A. Mialon, S. Mermoz, A. Bouvet, P. Richaume, A. Al Bitar, A. Al-Yaari, M. Brandt, T. Kaminski, T. Le Toan, Y. H. Kerr, and J.-P. Wigneron. An evaluation of smos l-band vegetation optical depth (l-vod) data sets: high sensitivity of l-vod to above-ground biomass in Africa. *Biogeosciences*, 15(14):4627–4645, 2018.

[37] Alireza Mahmoodi, Nemesio J. Rodriguez-Fernandez, Philippe Richaume, and Yann H. Kerr. Global estimation of surface soil moisture using neural networks trained by in-situ measurements and passive l-band telemetry. In *2021 IEEE International Geoscience and Remote Sensing Symposium IGARSS*. IEEE, July 2021.

**Appendices: Gibon et al. 2022 in prep.**

*Gibon et al. 2022*: an article based on the results of this study is in preparation in Remote Sensing for Environment.