# Diffuse reflectance spectroscopy for estimating soil properties: A technology for the 21st century

Raphael Viscarra Rossel, Thorsten Behrens, Eyal Ben-Dor, Sabine Chabrillat, José Alexandre Melo Demattê, Yufeng Ge, Cecile Gomez, César Guerrero, Yi Peng, Leonardo Ramirez-Lopez, et al.

► To cite this version:

Raphael Viscarra Rossel, Thorsten Behrens, Eyal Ben-Dor, Sabine Chabrillat, José Alexandre Melo Demattê, et al. Diffuse reflectance spectroscopy for estimating soil properties: A technology for the 21st century. European Journal of Soil Science, 2022, 73 (4), 10.1111/ejss.13271. hal-03738798

HAL Id: hal-03738798

https://hal.inrae.fr/hal-03738798

Submitted on 26 Jul 2022

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Diffuse reflectance spectroscopy for estimating soil properties: A technology for the 21st century

Raphael A. Viscarra Rossel<fnref n="1" /> ID | Thorsten Behrens<fnref n="2" /> | Eyal Ben-Dor<fnref n="3" /> |

Sabine Chabrillat<fnref n="4" />,<fnref n="5" /> | José Alexandre Melo Demattê<fnref n="6" /> ID | Yufeng Ge<fnref n="7" /> |

Cecile Gomez<fnref n="8" />,<fnref n="9" /> ID | César Guerrero<fnref n="10" /> | Yi Peng<fnref n="11" /> | Leonardo Ramirez-Lopez<fnref n="12" /> |

Zhou Shi¹³ | Bo Stenberg¹⁴ | Richard Webster¹⁵ | Leigh Winowiecki¹⁶ |

Zefang Shen<fnref n="1" />

## Abstract

Spectroscopic measurements of soil samples are reliable because they are highly repeatable and reproducible. They characterise the samples' mineral-organic composition. Estimates of concentrations of soil constituents are inevitably less precise than estimates obtained conventionally by chemical analysis. But the cost of each spectroscopic estimate is at most one-tenth of the cost of a chemical determination. Spectroscopy is cost-effective when we need many data, despite the costs and errors of calibration. Soil spectroscopists understand the risks of over-fitting models to highly dimensional multivariate spectra and have command of the mathematical and statistical methods to avoid them. Machine learning has fast become an algorithmic alternative to statistical analysis for estimating concentrations of soil constituents from reflectance spectra. As with any modelling, we need judicious implementation of machine learning as it also carries the risk of over-fitting predictions to irrelevant elements of the spectra. To use the methods confidently, we need to validate the outcomes with appropriately sampled, independent data sets. Not all machine learning should be considered 'black boxes'. Their interpretability depends on the algorithm, and some are highly interpretable and explainable. Some are difficult to interpret because of complex transformations or their huge and complicated network of parameters. But there is rapidly advancing research on explainable machine learning, and these methods are finding applications in soil science and spectroscopy. In many parts of the world, soil and environmental scientists recognise the merits of soil spectroscopy. They are building spectral libraries on which they can draw to localise the modelling and derive soil information for new projects within their domains. We hope our article gives readers a more balanced and optimistic perspective of soil spectroscopy and its future.

Comment on "Estimating soil chemical properties by diffuse reflectance spectroscopy: Promise versus reality"

This is an open access article under the terms of the [Creative Commons Attribution License](https://creativecommons.org/licenses/by/4.0/), which permits use, distribution and reproduction in any medium, provided the original work is properly cited.

© 2022 The Authors. *European Journal of Soil Science* published by John Wiley & Sons Ltd on behalf of British Society of Soil Science.

<footnote n="1">¹Soil and Landscape Science, School of Molecular and Life Sciences, Faculty of Science and Engineering, Curtin University, Perth, Australia</footnote>

<footnote n="2">²Data Science, Swiss Competence Center for Soils (KOBO), Zollikofen, Switzerland</footnote>

<footnote n="3">³The Remote Sensing Laboratory, Tel Aviv University, Tel Aviv, Israel</footnote>

<footnote n="4">⁴Remote Sensing, Helmholtz Center Potsdam GFZ German Research Centre for Geosciences, Potsdam, Germany</footnote>

<footnote n="5">⁵Leibniz University Hannover, Institute of soil science, Hannover, Germany</footnote>

<footnote n="6">⁶Department of Soil Science, Luiz de Queiroz College of Agriculture, University of São Paulo, Piracicaba, Brazil</footnote>

<footnote n="7">⁷Department of Biological Systems Engineering, University of Nebraska-Lincoln, Lincoln, Nebraska, USA</footnote>

<footnote n="8">⁸LISAH, University of Montpellier, IRD, INRAE, Institut Agro, Montpellier, France</footnote>

<footnote n="9">⁹Indo-French Cell for Water Sciences, IRD, Indian Institute of Science, Bangalore, India</footnote>

<footnote n="10">¹⁰Department of Agrochemistry and Environment, Universidad Miguel Hernández de Elche, Elche, Spain</footnote>

<footnote n="11">¹¹Food and Agriculture Organization of the United Nations, Viale delle Terme di Caracalla, Rome, Italy</footnote>

<footnote n="12">¹²Data Science Department, BUCHI Labortechnik AG, Flawil, Switzerland</footnote>
¹³Institute of Agricultural Remote Sensing and Information Technology Application, College of Environmental and Resource Sciences, Zhejiang University, Hangzhou, China

¹⁴Department of Soil and Environment, Swedish University of Agricultural Sciences, Skara, Sweden

¹⁵Rothamsted Research, Harpenden, UK

¹⁶World Agroforestry Centre (ICRAF), Nairobi, Kenya

**Correspondence**

Raphael A. Viscarra Rossel, Soil and Landscape Science, School of Molecular and Life Sciences, Faculty of Science and Engineering, Curtin University, G.P.O. Box U1987, Perth, WA 6845, Australia.

Email: r.viscarra-rossel@curtin.edu.au

**Funding information**

Australian Government, Grant/Award Number: ACSRIV000077

**Highlights**

* Spectroscopy is reliable because it is a highly repeatable and reproducible analytical technique.

* Spectra are calibrated to estimate concentrations of soil properties with known error.

* Spectroscopy is cost-effective for estimating soil properties.

* Machine learning is becoming ever more powerful for extracting accurate information from spectra, and methods for interpreting the models exist.

* Large libraries of soil spectra provide information that can be used locally to aid estimates from new samples.

**KEYWORDS**

calibration, machine learning, model localization, reflectance spectroscopy, regression, soil constituents, spectral libraries, validation

# 1 | INTRODUCTION

The recent Opinion article in this journal by McBride (2022) reviews the science underlying diffuse reflectance spectroscopy for determining concentrations of chemical constituents in soil. In the article, McBride criticises the techniques and the exaggerated claims made by some of its exponents. Some of the comments made by McBride are fair. The reflectance spectra of soil in the visible (vis; 400–700 nm), near-infrared (NIR; 700–2500 nm), and mid-infrared (MIR; 2500–20,000 nm) are influenced simultaneously by many soil components, each absorbing electromagnetic radiation at characteristic frequencies. Soil spectra are more complex than those of many other materials, making it more difficult to analyse and draw inferences from them. We should not readily expect to obtain accurate estimates of constituents that lack spectral response, which according to McBride, many practitioners seem to assume. Regression, whether based on ordinary least squares, more elaborate modelling or machine learning, is unlikely to provide reliable predictions of target variables that are only weakly correlated with the spectra. All modelling can be risky and suffer from over-fitting unless appropriately used and validated. Spectroscopy in these circumstances is no panacea; it is no certain substitute for well-established soil chemical, physical or biological analysis. We agree with McBride on these matters. With only a superficial review of the literature, we might find published studies that would have readers believe otherwise.

There are, however, important aspects of McBride's paper with which we disagree. The article is published as an 'Opinion', and we contend that this opinion is based on a partial and somewhat outdated appreciation of science and rapidly advancing technologies. We also contend that McBride has failed to appreciate the circumstances in which reflectance spectroscopy has merit. We elaborate on these aspects below.

# 2 | SOIL SPECTROSCOPY CAN BE 'TRUSTED'

McBride (2022) states that the spectroscopic method cannot be 'trusted' because of the 'indirect' relationship between the soil properties and the spectra. Using such a broad statement to describe soil spectroscopy is wrong. Diffuse reflectance spectroscopy is a well-established quantitative method long used in physical and analytical chemistry because atoms and molecules absorb radiation in specific wavelengths and have their own unique spectra (McClure, 2003; Pasquini, 2018; Workman Jr, 1996). Soil spectroscopy in the vis-NIR and MIR integrates the signals from the soil's minerals, organic matter, and water adsorbed or present in mineral structures (Clark et al., 1990; Nguyen et al., 1991; Viscarra Rossel & Hicks, 2015). The spectra provide a 'fingerprint' of the molecular composition of the soil matrix. Therefore, over the past four decades, research has shown that when soil's physical, chemical, and biological properties derive
from or are associated with the mineral-organic matrix, the spectra of air-dry soil can respond to variation in those properties. If the spectroscopic model accounts for the minerals and organic signals in the spectra as well as their interactions, then the method can reasonably accurately estimate the concentration of other constituents (Ben-Dor & Banin, 1995; Soriano-Disla et al., 2014; Stenberg et al., 2010; Viscarra Rossel et al., 2006).

Not all constituents are properties of the soil matrix, and those unrelated to it will contribute to the spectra only fortuitously. Any correlation to the spectra will be transient and, although perhaps locally significant, would not generally apply (e.g., Wetterlind et al., 2008).

Spectra obtained from soil under field conditions can contain substantial contributions from water. These are fairly linearly related to the water content (Bowers & Hanks, 1965; Lobell & Asner, 2002), providing an opportunity to develop valid spectroscopic calibrations for soil water (Baumann et al., 2022). If water is taken into account (Ji et al., 2015; Minasny et al., 2011; Wijewardane et al., 2016), field spectra can also be calibrated to estimate soil constituents (e.g., Li et al., 2015; Viscarra Rossel et al., 2017). But this research is ongoing, so we leave a discussion of it for now.

### 3 | SPECTROSCOPY COMPLEMENTS CONVENTIONAL ANALYSIS

McBride (2022) writes that in many publications on soil spectroscopy, while describing the method's benefits, proponents of spectroscopy have implied that their ultimate goal is to replace conventional soil testing. We have seen the papers and can imagine that the authors, in their enthusiasm, have exaggerated the benefits of soil spectroscopy. Clearly, such claims are wrong because estimates of soil properties rely on empirical calibrations of the spectra with the properties measured conventionally. Nevertheless, we should consider those comments in their written context.

The question of whether spectroscopy could replace conventional analysis was posed some time ago by Janik et al. (1998) who asked if MIR spectroscopy could replace soil extractions. They answered that, for the most part, spectroscopy could not, but stated that spectroscopy adds value to conventional methods and extends them because spectroscopy also helps to understand soil chemistry better. Proponents of spectroscopy understand this and treat it not to replace conventional analysis but to complement it. We need both to increase the data we can assemble to satisfy the growing demand for quantitative information on the soil to support new science and help meet the world's needs for food, fibre, climate adaptation,

environmental quality, and sustainable development (Viscarra Rossel & Bouma, 2016).

McBride repeatedly opines that ‘... spectral reflectance methods are not sufficiently reliable to replace conventional testing’. As above, the aim is not to replace conventional testing. Nonetheless, the statement is misleading; diffuse reflectance spectroscopy is ‘reliable’ in the sense that repeated measurements are consistently reproducible on any given sample of soil from one occasion to another. We know from a great deal of experience that repeated spectroscopic measurements of a soil sample produce spectra with almost identical mineral-organic absorbances at the same wavelengths (Stenberg et al., 2010), provided, of course, that the soil sample does not change. Perhaps McBride does not mean that spectroscopy is unreliable but that estimates of soil properties are inaccurate because of uncertainties in the spectroscopic modelling. We address this point below.

### 4 | THE EXPECTED ACCURACY OF SOIL SPECTROSCOPY

One of the main criticisms made by McBride (2022) is that the proponents of spectroscopy have not shown it to be as accurate as conventional methods. Of course, they have not. The reason is that the conventional methods for soil analysis are the de facto standards. Any other techniques that aim to produce the same results are inevitably less accurate because they rely on the calibration of the spectra against the standards and incur statistical error. When developing the calibrations, we assume that the analytical errors of the standard methods are small enough to be negligible. In some circumstances they can be substantial, however (O'Rourke & Holden, 2011; van Leeuwen et al., 2022; Viscarra Rossel & Bouma, 2016), and then those errors will carry over into the calibrations and reduce the accuracy of the spectroscopic estimates.

It also does not make sense to compare a spectroscopic estimate with a standard analytical technique on a one-to-one basis. The advantage of soil spectroscopy (and incidentally of other sensing methods, whether proximal or more remote) is that one can make many more measurements, at least an order of magnitude more than by the conventional laboratory analysis, for the same cost. So, the estimation variance of a spectroscopic estimate of the average of some soil properties will be smaller than from conventional analysis. Thus, when the dominant source of error in the individual estimates is random rather than systematic (Guerrero & Lorenzetti, 2021), and when the costs and errors of calibration are taken into account, spectroscopy is cost-effective (Li et al., 2022). Moreover, because its estimation variance is smaller than that of means of fewer replicate measurements obtained by the
more expensive standard analytical technique, the estimates
are also more informative. The advantages become even
more significant when one needs hundreds or thousands of
data, for example, for assessments of soil spatial and tempo-
ral variation, for mapping (e.g., Ramirez-Lopez et al., 2019;
Vågen et al., 2016; Viscarra Rossel et al., 2015), for model-
ling (e.g., Lee et al., 2021; Lee & Viscarra Rossel, 2020), and
in decision-support systems (e.g., Vågen et al., 2018). For
those reasons, the Australian Government Emission Reduc-
tion Fund (ERF) method for measuring and monitoring soil
carbon sequestration¹ is the first to include spectroscopy as
a way to improve the accuracy and reduce the cost of mea-
suring and monitoring (England & Viscarra Rossel, 2018;
Viscarra Rossel, Lobsey, et al., 2017; Viscarra Rossel &
Brus, 2018; Viscarra Rossel, Brus, et al., 2016).

A notable exception to much of the discussion above is
when the measurement of the soil property is made more
directly from the characteristic absorptions of the soil con-
stituent, for example, measures of soil colour or iron-oxide
and clay mineralogy (Clark, 1999; Farmer, 1974; Viscarra
Rossel, 2011; Viscarra Rossel et al., 2010). In such cases, the
spectroscopic measurements might be at least as accurate
as the conventional analysis (e.g., Janik et al., 1995).

5 | CALIBRATION, REGRESSION AND DIMENSIONALITY REDUCTION

Apart from the few exceptions mentioned, spectroscopic
estimates of soil constituents depend on the strength of the
relation between the target variables, such as the concentra-
tion of organic carbon and the absorbances in the various
wavelengths. The spectra produced by modern spectrome-
ters contain thousands or more wavelengths. Therefore, the
first task is a multivariate calibration, that is, obtaining an
equation from which to predict the concentration of the tar-
get constituent from the spectra. Some transformation of
the measurement scales is usually required, and pre-
processing to linearize the data and remove noise. One
might think that, at its simplest, that equation might mean
an ordinary least-squares multiple regression. However, we
know that with so many predictor variables, multiple
regression will suffer from a lack of selectivity of the predic-
tors to the target and redundancy and collinearity among
the predictors (Martens & Næs, 1989).

McBride (2022) recognises the problems; they quote
Bergstrom & West (2020), who write about the ‘curse of
dimensionality’, first proposed by (Bellman, 1957). How-
ever, McBride dismisses the methods for reducing the
number of dimensions used to overcome the problem.
One popular way is to do a principal components analysis
(PCA) of the spectra and then regress the target variable

on some of the leading components. The method is called
Principal components regression (PCR) (Hotelling, 1957).
Partial least squares regression (PLSR) (Wold et al., 1984)
is better because it weights the predictors to minimise the
prediction variances. McBride refers to these methods as
‘data manipulations’. One might regard PCR as such
because it makes no assumptions about the relations
between the leading components and target variables.
But to view PLSR in that way is a mistake; it is a sound
form of statistical analysis that optimises predictions and
reveals the relative importance of the predictor variables.
PLSR is a vital tool in chemometrics (Wold et al., 2001).
There are, of course, other ways of dealing with the many
multicollinear predictors, but McBride fails to mention
any of them. These, too, have been reported in the litera-
ture on soil spectroscopy. They include, for example,
wavelet multi-resolution analysis (Viscarra Rossel,
Behrens, et al., 2016; Viscarra Rossel, Brus, et al., 2016;
Viscarra Rossel & Lark, 2009; Vohland et al., 2016) and
the variance inflation factor (Song et al., 2021).

**5.1** | **Extrapolations and overfitting**

McBride (2022) criticises extrapolation beyond the limits
of the data on which the models are built and over-fitting,
that is, fitting a model to virtually meaningless informa-
tion, effectively to noise. They are fair comments, but they
are practices that statisticians have warned against for
many years; they are not confined to soil spectroscopy.

Unfortunately, included in the burgeoning publications
on spectroscopic modelling are many examples of poor
practice—of both unwise extrapolation and over-fitting.
They seem to arise from a poor understanding of spectros-
copy itself, naïve implementation of mathematical and sta-
tistical procedures, poor validation practice, and access to
computer software on which one needs only to ‘press a few
buttons’. As McBride shows, these articles are reasonably
easy to identify. Unfortunately, McBride fails to recognise
the opposite: the many articles that record sound practice.
When the modelling is robust (Hastie et al., 2009; Srivastava
et al., 2014), and the spectroscopic models are interrogated,
interpreted and validated, there is little risk of extrapolation
or over-fitting.

6 | MACHINE LEARNING

The increased capacity and speed of computers in the last
20 years have enabled mathematicians and data scientists
to develop algorithms for machine learning. McBride
(2022) is critical of these and calls them ‘black boxes’.
There are indeed examples that report on the spectroscopic
modelling with machine learning that lack adequate imple-
mentation and interpretation, undermining the confidence
in those models. If one only read those reports, there might
be some justification for labelling all machine learning as
obscure, uninterpretable ‘black boxes’.

We agree that machine learning should be used with
caution and judged on its performance. We should ask
the following questions before using machine learning
for spectroscopic modelling. Is it necessary? Even if not,
can we use it to our gain? Or is it a ‘sledgehammer to
crack a nut’ with no advantages over more straightforward and transparent regression methods? Whatever we
answer, we must recognise that machine learning is here
to stay (Heuvelink & Webster, 2022). Further, it will
become ever more powerful and potentially more helpful
as time passes. If we are to use machine learning for estimating soil constituents with spectra, then we must validate the outcomes with adequately sampled, independent
data (Brown et al., 2005; Spiegelhalter, 2019). Further, if
we are to understand the outcomes, we must learn how
the input variables contribute meaningfully to the predictions. McBride itemises the steps we should take but still
lacks confidence in the procedures.

The problem of modelling with highly dimensional
data does not affect all machine learning methods, or at
least not to the same extent. For example, methods based
on regression trees tend to minimise the vector space for
predictions and thus prevent the ‘curse of dimensionality’. Random forests (Breiman, 2001) create many trees,
but only a few variables bear on the target within any
one tree. Deep convolutional neural networks (CNN), a
more recent innovation, are proving effective for analysing intricate structures in high-dimensional data (LeCun
et al., 2015). They can effectively separate the signal from
the noise in the highly-dimensional, multi-collinear soil
spectra (Shen & Viscarra Rossel, 2021; Yang et al., 2020).

Before we conclude this section, we draw attention to
the interpretability of machine learning. Interpretability
varies depending on the algorithm and how one uses it
(Viscarra Rossel & Behrens, 2010). Spectroscopic model-
ling with regression trees is explainable and interpret-
able. For example, CUBIST (Quinlan, 1992), a now
popular (and well-performing) method for spectroscopic
modelling (Viscarra Rossel & Webster, 2012), uses
human-readable **if-then** conditions to split the spectra
into coherent ‘branches’ and fits (well-understood) piece-
wise multiple linear regressions to the outcomes of each
split. We agree that other methods, like support vector
machines (SVM) and deep CNNs, can be difficult to inter-
pret because of the complex data transformations or their
huge and complex network of parameters. However,
there is rapidly advancing research on explainable artificial
intelligence (XAI) (Gunning et al., 2019; Savage, 2022).

These developments, based on sound theoretical founda-
tions, are transforming the ‘black-boxes’ into ‘grey-’ and
even ‘white-boxes’ (e.g., Chen et al., 2020). The methods
are finding applications in soil science and soil spectros-
copy. For example, SHapley Additive exPlanations (SHAP)
(Lundberg & Lee, 2017), based on game theory
(Roth, 1988), has been used successfully for meaningful
interpretation in spectroscopic machine learning (Haghi
et al., 2021; Zhong et al., 2021).

7 | SPECTRAL LIBRARIES AND MODEL LOCALIZATION

For decades, soil scientists have been developing databases
(or libraries) of soil properties with corresponding spectra
for regions of various sizes, from strictly local to national
(e.g., Viscarra Rossel & Webster, 2012; Wijewardane et al.,
2018), continental (e.g., Stevens et al., 2013), and global
(Shepherd & Walsh, 2002; Viscarra Rossel, Behrens, et al.,
2016). Many of these libraries were developed by measur-
ing the spectra of soil samples stored in archives as lega-
cies from different experiments and surveys at different
scales. The analytical reference data in those libraries often
require significant analysis, preprocessing and harmonisa-
tion to ensure that they are consistent and of good quality
for modelling. Since the soil samples are not collected
using a sampling design suitable for developing a spectral
library and subsequent modelling, one must also judi-
ciously validate the spectroscopic models to prevent over-
optimistic results.

In earlier research (e.g., Brown et al. (2006) cited by
McBride (2022)), the main reason for developing spectral
libraries was to see if one could use them to build unique
calibrations to estimate soil properties. The approach was
taken from spectroscopic studies in other domains, devel-
oped for estimating the concentrations of substances that
are much less complex than soil. Soil scientists embraced
the approach directly and without alteration, often result-
ing in suboptimal spectroscopic models of soil properties
and a prejudiced perception of its potential.

Since then, research has shown that soil spectral
libraries should be used differently, not to make large
general models but as a source of information for build-
ing localised calibrations for specific contexts and pedo-
logic domains. We have learned that one must be wary of
using a general calibration derived from all of the data in
a country-wide national or global library for estimation
locally in a small region (Guerrero et al., 2016). Unlike
the estimates from calibrations derived locally, these gen-
eral models are likely to produce biased estimates
because the general relationships will not depict those
present locally and are likely also to mask local variations
(Shen et al., 2022). Thus, even if we had spectral libraries with an infinite number of samples and derived calibrations with all the data, this problem would remain.

Current understanding and ongoing research suggest that large and diverse spectral libraries are beneficial, but the development of methods for ‘localising’ the spectroscopic modelling is equally important. The literature contains reports on several of these methods. For instance, practical methods such as spiking (Guerrero et al., 2010; Wetterlind & Stenberg, 2010) and spiking with extra-weighting (Guerrero et al., 2014), the LOCAL algorithm (Shenk & Westerhaus, 1991), memory-based learning (MBL) methods that focus on a deterministic local search of the spectral library (Ramirez-Lopez et al., 2013), data-driven stochastic search methods such as RS-LOCAL (Lobsey et al., 2017; Shen et al., 2022), to feature-based deep transfer learning (Liu et al., 2018). McBride seems to have missed all of this research. Most recently, Shen et al. (2022) showed that instance-based deep transfer learning lessens the need for conventional analytical measurements. They suggest that as these methods and spectral libraries develop, the need for conventional analysis will further diminish and eventually disappear without losing accuracy that entirely local modelling would provide. Research is ongoing as those ideas need further experimentation and testing, however.

## 8 | CONCLUSIONS

The recent article in this journal by McBride (2022), under the heading ‘Opinion’, criticised reflectance spectroscopy for estimating the concentrations of soil constituents. Some of that criticism is fair; many exponents have exaggerated claims about the technology. Other aspects of McBride’s Opinion are outdated, incorrect or otherwise misleading. We countered McBride’s views to provide readers with more balanced insight into soil spectroscopy and its merits. In responding to McBride’s article, we have tried to distinguish between fair comments and what we regard as outdated statements of fact and false inferences. We have concentrated on what we regard as the most important matters and we have omitted issues that are discussed elsewhere, for example, the spectroscopic estimation of trace elements in soil (Baveye & Laba, 2015; Shi et al., 2015).

Like most new methods and technologies, soil spectroscopy has had its hiccups, and some practitioners have undoubtedly been over-enthusiastic in its application. However, as the technology develops and interest in soil spectroscopy grows further, practitioners understand the various methods better in practice and research and are becoming more adept at applying them. As a result, soil

spectroscopy is becoming an essential tool for measuring and monitoring soil and obtaining large amounts of quantitative soil information in a broad range of soil and environmental science applications. Most practitioners of soil spectroscopy do not seek to replace conventional analytical methods; instead, they see soil spectroscopy working alongside conventional analyses as an essential partner. Other scientists have recognised this too. The Global Soil Partnership of the Food and Agriculture Origin-ination (FAO) of the United Nations and its Global Soil Laboratory Network initiative on soil spectroscopy (GLOSOLAN-Spec) is helping to combine conventional soil testing and spectroscopy. It is also helping to bring the community together via a scientific community of practice to steer future developments and build capacity in soil spectroscopy globally but recognising that individual countries will have their particular approaches.

Soil is complex, and so are the relationships between its components and the spectra. Sophisticated technologies and mathematical and statistical methods are needed to characterise and deal with such complexity. However, just because we do not understand them entirely does not mean we should shy away from them. On the contrary, we should embrace them and strive to comprehend and use them to gain insights into the complexity of soil. We hope that our article gives readers and newcomers to soil spectroscopy a more objective, balanced and optimistic view of the subject and its future.

### ENDNOTE

### ACKNOWLEDGEMENTS

Raphael A. Viscarra Rossel received funding from the Australian Government via grant ACSRIV000077.

### DATA AVAILABILITY STATEMENT

Data sharing not applicable to this article as no datasets were generated or analysed during the current study.

### ORCID

Raphael A. Viscarra Rossel https://orcid.org/0000-0003-1540-4748

José Alexandre Melo Dematté https://orcid.org/0000-0001-5328-0323

Cecile Gomez https://orcid.org/0000-0002-2986-430X

<footnote n="1">¹ http://www.cleanenergyregulator.gov.au/ERF/Choosing-a-project-type/Opportunities-for-the-land-sector/Agricultural-methods/estimating-soil-organic-carbon-sequestration-using-measurement-and-models-method</footnote>
REFERENCES

Baumann, P., Lee, J., Behrens, T., Biswas, A., Six, J., McLachlan, G., & Viscarra Rossel, R. A. (2022). Modelling soil water retention and water-holding capacity with visible-near-infrared spectra and machine learning. *European Journal of Soil Science*, 73(2), e13220.

Baveye, P. C., & Laba, M. (2015). Visible and near-infrared reflectance spectroscopy is of limited practical use to monitor soil contamination by heavy metals. *Journal of Hazardous Materials*, 285, 137-139.

Bellman, R. (1957). *Dynamic programming*. Princeton University Press.

Ben-Dor, E., & Banin, A. (1995). Near-infrared analysis as a rapid method to simultaneously evaluate several soil properties. *Soil Science Society of America Journal*, 59, 364–372.

Bergstrom, C. T., & West, J. D. (2020). *Calling bullshit: The art of skepticism in a data-drive world*. Random House.

Bowers, S. A., & Hanks, R. J. (1965). Reflection of radiant energy from soils. *Soil Science*, 100, 130–138.

Breiman, L. (2001). Random forests. *Machine Learning*, 45, 5–32.

Brown, D. J., Bricklemyer, R. S., & Miller, P. R. (2005). Validation requirements for diffuse reflectance soil characterization models with a case study of VNIR soil C prediction in Montana. *Geoderma*, 129, 251–267.

Brown, D. J., Shepherd, K. D., Walsh, M. G., Dewayne Mays, M., & Reinsch, T. G. (2006). Global soil characterization with VNIR diffuse reflectance spectroscopy. *Geoderma*, 132, 273–290.

Chen, Z., Bei, Y., & Rudin, C. (2020). Concept whitening for interpretable image recognition. *Nature Machine Intelligence*, 2, 772-782.

Clark, R. N. (1999). Spectroscopy of rocks and minerals and principles of spectroscopy. In A. N. Rencz (Ed.), *Manual of remote sensing, volume 3, remote sensing for the earth sciences* (3rd ed., pp. 3–58). John Wiley & Sons.

Clark, R. N., King, T. V. V., Klejwa, M., Swayze, G. A., & Vergo, N. (1990). High spectral resolution reflectance spectroscopy of minerals. *Journal of Geophysical Research: Solid Earth and Planets*, 95(B8), 12653–12680.

England, J. R., & Viscarra Rossel, R. A. (2018). Proximal sensing for soil carbon accounting. *The Soil*, 4, 101–122.

Farmer, V. C. (1974). *The infrared spectra of minerals, mineralogical society monograph* (4th ed.). The Mineralogical Society.

Guerrero, C., & Lorenzetti, R. (2021). Use of composite samples and NIR spectroscopy to detect changes in SOC contents. *Geoderma*, 396, 115069.

Guerrero, C., Stenberg, B., Wetterlind, J., Viscarra Rossel, R. A., Maestre, F. T., Mouazen, A. M., Zornoza, R., Ruiz-Sinoga, J. D., & Kuang, B. (2014). Assessment of soil organic carbon at local scale with spiked NIR calibrations: Effects of selection and extra-weighting on the spiking subset. *European Journal of Soil Science*, 65, 248–263.

Guerrero, C., Wetterlind, J., Stenberg, B., Mouazen, A. M., Gabarrón-Galeote, M. A., Ruiz-Sinoga, J. D., Zornoza, R., & Viscarra Rossel, R. A. (2016). Do we really need large spectral libraries for local scale SOC assessment with NIR spectroscopy? *Soil and Tillage Research*, 155, 501–509.

Guerrero, C., Zornoza, R., Gómez, I., & Mataix-Beneyto, J. (2010). Spiking of NIR regional models using samples from target sites: Effect of model size on prediction accuracy. *Geoderma*, 158, 66–77.

Gunning, D., Stefik, M., Choi, J., Miller, T., Stumpf, S., & Yang, G.-Z. (2019). XAI-explainable artificial intelligence. *Science Robotics*, 4(37), eaay7120. https://doi.org/10.1126/scirobotics.aay7120

Haghi, R., Pérez-Fernández, E., & Robertson, A. (2021). Prediction of various soil properties for a national spatial dataset of scottish soils based on four different chemometric approaches: A comparison of near infrared and mid-infrared spectroscopy. *Geoderma*, 396, 115071.

Hastie, T., Tibshirani, R., & Friedman, J. H. (2009). *The elements of statistical learning: Data mining, inference, and prediction* (2nd ed.). Springer.

Heuvelink, G. B. M., & Webster, R. (2022). Spatial statistics and soil mapping: A blossoming partnership under pressure. *Spatial statistics*, 50, 100639.

Hotelling, H. (1957). The relations of the newer multivariate statistical methods to factor analysis. *British Journal of Statistical Psychology*, 10, 69–79.

Janik, L. J., Merry, R. H., & Skjemstad, J. O. (1998). Can mid infrared diffuse reflectance analysis replace soil extractions? *Australian Journal of Experimental Agriculture*, 38, 681–696.

Janik, L. J., Skjemstad, J. O., & Raven, M. D. (1995). Characterization and analysis of soils using mid-infrared partial least-squares. *1. Correlations with XRF-determined major-element composition. Australian Journal of Soil Research*, 33, 621–636.

Ji, W., Viscarra Rossel, R. A., & Shi, Z. (2015). Accounting for the effects of water and the environment on proximally sensed Vis-NIR soil spectra and their calibrations. *European Journal of Soil Science*, 66, 555–565.

LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. *Nature*, 521(7553), 436–444.

Lee, J., & Viscarra Rossel, R. A. (2020). Soil carbon simulation founded by different pool initialisation. *Nutrient Cycling in Agroecosystems*, 116, 245–255.

Lee, J., Viscarra Rossel, R. A., Zhang, M., Luo, Z., & Wang, Y.-P. (2021). Assessing the response of soil carbon in Australia to changing inputs and climate using a consistent modelling framework. *Biogeosciences*, 18, 5185–5202.

Li, S., Shi, Z., Chen, S., Ji, W., Zhou, L., Yu, W., & Webster, R. (2015). In situ measurements of organic carbon in soil profiles using Vis-NIR spectroscopy on the Qinghai–Tibet plateau. *Environmental Science & Technology*, 49, 4980–4987.

Li, S., Viscarra Rossel, R. A., & Webster, R. (2022). The cost-effectiveness of reflectance spectroscopy for estimating soil organic carbon. *European Journal of Soil Science*, 73, e13202.

Liu, L., Ji, M., & Buchroithner, M. (2018). Transfer learning for soil spectroscopy based on convolutional neural networks and its application in soil clay content mapping using hyperspectral imagery. *Sensors*, 18, 3169.

Lobell, D. B., & Asner, G. P. (2002). Moisture effects on soil reflectance. *Soil Science Society of America Journal*, 66, 722–727.

Lobsey, C., Viscarra Rossel, R. A., Roudier, P. & Hedley, C. (2017). RS-local data-mines information from spectral libraries to improve local calibrations. *European Journal of Soil Science*, 68, 840–852. Check caps for RS, 68, 840, 852.

Lundberg, S. M., & Lee, S.-I. (2017). A unified approach to interpreting model predictions. In *Proceedings of the 31st*
international conference on neural information processing sys-
tems. NIPS'17 (pp. 4768-4777) arXiv:1705.07874v2.

Martens, H., & Næs, T. (1989). *Multivariate Calibration*. John Wiley & Sons.

McBride, M. B. (2022). Estimating soil chemical properties by dif-
fuse reflectance spectroscopy: Promise versus reality. *European
Journal of Soil Science*, *73*, e13192.

McClure, W. F. (2003). 204 years of near infrared technology: 1800–2003. *Journal of Near Infrared Spectroscopy, 11*, 487–518.

Minasny, B., McBratney, A. B., Bellon-Maurel, V., Roger, J.-M., Gobrecht, A., Ferrand, L., & Joalland, S. (2011). Removing the effect of soil moisture from NIR diffuse reflectance spectra for the prediction of soil organic carbon. *Geoderma, 167–168*, 118–124.

Nguyen, T. T., Janik, L. J., & Raupach, M. (1991). Diffuse reflec-
tance infrared Fourier transform (drift) spectroscopy in soil
studies. *Australian Journal of Soil Research*, *29*, 49–67.

O'Rourke, S. M., & Holden, N. M. (2011). Optical sensing and chemometric analysis of soil organic carbon-a cost effective alternative to conventional laboratory methods. *Soil Use and Management*, *27*, 143–155.

Pasquini, C. (2018). Near infrared spectroscopy: A mature analytical technique with new perspectives–a review. *Analytica Chimica Acta*, *1026*, 8–36.

Quinlan, J. R. (1992). Learning with continuous classes. In A. Adams & L. Sterling (Eds.), *5th Australian conference on artificial intelligence* (Vol. 92, pp. 343–348). World Scientific.

Ramirez-Lopez, L., Behrens, T., Schmidt, K., Stevens, A., Demattê, J. A. M., & Scholten, T. (2013). The spectrum-based learner: A new local approach for modeling soil Vis-NIR spectra of complex datasets. *Geoderma, 195*, 268–279.

Ramirez-Lopez, L., Wadoux, A. M. J.-C., Franceschini, M. H. D., Terra, F. S., Marques, K. P. P., Sayão, V. M., & Demattê, J. A. M. (2019). Robust soil mapping at the farm scale with Vis-NIR spectroscopy. *European Journal of Soil Science*, *70*, 378–393.

Roth, A. E. (1988). *The Shapley value: Essays in honor of Lloyd S. Cambridge University Press, Cambridge.*

Savage, N. (2022). Breaking into the black box of artificial intelligence. *Nature*. https://doi.org/10.1038/d41586-022-00858-1

Shen, Z., Ramirez-Lopez, L., Behrens, T., Cui, L., Zhang, M., Walden, L., Wetterlind, J., Shi, Z., Sudduth, K. A., Baumann, P., Song, Y., Catambay, K., & Viscarra Rossel, R. A. (2022). Deep transfer learning of global spectra for local soil carbon monitoring. *ISPRS Journal of Photogrammetry and Remote Sensing, 188*, 190–200.

Shen, Z., & Viscarra Rossel, R. A. (2021). Automated spectroscopic modelling with optimised convolutional neural networks. *Scientific Reports, 11*(1), 1–12.

Shenk, J. S., & Westerhaus, M. O. (1991). Population definition, sample selection, and calibration procedures for near-infrared reflectance spectroscopy. *Crop Science, 31*, 469–474.

Shepherd, K. D., & Walsh, M. G. (2002). Development of reflectance spectral libraries for characterization of soil properties. *Soil Science Society of America Journal, 66*, 988–998.

Shi, T., Chen, Y., Liu, Y., & Wu, G. (2015). Response to “visible and near-infrared reflectance spectroscopy is of limited practical use to monitor soil contamination by heavy metals” by Philippe C. Baveye. *Journal of Hazardous Materials, 285*, 207.

Song, Y. Z., Shen, Z. F., Wu, P., & Viscarra Rossel, R. A. (2021). Wavelet geographically weighted regression for spectroscopic modelling of soil properties. *Scientific Reports, 11*, 17503.

Soriano-Disla, J. M., Janik, L. J., Viscarra Rossel, R. A., Macdonald, L. M., & McLaughlin, M. J. (2014). The performance of visible, near-, and mid-infrared reflectance spectroscopy for prediction of soil physical, chemical, and biological properties. *Applied Spectroscopy Reviews, 49*, 139–186.

Spiegelhalter, D. J. (2019). *The art of statistics: Learning from data*. Penguin Books.

Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. (2014). Dropout: A simple way to prevent neural networks from overfitting. *Journal of Machine Learning Research, 15*, 1929–1958.

Stenberg, B., Viscarra Rossel, R. A., Mouazen, A. M., & Wetterlind, J. (2010). Visible and near infrared spectroscopy in soil science. *Advances in Agronomy, 107*, 163–215.

Stevens, A., Nocita, M., Tóth, G., Montanarella, L., & van Wesemael, B. (2013). Prediction of soil organic carbon at the European scale by visible and near infrared reflectance spectroscopy. *PLoS One, 8*(6), e66409.

Vågen, T.-G., Winowiecki, L. A., Neely, C., Chesterman, S., & Bourne, M. (2018). Spatial assessments of soil organic carbon for stakeholder decision-making – A case study from Kenya. *The Soil, 4*, 259–266.

Vågen, T.-G., Winowiecki, L. A., Tondoh, J. E., Desta, L. T., & Gumbricht, T. (2016). Mapping of soil properties and land degradation risk in Africa using MODIS reflectance. *Geoderma*, *263*, 216–225.

van Leeuwen, C. C. E., Mulder, V. L., Batjes, N. H., & Heuvelink, G. B. M. (2022). Statistical modelling of measurement error in wet chemistry soil data. *European Journal of Soil Science*, *73*, e13137.

Viscarra Rossel, R. A. (2011). Fine-resolution multiscale mapping of clay minerals in Australian soils measured with near infrared spectra. *Journal of Geophysical Research: Earth Surface, 116*, F04023.

Viscarra Rossel, R. A., & Behrens, T. (2010). Using data mining to model and interpret soil diffuse reflectance spectra. *Geoderma*, *158*, 46–54.

Viscarra Rossel, R. A., Behrens, T., Ben-Dor, E., Brown, D. J., Demattê, J. A. M., Shepherd, K. D., Shi, Z., Stenberg, B., Stevens, A., Adamchuk, V., Aïchi, H., Barthès, B. G., Bartholomeus, H. M., Bayer, A. D., Bernoux, M., Böttcher, K., Brodský, L., du, C. W., Chappell, A., ... Ji, W. (2016). A global spectral library to characterize the world's soil. *Earth-Science Reviews, 155*, 198–230.

Viscarra Rossel, R. A., & Bouma, J. (2016). Soil sensing: A new paradigm for agriculture. *Agricultural Systems, 148*, 71–74.

Viscarra Rossel, R. A., & Brus, D. J. (2018). The cost-efficiency and reliability of two methods for soil organic C accounting. *Land Degradation & Development, 29*, 506–520.

Viscarra Rossel, R. A., Brus, D. J., Lobsey, C., Shi, Z., & McLachlan, G. (2016). Baseline estimates of soil organic carbon by proximal sensing: Comparing design-based, model-assisted and model-based inference. *Geoderma, 265*, 152–163.

Viscarra Rossel, R. A., Bui, E. N., de Caritat, P., & McKenzie, N. J. (2010). Mapping iron oxides and the color of australian soil
using visible-near-infrared reflectance spectra. *Journal of Geophysical Research: Earth Surface, 115*, F04031.

Viscarra Rossel, R. A., Chen, C., Grundy, M. J., Searle, R., Clifford, D., & Campbell, P. H. (2015). The Australian three-dimensional soil grid: Australia's contribution to the Global-SoilMap project. *Soil Research, 53*, 845-864.

Viscarra Rossel, R. A., & Hicks, W. S. (2015). Soil organic carbon and its fractions estimated by visible-near infrared transfer functions. *European Journal of Soil Science, 66*, 438-450.

Viscarra Rossel, R. A., & Lark, R. M. (2009). Improved analysis and modelling of soil diffuse reflectance spectra using wavelets. *European Journal of Soil Science, 60*, 453-464.

Viscarra Rossel, R. A., Lobsey, C. R., Sharman, C., Flick, P., & McLachlan, G. (2017). Novel proximal sensing for monitoring soil organic C stocks and condition. *Environmental Science & Technology, 51*, 5630-5641.

Viscarra Rossel, R. A., Walvoort, D. J. J., McBratney, A. B., Janik, L. J., & Skjemstad, J. O. (2006). Visible, near infrared, mid infrared or combined diffuse reflectance spectroscopy for simultaneous assessment of various soil properties. *Geoderma, 131*, 59-75.

Viscarra Rossel, R. A., & Webster, R. (2012). Predicting soil properties from the australian soil visible-near infrared spectroscopic database. *European Journal of Soil Science, 63*, 848-860.

Vohland, M., Ludwig, M., Harbich, M., Emmerling, C., & Thiele-Bruhn, S. (2016). Using variable selection and wavelets to exploit the full potential of visible-near infrared spectra for predicting soil properties. *Journal of Near Infrared Spectroscopy, 24*, 255-269.

Wetterlind, J., & Stenberg, B. (2010). Near-infrared spectroscopy for within-field soil characterization: Small local calibrations compared with national libraries spiked with local samples. *European Journal of Soil Science, 61*, 823-843.

Wetterlind, J., Stenberg, B., & Jonsson, A. (2008). Near infrared reflectance spectroscopy compared with soil clay and organic matter content for estimating within-field variation in N uptake in cereals. *Plant and Soil, 302*, 317-327.

Wijewardane, N. K., Ge, Y., & Morgan, C. L. S. (2016). Prediction of soil organic and inorganic carbon at different moisture contents with dry ground VNIR: A comparative study of different approaches. *European Journal of Soil Science, 67*, 605-615.

Wijewardane, N. K., Ge, Y. F., Wills, S., & Libohova, Z. (2018). Predicting physical and chemical properties of us soils with a mid-infrared reflectance spectral library. *Soil Science Society of America Journal, 82*, 722-731.

Wold, S., Ruhe, A., Wold, H., & Dunn, W. J., III. (1984). The collinearity problem in linear regression. The partial least squares (PLS) approach to generalized inverses. *SIAM Journal on Scientific and Statistical Computing, 5*, 735-743.

Wold, S., Sjöström, M., & Eriksson, L. (2001). PLS-regression: A basic tool of chemometrics. *Chemometrics and Intelligent Laboratory Systems, 58*, 109-130.

Workman, J. J., Jr. (1996). Interpretive spectroscopy for near infrared. *Applied Spectroscopy Reviews, 31*, 251-320.

Yang, J., Wang, X., Wang, R., & Wang, H. (2020). Combination of convolutional neural networks and recurrent neural networks for predicting soil properties using Vis-NIR spectroscopy. *Geoderma, 380*, 114616.

Zhong, L., Guo, X., Xu, Z., & Ding, M. (2021). Soil properties: Their prediction and feature extraction from the LUCAS spectral library using deep convolutional neural networks. *Geoderma, 402*, 115366.

**How to cite this article:** Viscarra Rossel, R. A., Behrens, T., Ben-Dor, E., Chabrillat, S., Demattê, J. A. M., Ge, Y., Gomez, C., Guerrero, C., Peng, Y., Ramirez-Lopez, L., Shi, Z., Stenberg, B., Webster, R., Winowiecki, L., & Shen, Z. (2022). Diffuse reflectance spectroscopy for estimating soil properties: A technology for the 21st century. *European Journal of Soil Science, 73*(4), e13271.
https://doi.org/10.1111/ejss.13271