# What is cost-efficient phenotyping? Optimizing costs for different scenarios

Daniel Reynolds, Frederic Baret, Claude Welcker, Aaron Bostrom, Joshua Ball, Francesco Cellini, Argelia Lorence, Aakash Chawade, Mehdi Khafif, Koji Noshita, et al.

► To cite this version:

Daniel Reynolds, Frederic Baret, Claude Welcker, Aaron Bostrom, Joshua Ball, et al. What is cost-efficient phenotyping? Optimizing costs for different scenarios. Plant Science, 2019, 282, pp.14-22. 10.1016/j.plantsci.2018.06.015. hal-02628161

HAL Id: hal-02628161

https://hal.inrae.fr/hal-02628161

Submitted on 26 May 2020

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
Accepted Manuscript

Title: What is cost-efficient phenotyping? Optimizing costs for different scenarios

Authors: Daniel Reynolds, Frederic Baret, Claude Welcker, Aaron Bostrom, Joshua Ball, Francesco Cellini, Argelia Lorence, Aakash Chawade, Mehdi Khafif, Koji Noshita, Mark Mueller-Linow, Ji Zhou, François Tardieu

PII: S0168-9452(17)31148-2

DOI: https://doi.org/10.1016/j.plantsci.2018.06.015

Reference: PSL 9884

To appear in: *Plant Science*

Received date: 5-12-2017

Revised date: 17-5-2018

Accepted date: 13-6-2018

<figure><img src="image_2.png" /></figure>

Please cite this article as: Reynolds D, Baret F, Welcker C, Bostrom A, Ball J, Cellini F, Lorence A, Chawade A, Khafif M, Noshita K, Mueller-Linow M, Zhou J, Tardieu F, What is cost-efficient phenotyping? Optimizing costs for different scenarios, *Plant Science* (2018), https://doi.org/10.1016/j.plantsci.2018.06.015

This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
Review. What is cost-efficient phenotyping? Optimizing costs for different scenarios

Daniel Reynolds<sup>+1</sup>, Frederic Baret<sup>+2</sup>, Claude Welcker<sup>+3</sup>, Aaron Bostrom<sup>+1</sup>, Joshua Ball<sup>1</sup>, Francesco Cellini<sup>4</sup>, Argelia Lorence<sup>5</sup>, Aakash Chawade<sup>6</sup>, Mehdi Khafif<sup>7</sup>, Koji Noshita<sup>8</sup>, Mark Mueller-Linow<sup>9</sup>, Ji Zhou<sup>*1,10</sup>, François Tardieu<sup>*3</sup>

<sup>1</sup> Earlham Institute, Norwich Research Park, Norwich UK, NR4 7UH

<sup>2</sup> INRA EMMAH – CAPTE, address, Avignon France

<sup>3</sup> INRA Univ Montpellier, LEPSE, 2 place Viala 34060 Montpellier France

<sup>4</sup> Agenzia Lucana di Sviluppo e di Innovazione in Agricoltura 75010 Metaponto (MT)
Italy

<sup>5</sup> Phenomics Facility, Arkansas Biosciences Institute, Arkansas State University, Jonesboro,
Arkansas, USA

<sup>6</sup>Department of Plant Breeding, Swedish University of Agricultural Sciences (SLU), P.O. Box 101,
230 53 Alnarp, Sweden

<sup>7</sup> Université de Toulouse, INRA, CNRS, LIPM Castanet-Tolosan, France.

<sup>8</sup> Japan Science and Technology Agency (JST) Precursory Research for Embryonic Science and
Technology (PRESTO), Graduate School of Agriculture and Life Science, The University of Tokyo

<sup>9</sup> Institute of Bio- and Geosciences (IBG), IBG-2: Plant Sciences, Forschungszentrum Juelich
GmbH, Juelich, Germany

<sup>10</sup> Plant Phenomics Research Center, Nanjing Agricultural University, Nanjing China, 210095

<footnote>+Joint first authors</footnote>

<footnote>*Corresponding Authors:</footnote>

<footnote>francois.tardieu@inra.fr, ji.zhou@earlham.ac.uk or ji.zhou@njau.edu.cn</footnote>
**Highlights**

* New technologies considerably reduce the costs of sensors and automated vehicles

* Low investment in sensors, vehicles or pipelines present trade-offs with labor costs

* Plant/plot handling and labor costs represent the major proportion of costs in phenotyping experiments

* The costs of high-throughput experiments in the field and in automated platforms is similar regardless of vehicles

* The development of software applications (e.g. imaging, phenotypic analyses, models, information system) is a major part of costs

**Abstract**

Progress in remote sensing and robotic technologies decreases the hardware costs of phenotyping. Here, we first review cost-effective imaging devices and environmental sensors, and present a trade-off between investment and manpower costs. We then discuss the structure of costs in various real-world scenarios. Hand-held low-cost sensors are suitable for quick and infrequent plant diagnostic measurements. In experiments for genetic or agronomic analyses, (i) major costs arise from plant handling and manpower; (ii) the total costs per pot/microplot are similar in robotized platform or field experiments with drones, hand-held or robotized ground vehicles; (iii) the cost of vehicles carrying sensors represents only 5-26% of the total costs. These conclusions depend on the context, in particular for labor cost, the quantitative demand of phenotyping and the number of days available for phenotypic measurements due to climatic constraints. Data analysis represents 10-20% of total cost if pipelines have already been developed. A trade-off exists between the initial high cost of pipeline development and labor cost of manual operations. Overall, depending on the context and objectives, "cost-effective" phenotyping may involve either low investment ("affordable"
phenotyping"), or initial high investments in sensors, vehicles and pipelines that result in higher quality and lower operational costs.

**Keywords** Phenotyping; Phenomics; Cost; imaging; information system; affordable;

**Introduction**

The observation of growing plants can involve operations of different nature. For instance, when a farmer visits fields to decide if and when an operation needs to be carried out, e.g. irrigation, fertilization or harvest, this is essentially based on direct observations that may be helped by low-throughput tools. The same tools can be used in nurseries when a breeder rapidly inspects tens of thousands of plants of a population with the aim of identifying, for instance, plants of abnormal aspect or with high sensitivity to a disease. At the other extreme, genome wide association studies (GWAS) or genomic predictions require analysis of hundreds of lines to identify the genetic variability of traits associated with plant performance in diverse conditions. This translates into thousands of plants in greenhouse robotized platforms, or of microplots (i.e. a plot of typically 4-10 m² with a single genotype) in field experiments. Such experiments involve (i) novel technologies for collecting relevant images of each plant or microplot, able to characterize the temporal and spatial variability of traits; (ii) the design and maintenance of pipelines of image analyses allowing one to extract quantitative traits from images; (iii) analyses of datasets originating from different installations at different scales (e.g. phenotyping platforms in greenhouses or in the field at organ, plant or canopy levels); and (iv) shared information systems able to manage and store data in such a way that data can be re-used or re-analyzed by the scientific community [1-3].

The concept of “affordable phenotyping” or “cost-effective phenotyping” has developed rapidly in recent years due to decreasing cost of equipment such as low-cost environmental sensors [4] or smartphone-embedded and mobile imaging sensors [5]. Indeed, cost-effective phenotyping approaches have been utilized to capture image- and sensor-based crop performance datasets
in greenhouses and in the field [6–8]. For example, ground-based portable devices [9,10] have
been used to estimate canopy photosynthesis rate at key developmental stages; mobile phone
cameras are also used to capture crop disease symptoms and plant morphology [11–15];
unmanned aerial vehicles (UAVs) equipped with relatively low-cost RGB (red-green-blue)
cameras are employed to study crop performance and field variability under different growing
conditions [16–18].

Depending on the number and complexity of operations associated with the observation of a given set of phenotypic traits, the cost of equipment can represent a variable fraction of the total cost of the phenotyping program. Hence, the cost of specific pieces of equipment should be considered as a part of the costs of the whole phenotyping process. For example, low-cost hardware can be appropriate for diagnostic or quick characterization of a few plants in a field experiment. If many plants or plots have to be sampled several times during the crop cycle, this may result in higher cost related to the additional human effort required for the analysis of poorly calibrated and documented data, in order to obtain interpretable and heritable variables.

Plant breeding programs are also potential end-users of phenomics and need to analyze
whether the investment in a particular phenotyping technology will achieve a justifiable
increase in the rate of genetic gain. It is important to acknowledge here that, at this stage, the
extent to which phenomics can substantially increase this rate is discussed. Breeders have been
successful in increasing yield, e.g. in wheat [19,20] and maize [21], essentially based on direct
selection for yield. The success of trait-based selection has been focused on visually observable
traits such as anthesis-silking interval in maize, disease symptoms, growth phenotypes [22], and
flowering [23], which do not require high investment. Novel breeding techniques such as
genomic selection may reinforce the power of yield-based selection perhaps at the expense of
trait-based selection [20,24], thereby decreasing the interest of phenotypic analyses to focus on
increasing the average yield in a given region [20]. It has been proposed that the contribution of
phenomics to pre-breeding may involve novel biological applications, for instance (i) where and
when do genotypes or alleles present in the genetic diversity present comparative advantages,
and (ii) whether one can make the best use of combinations of alleles controlling adaptive traits
(e.g. the controls of stomatal conductance or growth) as a function of environmental conditions [25]. These questions involve a combination of phenomics, modelling and genomic prediction to assess the genetic and environmental controls of plant adaptation [25]. Addressing the above questions may be essential for breeding in a context of climate change, but it is currently upstream of most breeding programs. Until clear contributions of phenomics to breeding have been demonstrated in particular contexts, it might be misleading to attempt to evaluate the efficiency of phenomics techniques, either ‘envirotyping’ or plant measurements, in terms of cost per unit genetic gain.

Hence, we hereby focus on the costs of all operations involved in phenomics, and not on the efficiency of their costs for breeding. We first review the current imaging techniques and vehicles carrying the corresponding sensors. We then present the structure of costs associated with phenomics based on case studies for different experiments in the field or in indoor controlled conditions and for different imaging systems.

# I Imaging techniques with a range of hardware costs

## 1.1 Handheld phenotyping technologies

Small, lightweight and reusable devices considerably reduce the hardware costs associated with handheld phenotyping at canopy or leaf level in field conditions, but also at plant level in indoor conditions. For example, using an advanced software approach and commercially available handheld digital cameras, 3D reconstructions at organ level can either be accomplished by combining tens of images of a single plant taken by hand with structure-from-motion and multi-view stereo techniques [26] or by using stereo camera setups and stereo image processing [27]. A 3D reconstruction of a plant row has been performed using a bespoke hand-held sensor platform [28], while a standard RGB camera was used to record color information of scanned areas. A visual-inertia and 2D LiDAR (Light Detection and Ranging) sensor contributed to the reconstruction of colored 3D models of crop areas. Another device connects infrared temperature sensing, GPS positioning and a normalized difference vegetation index (NDVI) sensor, together with a standard laptop mounted to a hand-held pole [29]. A handheld device combines light-emitting diode (LED) lights with visible and infrared sensors in a package able to
calculate light transmission through the surface of a leaf, fluorescence-based kinetics and photosynthesis-associated variables [30]. Standard RGB cameras have been widely used to characterize the canopy structure [31,32], with adaptation to smartphone cameras [15].

The phenotyping devices described above present several limitations. Lower investment costs are most often at the expense of labor-intensive manual control and analysis, otherwise they may lead to the production of non-repeatable datasets. Indeed, these approaches require human decisions for the imaged area, the selection of regions of interest, and, finally, analytical software to standardize and analyze the captured data [5]. Furthermore, the scale of measurement is limited without costly and complex machinery. Hence, it can be considered that handheld devices are most appropriate for actions with limited throughput carried out by experienced plant specialists

## 1.2 Aerial imaging for large-scale phenotyping

Aerial imagery for field conditions provides a sufficient throughput to sample all the plots of a field experiment (typically thousands of microplots) within a short time interval. It is efficient when targeting canopy characteristics that may vary considerably within a short time interval such as canopy temperature [33,34] or changes in canopy structure due to leaf rolling [35]. Traditional manned helicopters are still used because of the heavy payload capacity [36]. Nevertheless, three factors have triggered the rapid development of UAVs for field phenotyping applications in the last five years: (i) the increasing autonomy reliability and payload capacity, (ii) the decrease of the corresponding cost, together with an increase in sensor performance, and (iii) the development of image processing software allowing to precisely compute the position of the UAV corresponding to each individual image and to create an orthomosaic image map of the field [17]. The high-resolution imagery provided by consumer grade RGB cameras has been used to count plants and organs [37] and to evaluate the cover fraction [38]. Using the same RGB cameras, the shape from motion algorithm creates the dense 3D point clouds from which plant height is derived with a very good accuracy [39-42]. Light-weight LiDAR was also tentatively mounted on UAVs to get a more direct estimation of plant height
and canopy related traits [43]. Multispectral and hyperspectral images were used to assess canopy characteristics including the green area index [44] and canopy temperature [34].

The high-throughput of UAV-based observations and its relatively affordable cost makes it potentially very efficient for field phenotyping. However, it needs to operate under favorable conditions, i.e. with no rain, when the illumination is relatively stable and when the wind is not too strong (typically wind speed lower than 35km h⁻¹). This limits the proportion of days during which this technique can be used, thereby increasing the cost per day (see Section II). Furthermore, the massive number of images produced and the intensive computation required to accurately locate images and extract the corresponding microplots contributes to the significant increase of the cost of the traits analyses derived from this technique. Except for LiDAR techniques, the passive nature of UAV observations (the sun being the unique light source) makes the quantification of traits prone to biases due to the specific illumination conditions at the time of image acquisition [45,46]. Recently, UAV costs have increased due to legislation and training issues (see section II).

## 1.3 Imaging with ground vehicles

Low-cost mobile phenotyping systems have been developed by attaching imaging components to existing farm equipment. For example, a tractor can pull a trailer equipped with sensors including a color camera, multiple laser distance scanners, and a hyperspectral imaging sensor [6,46]. Simpler moveable carts have been designed to reduce costs by not requiring pre-existing agricultural equipment, but this is limited to crops with relatively low plant height [47,48]. Alternatively, a large sealed box has been placed around individual plots to capture multi-spectral measurements [49]. A standalone manned vehicle has been developed to carry a thermal infrared camera and a low-cost LiDAR together with light riggings and height adjustable mechanism [4]. Similarly, a mobile phenotyping platform has been developed, equipped with fully adjustable and swappable sensors [29].

The hidden costs of using the phenotyping devices presented above are data calibration, data management and processing. Calibrating the data captured by sensors with manned vehicles in the field can be a time-consuming task due to wide variations in different sensor groups as well
as field regions. It is not only costly but also technically complex to consistently store large quantities of images and sensor data throughout the growing season and associating important metadata (e.g. a time stamp and the corresponding spatial coordinates). Furthermore, well-trained, thus expensive, specialists are needed to operate these manned phenotyping devices.

As a result, more expensive autonomous robotic vehicles have been developed [50]. For example, a fully automatic unmanned robot was specifically designed for field phenotyping applications, controlled by an RTK-GPS positioning system with centimeter accuracy and equipped with modular sensors including LiDAR, multispectral cameras and high resolution RGB cameras [47]. Other vehicles can collect images in a field, together with performing tasks such as seeding, weeding, and harvesting [51]. A robot system has been used to image and analyze berry structure and color in grapevine breeding [48]. Robots with specific phenotyping tasks have also been developed to work alongside a static tower system [52]. Such robotic solutions offer the capacity to use artificial illumination (active imaging), independent from natural illumination conditions (even during the night or cloudy days).

## 1.4 Environmental characterization and envirotyping

Weather stations with data loggers are now widely available for a much reasonable price, thereby making hourly environmental characterization a routine procedure. This can be extended to additional measurements such as soil water content/potential and soil temperature. For instance, electronic tensiometers have been deployed in a network of field experiments for a limited cost [53]. The same applies to installations in controlled conditions, for which measurements of local environmental conditions can be performed with a time step of minutes [54]. Using an "open hardware" design strategy, soil moisture data loggers have been produced using commercially available electronics and sensors [7]. Usability is increased by data transmission over General Packet Radio Service (GPRS), allowing results to be collated off-site without manual harvesting. In addition to GPRS, radio transmission can also be used for data communication within a more complicated network of modular devices [55].

A specific sampling strategy is required to represent the spatial variability of environmental conditions in the field while using fixed sensors. Another problem is the software R&D costs to
cross-reference different static devices in order to extract meaningful information from
collected crop image series and climate datasets using advanced computer vision and data
analytic packages [53]. Small workstations have been developed to provide plot level crop
growth traits as well as micro-environment variables [56]. Multiple sensor types can be
integrated into single-board computers that can then form a scalable, multi-point in-field
network to assist decision making processes such as crop management and line selection.
Modelling is another efficient method for assessing the spatial variability of environmental
conditions, in particular in greenhouse platforms, thereby limiting the number of
environmental sensors deployed in experiments [54].

To our knowledge, the use of sensor networks is currently the main contribution of phenomics
to plant breeding, via the development of ‘envirotyping’ [57–59]. It has been increasingly used
by breeding companies for the identification of environmental scenarios in which combinations
of alleles have positive effects on yield [24,53], the identification of target populations of
environments associated with a breeding program [24,60], or even the definition of new
criteria for developing commercial makes of resilient genotypes [61].

II Costs associated with image capture represent a limited fraction of the overall
cost of phenotyping

**2.1 A method for calculating costs in field and greenhouse platforms**

Calculating costs with a consistent method for field and platform phenotyping is a challenging
task because it is associated with hypotheses and simplification that are debatable by nature. In
Tables 1 and 2, examples for calculation of costs are shown in the field with either automated
ground vehicle, a hand-driven ground vehicle (e.g. handcart or wheelbarrow style trolleys) or a
UAV, or in controlled conditions with a robotized phenotyping platform. Table 1 presents costs
associated with imaging for the typical number of plants or microplots in experiments for each
technique, under two scenarios: (i) in the ‘offer limited’ scenario, the use of devices is limited
by the availability of equipment or personnel; (ii) in the ‘demand limited’ scenario, it is limited
by the number of applications for experiments by public or private users. Both scenarios can co-
exist, for example between years depending on the amount of available funding for Plant
Science, or between installations depending on the demand at a given time. Table 2 presents all
costs associated with a typical experiment using methods presented in Table 1 in the two above
scenarios, including costs for infrastructure, data management and data storage. Both tables
result from surveys performed in the French phenotyping infrastructure Phenome-EMPHASIS.fr
project (www.phenome-EMPHASIS.fr), weighted with information generated from other
infrastructures in UK, USA and Germany. It is noteworthy that these costs correspond to
countries where the labor cost is high. Hence, the conclusions of this study need to be
contextualized.

In field experiments, the cost for imaging (e.g. vector and sensors) was calculated over the
whole lifetime of the considered device, taking into account the number of imaged plots per
year (number of days of use per year x number of plots measured every day), and the expected
lifetime of the considered device (in years). The investment cost is therefore expressed per
plot.day per year. The number of days of use per year differs between techniques, and varies
between sites with the frequency of weather limitations. For instance, this number is higher for
automated ground vehicles with active imaging assisted with artificial light (which can be used
even in very cloudy or night conditions) than for hand-held ground vehicles (limited by light
intensity because of passive imaging) and UAVs (limited by weather constraints, in particular
wind, rain, and light because of passive imaging). This results in costs relative to that of the
automated ground vehicle of 1.00, 0.83 and 0.67, respectively for an automated ground vehicle,
a hand-held ground vehicle and a UAV (Table 1). The costs also depend on the local demand for
the selected device: the investment cost per plot.day per year was calculated as higher if the
use of the device was limited due to low demand (scenario 2 in Table 1) than if the device was
used at full capacity (scenario 1). Additionally, the calculations shown in Table 1 also depend on
the expected lifetime of the considered device, which is higher for a ground vehicle than for a
UAV. Sensors were considered as having a shorter expected duration than vehicles because of
obsolescence. The labor cost was calculated by dividing the annual cost (220 working days per
year) by the number of days required for the considered operation and the number of
microplots to be sampled per year. The same calculations were considered for a robotized platform, expressed per plant.day. In the case presented in Table 1, the platform was considered as being used in three experiments per year, with a 90-day duration each.

The above information was then used for calculating the cost of a typical experiment (Table 2), either in a field platform with 1,700 microplots (e.g. 284 genotypes, 2 treatments and 3 replicates) and 10 days of measurement to monitor the crop cycle, or in a platform with 1,700 plants over 90 days. The costs for plant handling, for image capture, image analysis, data analysis itself and data storage considered in the analysis are presented in Table 2. Data in Tables 1 and 2 are presented below.

## 2.2 A high cost for plant management

Phenotyping is, by definition, associated with a field, a greenhouse or a growth chamber in which experiments are carried out. Field phenotyping involves a cost of typically $30 to $50 USD per microplot for one experiment, resulting in $68K USD for a typical experiment involving 1,700 microplots necessary for genetic analyses (Table 2). This price is used internally or externally by many breeding companies and includes the cost of hiring the field, plant management, irrigation and harvest. Greenhouse experiments are also expensive, with a typical investment of one million dollars for a greenhouse equipped with climatic control and surrounding facilities allowing compost management, potting and cleaning. Another million is required for the robots associated with the handling of the thousands of plants involved in genetic analyses, including imaging cabins, watering and weighing stations and conveyors. With the hypothesis of a given equipment used for 15 years with three experiments per year, this investment results in a cost of $67K USD for an experiment handling 1,700 plants, to which one adds a cost of $5K USD for electricity and potting compost. The cost per unit sample (microplot or plant) is therefore similar to experiments either in the field or in a robotized platform (Table 2). Interestingly, some platforms are in open air [62], thereby avoiding the cost of a greenhouse. This considerably decreases experimental costs, provided that climatic conditions at the dedicated site allow several experiments per year in open air; otherwise this approach could result in a high cost per experiment if only one experiment can be
accomplished per year. Overall, the high price per microplot in the field or per plant in the greenhouse suggests that phenotyping experiments are expensive *per se* before any phenotypic analyses are carried out.

**2.3 Investing in an appropriate environmental characterization results in comparatively low cost for a high return.**

The cost of environmental sensors has decreased rapidly (see section 1.4): climate sensors for temperature and humidity normally cost less than $5 USD per unit. Commercial devices can provide, for a few thousand dollars, hourly measurements of the main environmental variables necessary to characterize an experiment site, including light, air temperature, relative humidity, rainfall, and wind speed. Soil water potential can also be characterized for a few hundred dollars with tensiometers, and soil water content for a few thousand dollars with capacitive sensors. In the calculations presented in Tables 1 and 2, this investment results in a cost of less than $10K USD per field, with an assumption that the installed devices can last for about four years. An appropriate environmental characterization is therefore a cheap investment compared with plant management. Importantly, it allows joint analyses of several experiments both in the field and greenhouse, thereby improving one's ability to analyze datasets based on environmental scenarios or regression analyses [53,63]. Most breeding companies have now invested in this domain. Their feedback (personal communication), consistent with our perception, is that the major cost associated with environmental characterization is manpower because sensors have to be installed, then checked regularly and datasets need to be collected and then analyzed by semi-automated methods. In particular, for detection of outlier dates or sites, extra human costs are inevitable when many sensors are deployed under natural conditions.

**2.4 Imaging costs: a trade-off between investment and labor costs**

Imaging costs reported in Table 1 include the cost of the vector (e.g. manual measurements, UAV, ground vehicles), imaging hardware and associated software. These costs can range from a few dollars, in case of a person carrying a cell phone equipped with an imaging software, to hundreds of thousand dollars for a fully-equipped ground vehicle.
2.4.1 The choice of a vehicle mostly depends on the demand for microplots per year.

Portable devices have shown their ability to collect plant images in the field but their throughput is low and they require experienced specialists (see Section 1.1). This limits their application to relatively infrequent phenotyping for decision making or characterization of outlier genotypes. We have therefore not considered them in the calculations of Tables 1 and 2, because they respond to a different use in relation to the costs associated with high-throughput phenotyping.

UAVs are relatively cheap (a few thousand dollars) and can cover typically 4,000 microplots per day in 2-3 flights, resulting in a low-cost investment per plot.day. However, their expected lifespan is typically two years and their use is limited by weather conditions such as rainfall, wind and cloud coverage. Significant costs for insurance may occur in some countries. The manpower costs may be high in some countries due to civil aviation rules requiring authorizations and permits, leading to a cost of tens of thousands of dollars for training at least three persons per site. A calculation based on a throughput of 4,000 microplots per day, 40 available days per year, a lifetime of two years and personnel costs, still results in the lower cost compared with other vehicles ($0.29 USD per plot.day per year, scenario 1 Table 1). This cost is increased to $0.98 USD per plot.day per year in case of a lower demand of only 4,000 microplots per year (scenario 2, Table 1).

Hand-held ground vehicles have a cost of a few ten thousand dollars, excluding sensors. They can reach a throughput of around 100 microplots per hour. However, this approach struggles if aiming at measuring thousands of plots with high frequency. Indeed, it requires well-trained personnel who can manage the device, but who also accepts to push it for weeks during key developmental stages, sometimes in bad weather conditions. This can cause difficulties in the management of the personnel. We have considered a throughput of 800 microplots per day over 50 days per year, which is probably a maximum in many countries but can be extended in others. The corresponding cost in Table 1 is $0.98 USD per microplot.day per year. This cost is valid in the two hypotheses for demand in Table 1, because this method is associated with a lower throughput than UAVs.
Automated ground vehicles can be used over a larger number of days than hand-held ground vehicles and UAVs, calculated as 60 days per year in Table 1 (5 months, 12 days per month). This can be increased in case of fully automated vehicles equipped with active imaging (with autonomous lighting), which allows their use in any conditions including during the night. Their investment cost is high and essentially depends on the plant species used in experiments. For instance, a vehicle allowing imaging cereal crops with 60 cm height grown in rows can lead to an investment of typically $300K USD, but the investment increases if the vehicle must also be used for phenotyping tall species such as sorghum or maize, and/or crops that are not grown in rows such as canola (typically $500K USD). Taking into account the total investment, a throughput of 1,200 microplots per day, a lifetime of 20 years and the personnel costs, the cost is $1.02 USD per microplot.day per year in scenario 1 with a fully occupied usage, but will increase to $1.67 USD per microplot.day per year in scenario 2 with a limited demand.

Hence, the investment cost corresponding to vectors largely depends on the use of the chosen vector. For instance, robotized and hand-held ground vehicles result in similar costs if they are used to their maximum potential (i.e. a high demand), whereas the robotized ground vehicle is the most expensive option in a scenario with a limited demand. Similarly, UAVs appear to be a low-cost option in the scenario with a high demand, whereas costs of UAVs and ground vehicles are higher with a lower demand. An alternative solution for UAVs might be to rely on specialized companies that carry out measurements. However, the economic models for such services in phenotyping experiments are not yet stabilized.

### 2.4.2 The cost of imaging devices is similar to those of vehicles that carry sensors

The costs of cameras (several hundred dollars per unit), portable multi-spectral devices ($5-10K USD), and mobile LiDAR ($10-200K USD, depending on the resolution) are also high. The lifespan of multi-spectral sensors and LiDAR can be several years, but they have been limited to four years in Table 1 because of obsolescence. Personnel costs result in around $0.2-0.4 USD per microplot.day in European conditions, which can vary due to the frequency of phenotyping, selected imaging sensors, and associated training costs. On these bases, the cost of imaging was similar to ground vehicles, but much higher than UAVs.
## 2.5 Costs of typical experiments

The remaining costs need to be calculated for a typical experiment. We have considered experiments with 1,700 microplots in the field or with 1,700 plants in a robotized platform, together with the costs for plant handling as described in section 2.2, image capture in section 2.4, plus the costs of image analysis, data analysis and data storage presented below.

### 2.5.1 Image analysis: a tradeoff between investment in automated workflows and day-to-day labor costs.

With the advances in computer vision algorithms and machine learning based classification methods [5,64,65], many image analysis tasks can be accomplished automatically in a high-throughput fashion. A tradeoff therefore exists between the time dedicated to the development of imaging pipelines and that dedicated to day-to-day image analysis. Several public packages are under development and will hopefully relieve the bottleneck of image analysis [66–68]. This is already largely the case in automated phenotyping platforms, in which routine traits (e.g. plant volume, area or height) are extracted automatically in real time [69–71]; however, sound automatic workflows remain to be required for image series acquired by UAVs or ground vehicles. In both cases, the design of a specific pipeline can result in a cost of nearly $250-500K USD, if the pipeline is aimed at being sufficiently flexible for different types of users. Much cheaper data acquisition tools are commercially available, designed by companies or plant research laboratories. However, they are often proprietary, designed for specific requests and hence not flexible enough for wider applications. An interesting alternative is that public consortia develop and release flexible analytic workflows, which can then be used and continuously developed by the scientific community through an ‘open science and open source’ approach. This is currently carried out by different consortia.

A cost tradeoff also exists between the quality of images and the time for image analysis. For example, if a standard imaging protocol has not been properly conveyed to end-users (e.g. how to ensure lighting condition and image clarity, how to minimize color distortion, and how to select regions of interest), extra computational work is required to improve the quality of raw data captured by low-cost devices, different formats of raw data might require ongoing
licensing or extra fees to carry out trait analysis as well as continued maintenance for future references.

The costs in Table 2 are based on the hypothesis of existing workflows and therefore do not consider the cost of their development. With this hypothesis, they still represent 10-20% of the cost of image capture. As stated above, this cost increases by hundreds of thousands of dollars if the cost of developing workflows is taken into account. It is also considerably higher if image analysis is performed manually.

### 2.5.2 High costs for data analysis resulting in the identification of traits

The datasets resulting from phenotyping projects are difficult to analyze because they are voluminous, complex, heterogeneous, plagued with errors and only can be handled with up-to-date scientific and mathematical tools. For example, a recent project (EU DROPS) required four full-time PhD students, engineers or post-docs, three technicians and two permanent scientists for four years to conduct data analysis related tasks. This involved compiling and cleaning the datasets collected in fields and greenhouse experiments, designing novel tools for extracting traits from the raw data, and performing cross-scale analyses and genetic analysis. Overall, this procedure recorded a cost of about half a million dollars, i.e. about the same amount dedicated to image analysis in the hypotheses of Table 2.

A tradeoff exists between the time dedicated for data capture and analysis. Currently, many phenotyping projects rely on analytic software solutions that are either customized for specific hardware or based on proprietary or specialized software solutions. Similarly, data collected with cost-effective phenotyping approaches are often analyzed manually, which is time consuming, prone to errors and expensive due to additional human costs. Developing workflows with a reproducible data analysis strategy therefore corresponds to a high extra-cost for individual experiments, but it can be considered as a good investment at the level of a broader scientific community, because, in this way, data can be shared, re-used and re-analyzed.
Overall, the cost of data analysis is the most underestimated part of many phenotyping projects. In the same way as for image analysis, data analysis costs presented in Table 2 are based on the availability of existing workflows. They considerably increase if workflows need to be developed during the projects, or the whole analysis is performed manually. Based on these hypotheses, the costs required for estimating trait values are similar to those of image analysis. Together, costs of image and data analysis represent 30-200% of the cost of image capture, a factor that is rarely considered for the overall costs of phenotyping.

### 2.5.3 Costs associated with data storage and organization ensure the possibility of reusing datasets

The datasets collected above carry more information than any group can handle alone. It is therefore vital for the plant science community to ensure that datasets can be managed in a way that they can be accessed and re-analyzed by scientists that have not been involved in the data collection. By doing that, researchers should be able to trace the history of plants, re-analyze sensor- and image-based datasets with existing or new methods and check sensors in case of inconsistencies. This requires information systems capable of collecting, managing, and presenting thousands of data points and images collected in multiple experiments, together with necessary metadata (FAIR standard: findable, accessible, interoperable and reusable). Such information systems are based on elaborate protocols to describe content and format of phenotypic information [56,72], as well as a standardized description of all involved objects (i.e. plants, organs, sensors, phenotyping facilities) via ontologies [73,74].

The cost for elaborating such information systems involves tens of person-months of computer scientists. As stated in earlier paragraphs, this requires an effort at the level of international consortium. The costs in the hypotheses of Table 2 are based on a pre-existing information system and only consider the cost of data storage ($32 USD per terabyte per year).

## III An unexpected structure of costs has large consequences on conclusions about cost effectiveness
An overall inspection of Table 2 results in a view of phenotyping costs that largely differs from an initial intuition that one might have. In the hypotheses considered in Table 2:

- The cost for handling microplots or plants is by far the highest and is similar in the field and in robotized platforms. The former was based on current costs in most breeding companies; the latter was considered the cost of the greenhouse and of the robot. The cost of microplot or plant handling represents 65-77% of the total cost of phenotyping, across types of vehicles, hypotheses or location of experiments in a field or a robotized greenhouse.

- The labor cost represents a large proportion of the total cost, from 30% to 100% of the cost of vehicles and sensors for data analysis, plus the costs associated with image capture itself. As stated above, these costs are under-estimated in Table 2, because they assume that pipelines already exist for image analysis, trait measurements and associated information systems. These costs would considerably increase if the development of pipelines was taken into account, or if all the data processing was considered as manually accomplished.

- Investment itself represents only 10-20% of the total of phenotyping costs, whereas most discussions on costs focus on investment.

Hence, phenotyping may be one of the few cases in which intuition about cost-effectiveness is not appropriate because (i) it tends to considerably under-estimate personnel and structural costs, (ii) it may lead to choosing tools that are immediately usable and relatively affordable; however, the examples shown previously indicate that heavier investment could result in a more efficient chain for extracting meaningful information.

Other non-intuitive facts also emerge from Table 2 through comparing experiments in robotized platforms in the greenhouse or in the field with imaging based on different vehicles. First, the costs of experiments in a robotized platform are similar to those in the field (Table 2). Second, the total costs of phenotyping do not greatly differ with the choice of vehicles in field experiments. As discussed above, the optimum choice in terms of cost depends on scenarios: for a high demand of phenotyping, the three vehicles result in similar costs, with a slightly lower cost for UAVs; costs increase with a lower demand for the three vehicles, with a slight cost advantage for hand-held ground vehicles. However, these differences are small and context
dependent, so a pure cost analysis does not result in an obvious choice between field and platform experiments, or for one of the three considered vehicles.

Overall, the above shows that the cost of phenotyping experiment is high if all related costs are considered (Table 2). However, this statement needs to be contextualized. In some cases, light phenotyping represents a small marginal cost of an operation or experiment that is carried out, for instance, when a farmer needs to take an adequate decision or a breeder needs to keep track of some simple operations, the cost of crop management is not considered and the need for data analysis and storage is limited. Mobile phones or inexpensive UAV flights for light phenotyping are therefore highly valuable in these cases. In the other extreme, a phenotyping project aiming to characterize hundreds of genotypes needs to take all costs into account, resulting in a high overall cost for plant handling together with a high manpower cost for data analysis and data storage. Investment in vehicles and imaging devices therefore represent a limited proportion of the total cost. In this case, the choice of vehicle (UAV vs ground vehicle), location, and experiments (field vs platform) should be taken into consideration together with other factors, i.e. the nature and the precision of the desired traits as well as the constraints linked to the management of personnel.

Numerous trade-offs have been presented here between investment and operational costs, for example, the choice of vehicles, imaging techniques, or image analysis workflows. Hence, 'affordable phenotyping', considered as the way to obtain a maximum of images in a minimum time frame with low investment costs, may be counter-productive in many cases. Similarly, the development of analysis pipelines represents a large investment but often lead to cheaper and more reproducible datasets than manual or tailored analysis in the long term. These trade-offs depend on local conditions, such as the availability, the cost of manpower, and the number of days, during which a given device can be used per year due to climatic or other constraints. None of the devices or techniques discussed above can be considered as cost-effective or cost-ineffective *per se*, as nearly all of them can be considered adequate for specific tasks under defined conditions and ineffective in other circumstances.

It is therefore essential that costs are reasoned in relation to (i) the precision, repeatability and heritability required in a given phenotyping task (ii) local personnel costs (training, data
transfer, data calibration, data analysis and data management) that greatly vary between projects and countries, (iii) the cost per unit plot or trait, which can largely differ between methods depending on local climatic and economic conditions. If all of these elements are taken into account, 'cost effective' phenotyping may in some cases involve low investment ('affordable phenotyping'), and in other cases involve an initial high investment that results in low operational costs together with high quality outcomes. Finally, for breeding purpose, phenotyping costs also need to be analyzed in terms of their contribution to the rate of genetic gain. Direct ratios cannot be established at this stage because of uncertainties about the scalability of measured traits towards yield in the absence of case studies combining phenomics, modelling and genomic prediction [75]. However, one piece of equipment and associated methods in phenomics have already shown their contribution to breeding: it has been observed here that the investment in sensor networks for environmental characterization has a clear value for interpretation of the genotype x environment interaction, and for weighing the investment in specific breeding programs in relation to the frequency of corresponding target populations of environments.

## Acknowledgements

Authors are grateful to the phenotyping communities in the French and UK phenotyping infrastructures for cost evaluation. FT, CW and FB thank the ANR-PIA project PHENOME FPPN (ANR-11-INBS-0012) and EPPN<sup>2020</sup> (UE H2020 grant agreement No 731013) for partly funding this work. JZ, DR, AB and JB are funded by the Biotechnology and Biological Sciences Research Council (BBSRC), Core Strategic Programme Grant (BB/CSP17270/1) at EI, BBSRC's Designing Future Wheat Strategic Programme (BB/P016855/1) to G. Moore and (BBS/E/T/000PR9785) to JZ.

## References

[1] F. Tardieu, L. Cabrera-Bosquet, T. Pridmore, M. Bennett, Plant Phenomics, From Sensors to Knowledge, Curr. Biol. 27 (2017) R770–R783. doi:10.1016/j.cub.2017.05.055.
[2] R.T. Furbank, M. Tester, Phenomics--technologies to relieve the phenotyping bottleneck., Trends Plant Sci. 16 (2011) 635-44. doi:10.1016/j.tplants.2011.09.005.

[3] F. Fiorani, U. Schurr, Future scenarios for plant phenotyping., Annu. Rev. Plant Biol. 64 (2013) 267-91. doi:10.1146/annurev-arplant-050312-120137.

[4] D. Deery, J. Jimenez-Berni, H. Jones, X. Sirault, R. Furbank, Proximal Remote Sensing Buggies and Potential Applications for Field-Based Phenotyping, 2014. doi:10.3390/agronomy4030349.

[5] D. Rousseau, H. Dee, T. Pridmore, Imaging Methods for Phenotyping of Plant Traits, in: S. Kumar (Ed.), Phenomics Crop Plants Trends, Options Limitations, Springer India, 2015: pp. 61-74. doi:DOI 10.1007/978-81-322-2226-2_5.

[6] L. Busemeyer, D. Mentrup, K. Möller, E. Wunder, K. Alheit, V. Hahn, H.P. Maurer, J.C. Reif, T. Würschum, J. Müller, F. Rahe, A. Ruckelshausen, Breedvision - A multi-sensor platform for non-destructive field-based phenotyping in plant breeding, Sensors (Switzerland). 13 (2013) 2830-2847. doi:10.3390/s130302830.

[7] G. Bitella, R. Rossi, R. Bochicchio, M. Perniola, M. Amato, A novel low-cost open-hardware platform for monitoring soil water content and multiple soil-air-vegetation parameters, Sensors (Switzerland). 14 (2014) 19639-19659. doi:10.3390/s141019639.

[8] M.J. Aitkenhead, G.J. Gaskin, N. Lafouge, C. Hawes, Phylis: A low-cost portable visible range spectrometer for soil and plants, Sensors (Switzerland). 17 (2017). doi:10.3390/s17010099.

[9] A. Pask, J. Pietragalla, D. Mullan, M. Reynolds, Physiological Breeding II: A Field Guide to Wheat Phenotyping, CIMMYT, Mexico, 2012. doi:10.1017/CBO9781107415324.004.

[10] J. Barker, N. Zhang, J. Sharon, R. Steeves, X. Wang, Y. Wei, J. Poland, Development of a field-based high-throughput mobile phenotyping platform, Comput. Electron. Agric. 122 (2016) 74-85. doi:10.1016/j.compag.2016.01.017.

[11] H.M. Easlon, A.J. Bloom, Easy Leaf Area: Automated Digital Image Analysis for Rapid and Accurate Measurement of Leaf Area, Appl. Plant Sci. 2 (2014) 1400033.
doi:10.3732/apps.1400033.

[12] D. Kelly, A. Vatsa, W. Mayham, L. Ng, A. Thompson, T. Kazic, An opinion on imaging challenges in phenotyping field crops, Mach. Vis. Appl. 27 (2016) 681–694.
doi:10.1007/s00138-015-0728-4.

[13] Y. Intaravannea, S. Sumriddetchkajorn, Android-based rice leaf color analyzer for estimating the needed amount of nitrogen fertilizer, Comput. Electron. Agric. 116 (2015) 228–233. doi:https://doi.org/10.1016/j.compag.2015.07.005.

[14] E. Komyshev, M. Genaev, D. Afonnikov, Evaluation of the SeedCounter, A Mobile Application for Grain Phenotyping, Front. Plant Sci. 7 (2017) 1–9.
doi:10.3389/fpls.2016.01990.

[15] R. Confalonieri, M. Foi, R. Casa, S. Aquaro, E. Tona, M. Peterle, A. Boldini, G. De Carli, A. Ferrari, G. Finotto, T. Guarneri, V. Manzoni, E. Movedi, A. Nisoli, L. Paleari, I. Radici, M. Suardi, D. Veronesi, S. Bregaglio, G. Cappelli, M.E. Chiodini, P. Dominoni, C. Francone, N. Frasso, T. Stella, M. Acutis, Development of an app for estimating leaf area index using a smartphone. Trueness and precision determination and comparison with other indirect methods, Comput. Electron. Agric. 96 (2013) 67–74.
doi:10.1016/J.COMPAG.2013.04.019.

[16] A. Burkart, V.L. Hecht, T. Kraska, R. U., Phenological analysis of unmanned aerial vehicle based time series of barley imagery with high temporal resolution, Precis. Agric. (2017) 1–13. doi:oi.org/10.1007/s11119-017-9504-y.

[17] S. Sankaran, L.R. Khot, C.Z. Espinoza, S. Jarolmasjed, V.R. Sathuvalli, G.J. Vandemark, P.N. Miklas, A.H. Carter, M.O. Pumphrey, R.R.N. Knowles, M.J. Pavek, Low-altitude, high-resolution aerial imaging systems for row and field crop phenotyping: A review, Eur. J. Agron. 70 (2015) 112–123. doi:10.1016/j.eja.2015.07.004.

[18] T. Duan, B. Zheng, W. Guo, S. Ninomiya, Y. Guo, S.C. Chapman, T.D. A, B.Z. A, W.G. C, S.N. C, Y.G. B, S.C.C. A, Comparison of ground cover estimates from experiment plots in cotton, sorghum and sugarcane based on images and ortho-mosaics captured by UAV,
Funct. Plant Biol. (2016). doi:10.1071/FP16123.

[19] N. Brisson, P. Gate, D. Gouache, G. Charmet, F.X. Oury, F. Huard, Why are wheat yields stagnating in Europe? A comprehensive data analysis for France, F. Crop. Res. 119 (2010) 201–212. doi:10.1016/j.fcr.2010.07.012.

[20] V.O. Sadras, R.A. Richards, Improvement of crop yield in dry environments: Benchmarks, levels of organisation and the role of nitrogen, J. Exp. Bot. 65 (2014) 1981–1995. doi:10.1093/jxb/eru061.

[21] M. Cooper, F.A. van Eeuwijk, G.L. Hammer, D.W. Podlich, C. Messina, Modeling QTL for complex traits: detection and context for plant breeding, Curr. Opin. Plant Biol. 12 (2009) 231–240. doi:10.1016/j.pbi.2009.01.006.

[22] J. Zhou, C. Applegate, A.D. Alonso, D. Reynolds, S. Orford, M. Mackiewicz, S. Griffiths, S. Penfield, N. Pullen, Leaf-GP: An Open and Automated Software Application for Measuring Growth Phenotypes for Arabidopsis and Wheat, Plant Methods. 13 (2017) 1–30. doi:https://doi.org/10.1101/180083.

[23] W. Guo, T. Fukatsu, S. Ninomiya, Automated characterization of flowering dynamics in rice using field-acquired time-series RGB images, Plant Methods. 11 (2015). doi:10.1186/s13007-015-0047-9.

[24] M. Reynolds, P. Langridge, Physiological breeding, Curr. Opin. Plant Biol. 31 (2016) 162–171. doi:10.1016/j.pbi.2016.04.005.

[25] F. Tardieu, T. Simonneau, B. Muller, The Physiological Basis of Drought Tolerance in Crop Plants: A Scenario-Dependent Probabilistic Approach, Annu. Rev. Plant Biol. 69 (2018) 733–759. doi:10.1146/annurev-arplant-042817-040218.

[26] J.C. hristian Rose, S. Paulus, H. Kuhlmann, Accuracy analysis of a multi-view stereo approach for phenotyping of tomato plants at the organ level, Sensors (Basel). 15 (2015) 9651–9665. doi:10.3390/s150509651.

[27] M. Müller-Linow, F. Pinto-Espinosa, H. Scharr, U. Rascher, The leaf angle distribution of natural plant populations: assessing the canopy with a novel software tool, Plant
Methods. 11 (2015) 11. doi:10.1186/s13007-015-0052-z.

[28] R. Khanna, J. Rehder, M. Martin, E. Galceran, R. Siegwart, Studying Phenotypic Variability in Crops using a Hand-held Sensor Platform, IROS Work. Agri-Food Robot. (2015).

[29] J.L. Crain, Y. Wei, J. Barker, S.M. Thompson, P.D. Alderman, M. Reynolds, N. Zhang, J. Poland, Development and deployment of a portable field phenotyping platform, Crop Sci. 56 (2016) 965–975. doi:10.2135/cropsci2015.05.0290.

[30] S. Kuhlgert, G. Austic, R. Zegarac, I. Osei-Bonsu, D. Hoh, M.I. Chilvers, M.G. Roth, K. Bi, D. TerAvest, P. Weebadde, D.M. Kramer, MultispeQ Beta: a tool for large-scale plant phenotyping connected to the open PhotosynQ network., R. Soc. Open Sci. 3 (2016) 160592. doi:10.1098/rsos.160592.

[31] I. Jonckheere, S. Fleck, K. Nackaerts, B. Muys, P. Coppin, M. Weiss, F. Baret, Review of methods for in situ leaf area index determination: Part I. Theories, sensors and hemispherical photography, Agric. For. Meteorol. 121 (2004) 19–35. doi:10.1016/J.AGRFORMET.2003.08.027.

[32] M. Weiss, F. Baret, G.J. Smith, I. Jonckheere, P. Coppin, Review of methods for in situ leaf area index (LAI) determination, part II: Estimation of LAI, errors and sampling., Agric. For. Meteorol. 121 (2004) 37–53. doi:10.1016/j.agrformet.2003.08.001.

[33] J. Berni, P.J. Zarco-Tejada, L. Suarez, E. Fereres, Thermal and Narrowband Multispectral Remote Sensing for Vegetation Monitoring From an Unmanned Aerial Vehicle, IEEE Trans. Geosci. Remote Sens. 47 (2009) 722–738. doi:10.1109/TGRS.2008.2010457.

[34] H.F. Phenotyping, M.F. Dreccer, E. Holland, B. Zheng, T.J. Ling, Pheno-Copter: A Low-Altitude, Autonomous Remote-Sensing Robotic Helicopter for High-Throughput Field-Based Phenotyping, Agronomy. 4 (2014) 279–301. doi:10.3390/agronomy4020279.

[35] F. Baret, S. Madec, K. Irfan, J. Lopez, A. Comar, D. Dutartre, S. Praud, M.H. Tixier, Leaf rolling in maize crops : from leaf scoring to canopy level measurements for phenotyping , BioRxiv. (2017) 1–35.

[36] D.M. Deery, G.J. Rebetzke, J.A. Jimenez-Berni, R.A. James, A.G. Condon, W.D. Bovill, P.
Hutchinson, J. Scarrow, R. Davy, R.T. Furbank, Methodology for High-Throughput Field Phenotyping of Canopy Temperature Using Airborne Thermography, Front. Plant Sci. 7 (2016) 1–13. doi:10.3389/fpls.2016.01808.

[37] X. Jin, S. Liu, F. Baret, M. Hemerlé, A. Comar, Estimates of plant density of wheat crops at emergence from very low altitude UAV imagery, Remote Sens. Environ. 198 (2017) 105–114. doi:10.1016/J.RSE.2017.06.007.

[38] A. Gracia-Romero, S.C. Kefauver, O. Vergara-Diaz, M.A. Zaman-Allah, B.M. Prasanna, J.E. Cairns, J.L. Araus, Comparative performance of ground versus aerially assessed RGB and multispectral indices for early-growth evaluation of maize performance under phosphorus fertilization I n r e v i e w, Front. Plant Sci. 8 (2017) 1–13. doi:10.3389/fpls.2017.02004.

[39] F.H. Holman, A.B. Riche, A. Michalski, M. Castle, M.J. Wooster, M.J. Hawkesford, High throughput field phenotyping of wheat plant height and growth rate in field plot trials using UAV based remote sensing, Remote Sens. 8 (2016). doi:10.3390/rs8121031.

[40] L. Malambo, S.C. Popescu, S.C. Murray, E. Putman, N.A. Pugh, D.W. Horne, G. Richardson, R. Sheridan, W.L. Rooney, R. Avant, M. Vidrine, B. McCutchen, D. Baltensperger, M. Bishop, Multitemporal field-based plant height estimation using 3D point clouds generated from small unmanned aerial systems high-resolution imagery, Int. J. Appl. Earth Obs. Geoinf. 64 (2018) 31–42. doi:10.1016/j.jag.2017.08.014.

[41] K. Watanabe, W. Guo, K. Arai, H. Takanashi, H. Kajiya-Kanegae, M. Kobayashi, K. Yano, T. Tokunaga, T. Fujiwara, N. Tsutsumi, H. Iwata, High-Throughput Phenotyping of Sorghum Plant Height Using an Unmanned Aerial Vehicle and Its Application to Genomic Prediction Modeling, Front. Plant Sci. 8 (2017) 1–11. doi:10.3389/fpls.2017.00421.

[42] W. Li, Z. Niu, H. Chen, D. Li, M. Wu, W. Zhao, Remote estimation of canopy height and aboveground biomass of maize using high-resolution stereo images from a low-cost unmanned aerial vehicle system, Ecol. Indic. 67 (2016) 637–648. doi:10.1016/j.ecolind.2016.03.036.
[43] Y. Lin, LiDAR: An important tool for next-generation phenotyping technology of high potential for plant phenomics?, Comput. Electron. Agric. 119 (2015) 61–73.
doi:10.1016/j.compag.2015.10.011.

[44] A. Verger, N. Vigneau, C. Che??ron, J.M. Gilliot, A. Comar, F. Baret, Green area index from an unmanned aerial system over wheat and rapeseed crops, Remote Sens. Environ. 152 (2014) 654–664. doi:10.1016/j.rse.2014.06.006.

[45] M. Ishihara, Y. Inoue, K. Ono, M. Shimizu, S. Matsuura, The impact of sunlight conditions on the consistency of vegetation indices in croplands-Effective usage of vegetation indices from continuous ground-based spectral measurements, Remote Sens. 7 (2015) 14079–14098. doi:10.3390/rs71014079.

[46] B. Stark, T. Zhao, Y. Chen, An analysis of the effect of the bidirectional reflectance distribution function on remote sensing imagery accuracy from Small Unmanned Aircraft Systems, in: 2016 Int. Conf. Unmanned Aircr. Syst., ICUAS, Arlington, VA, USA, 2016. doi:10.1109/ICUAS.2016.7502566.

[47] J.W. White, M.M. Conley, A flexible, low-cost cart for proximal sensing, Crop Sci. 53 (2013) 1646–1649. doi:10.2135/cropsci2013.01.0054.

[48] M.P. Cendrero-Mateo, O. Muller, H. Albrecht, A. Burkart, S. Gatzke, B. Janssen, B. Keller, N. Körber, T. Kraska, S. Matsubara, J. Li, M. Müller-Linow, R. Pieruschka, F. Pinto, P. Rischbeck, A. Schickling, A. Steier, M. Watt, U. Schurr, U. Rascher, Field Phenotyping: Challenges and Opportunities, Terr. Ecosyst. Res. Infrastructures. (2017) 53–80. doi:10.1201/9781315368252-4.

[49] J. Svensgaard, T. Roitsch, S. Christensen, Development of a Mobile Multispectral Imaging Platform for Precise Field Phenotyping, Agronomy. 4 (2014) 322–336.
doi:10.3390/agronomy4030322.

[50] K. Jensen, S.H. Nielsen, R.N. Jorgensen, A. Bogild, N.J. Jacobsen, O.J. Jorgensen, C.L. Jaeger-Hansen, A LOW COST, MODULAR ROBOTICS TOOL CARRIER FOR PRECISION AGRICULTURE RESEARCH K. Jensen, S.H. Nielsen, and R.N. Jørgensen, Int. Conf. Precis.
Agric. (2012).

[51] L. Grimstad, P. From, The Thorvald II Agricultural Robotic System, Robotics. 6 (2017) 24.
doi:10.3390/robotics6040024.

[52] A. Shafiekhani, S. Kadam, F.B. Fritschi, G.N. Desouza, Vinobot and vinoculer: Two robotic platforms for high-throughput field phenotyping, Sensors. 17 (2017) 1–23.
doi:10.3390/s17010214.

[53] E. Millet, C. Welcker, W. Kruijer, S. Negro, S. Nicolas, S. Praud, N. Ranc, T. Presterl, R. Tuberosa, Z. Bedo, X. Draye, B. Usadel, A. Charcosset, F. van Eeuwijk, F. Tardieu, A. Coupel-Ledru, C. Bauland, Genome-wide analysis of yield in Europe: allelic effects as functions of drought and heat scenarios, Plant Physiol. 172 (2016) pp.00621.2016.
doi:10.1104/pp.16.00621.

[54] L. Cabrera-Bosquet, C. Fournier, N. Brichet, C. Welcker, B. Suard, F. Tardieu, High-throughput estimation of incident light, light interception and radiation-use efficiency of thousands of plants in a phenotyping platform, New Phytol. 212 (2016) 269–281.
doi:10.1111/nph.14027.

[55] G. Villarrubia, J.F. De Paz, D.H. De La Iglesia, J. Bajo, Combining multi-agent systems and wireless sensor networks for monitoring crop irrigation, Sensors (Switzerland). 17 (2017).
doi:10.3390/s17081775.

[56] J. Zhou, D. Reynolds, D. Websdale, T. Le Cornu, O. Gonzalez-Navarro, C. Lister, S. Orford, S. Laycock, G. Finlayson, T. Stitt, M. Clark, M. Bevan, S. Griffiths, CropQuant: An automated and scalable field phenotyping platform for crop monitoring and trait measurements to facilitate breeding and digital agriculture, BioRxiv. (2017) 1–17.
doi:https://doi.org/10.1101/161547.

[57] K. Chenu, M. Cooper, G.L. Hammer, K.L. Mathews, M.F. Dreccer, S.C. Chapman, Environment characterization as an aid to wheat improvement: Interpreting genotype-environment interactions by modelling water-deficit patterns in North-Eastern Australia, J. Exp. Bot. 62 (2011) 1743–1755. doi:10.1093/jxb/erq459.
[58] M.T. Harrison, F. Tardieu, Z. Dong, C.D. Messina, G.L. Hammer, Characterizing drought stress and trait influence on maize yield under current and future conditions, Glob. Chang. Biol. 20 (2014) 867–878. doi:10.1111/gcb.12381.

[59] Y. Xu, Envirotyping for deciphering environmental impacts on crop plants, Theor. Appl. Genet. 129 (2016) 653–673. doi:10.1007/s00122-016-2691-5.

[60] M. Cooper, C. Gho, R. Leafgren, T. Tang, C. Messina, Breeding drought-tolerant maize hybrids for the US corn-belt: Discovery to product, J. Exp. Bot. 65 (2014) 6191–6194. doi:10.1093/jxb/eru064.

[61] A. Murigneux, F. HENRIOT, M. PERSONNE, M. RENAULT, C. DELLUC, R. DEBEUF, Method for determining drought tolerance in maize, PCT/EP2016/063067, 2015. https://patents.google.com/patent/WO2016198471A1/en.

[62] V. Vadez, J. Kholová, G. Hummel, U. Zhokhavets, S.K. Gupta, C.T. Hash, LeasyScan: A novel concept combining 3D imaging and lysimetry for high-throughput phenotyping of traits controlling plant water budget, J. Exp. Bot. 66 (2015) 5581–5593. doi:10.1093/jxb/erv251.

[63] S. Lacube, C. Fournier, C. Palaffre, E.J. Millet, F. Tardieu, B. Parent, Distinct controls of leaf widening and elongation by light and evaporative demand in maize, Plant Cell Environ. 40 (2017) 2017–2028.

[64] S. van der Walt, J.L. Schönberger, J. Nunez-Iglesias, F. Boulogne, J.D. Warner, N. Yager, E. Gouillart, T. Yu, Scikit-image: image processing in Python., PeerJ. 2 (2014) 1–18. doi:10.7717/peerj.453.

[65] J. Howse, OpenCV Computer Vision with Python, 1st ed., Packt Publishing Ltd., Birmingham, UK, 2013.

[66] A. Singh, B. Ganapathysubramanian, A.K. Singh, S. Sarkar, Machine Learning for High-Throughput Stress Phenotyping in Plants, Trends Plant Sci. 21 (2016) 110–124. doi:10.1016/j.tplants.2015.10.015.

[67] M.P. Pound, J.A. Atkinson, A.J. Townsend, M.H. Wilson, M. Griffiths, A.S. Jackson, A.
Bulat, G. Tzimiropoulos, D.M. Wells, E.H. Murchie, T.P. Pridmore, A.P. French, Deep machine learning provides state-of-the-art performance in image-based plant phenotyping, Gigascience. 6 (2017) 1–10. doi:10.1093/gigascience/gix083.

[68] M.P. Pound, J.A. Atkinson, D.M. Wells, T.P. Pridmore, A.P. French, Deep Learning for Multi-task Plant Phenotyping, in: IEEE Int. Conf. Comput. Vis., 2017: pp. 2055–2063. doi:10.1101/204552.

[69] C. Klukas, D. Chen, J.-M. Pape, Integrated Analysis Platform: An Open-Source Information System for High-Throughput Plant Phenotyping, Plant Physiol. 165 (2014) 506–518. doi:10.1104/pp.113.233932.

[70] M.J. Hawkesford, A. Lorence, Plant phenotyping : increasing throughput and precision at multiple scales, Funct. Plant Biol. (2017). doi:10.1071/FP.

[71] N. Kirchgessner, F. Liebisch, K. Yu, J. Pfeifer, M. Friedli, A. Hund, A. Walter, The ETH field phenotyping platform FIP: A cable-suspended multi-sensor system, Funct. Plant Biol. 44 (2017) 154–168. doi:10.1071/FP16165.

[72] C. Pradal, S. Artzet, J. Chopard, D. Dupuis, C. Fournier, M. Mielewczik, V. Nègre, P. Neveu, D. Parigot, P. Valduriez, S. Cohen-Boulakia, InfraPhenoGrid: A scientific workflow infrastructure for plant phenomics on the Grid, Futur. Gener. Comput. Syst. 67 (2017) 341–353. doi:10.1016/j.future.2016.06.002.

[73] Y.F. Li, G. Kennedy, F. Ngoran, P. Wu, J. Hunter, An ontology-centric architecture for extensible scientific data management systems, Futur. Gener. Comput. Syst. 29 (2013) 641–653. doi:10.1016/j.future.2011.06.007.

[74] H. Ćwiek-Kupczyńska, T. Altmann, D. Arend, E. Arnaud, D. Chen, G. Cornut, F. Fiorani, W. Frohmberg, A. Junker, C. Klukas, M. Lange, C. Mazurek, A. Nafissi, P. Neveu, J. van Oeveren, C. Pommier, H. Poorter, P. Rocca-Serra, S.-A. Sansone, U. Scholz, M. van Schriek, Ü. Seren, B. Usadel, S. Weise, P. Kersey, P. Krajewski, Measures for interoperability of phenotypic data: minimum information requirements and formatting, Plant Methods. 12 (2016) 44. doi:10.1186/s13007-016-0144-4.
[75] J. Crain, S. Mondal, J. Rutkoski, R.P. Singh, J. Poland, Combining High-Throughput Phenotyping and Genomic Information to Increase Prediction and Selection Accuracy in Wheat Breeding, Plant Genome. 11 (2018) 0. doi:10.3835/plantgenome2017.05.0043.
Table 1. Imaging costs involving vehicle, sensors, associated software and personnel in field experiments or in a robotized platform, for two scenarios of demand for phenotyping (offer or demand-limited) and, in the field, three categories of vehicles (vectors) carrying sensors (automated or hand-held ground vehicle or unmanned aerial vehicle (UAV). Costs are expressed in US dollars per plot.day per year (field) or plant.day per year (robotized platform), with the principles of calculations in the panel “vector”. Costs of manpower are calculated per year and plot.day or plant.day. Two scenarios are considered for field conditions: in scenario 1 (offer limited), the demand for phenotyping exceeds the capacity of the system; in scenario 2 (demand limited) the demand represents a maximum of 4000 microplots per year.

<table><thead><tr><th></th><th></th><th></th><th></th><th></th><th></th><th></th><th colspan="2">Manpower + training</th><th>Maintenance</th><th>Costing</th></tr><tr><th></th><th>Hypotheses for each scenario</th><th>Days of use year<sup>-1</sup></th><th>Throughput, µplot or plant day<sup>-1</sup></th><th>Expected duration, year</th><th>Investment k$</th><th>Investment $ per plot per day vector life</th><th>Equivalent calculation, 4 year life</th><th>$ year<sup>-1</sup></th><th>per plot day per year</th><th>$ per plot day per year</th></tr></thead><tbody><tr><td>High throughput field experiments, 'offer limited'</td><td>Limited by availability of equipment and personnel.</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr></tbody></table>
<table><tr><td>Automated ground vehicle</td><td>60</td><td>1200</td><td>20</td><td>430</td><td>0.30</td><td>0.24</td><td>19564</td><td>0.2717</td><td>15000</td><td>0.2083</td><td><strong>1.02</strong></td></tr><tr><td>Hand-held ground vehicle</td><td>50</td><td>800</td><td>15</td><td>50</td><td>0.08</td><td>0.44</td><td>15553</td><td>0.3888</td><td>3000</td><td>0.0750</td><td><strong>0.98</strong></td></tr><tr><td>UAV</td><td>40</td><td>4000</td><td>2</td><td>10</td><td>0.03</td><td>0.09</td><td>24545</td><td>0.1534</td><td>2000</td><td>0.0125</td><td><strong>0.29</strong></td></tr><tr><td>High throughput field experiments, 'demand limited'</td><td>Limited by the demand for microplot per year. 40000 μplots year<sup>-1</sup></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Automated ground vehicle</td><td>33</td><td>1200</td><td>20</td><td>430</td><td>0.54</td><td>0.44</td><td>12873</td><td>0.3218</td><td>15000</td><td>0.3750</td><td><strong>1.67</strong></td></tr><tr><td>Hand-held ground vehicle</td><td>50</td><td>800</td><td>15</td><td>50</td><td>0.08</td><td>0.44</td><td>15553</td><td>0.3888</td><td>3000</td><td>0.0750</td><td><strong>0.98</strong></td></tr><tr><td>UAV</td><td>10</td><td>4000</td><td>2</td><td>10</td><td>0.13</td><td>0.38</td><td>17018</td><td>0.4255</td><td>2000</td><td>0.0500</td><td><strong>0.98</strong></td></tr><tr><td>Robotized indoor platform</td><td>Limited by availability of equipment and personnel.</td><td>270</td><td>1700</td><td>15</td><td>1000</td><td>0.15</td><td>0.02</td><td>103618</td><td>0.2257</td><td>15000</td><td>0.0327</td><td><strong>0.42</strong></td></tr></table>
Table 2 Distribution of costs in typical experiments in the field (1,700 microplots with 10 days of observation) or robotized platforms (1700 plants with 90 days). Hypotheses are as above. The cost of microplot or plant handling represents either the current costs per plot (field) or the cost of greenhouse plus robot, together with manpower (robotized platform). Note that the cost of the robot was considered in "investment" in Table 1 but is in "plant handling" in Table 2 for easier comparison with the field. Robots are used for both plant handling and imaging in robotized platforms. 'Meas' stands for 'measurements'

<table><thead><tr><th></th><th></th><th>Cost µplots or<br>plant handling<br>k$</th><th>Image<br>capture<br>k$</th><th>Image analysis<br>k$</th><th>Trait<br>analysis<br>k$</th><th>Data<br>storage<br>10<br>years,<br>k$</th><th>Total<br>k$</th><th>%<br>investment</th></tr></thead><tbody><tr><td colspan="9"><strong>High throughput<br>field experiments,<br>'offer limited'</strong></td></tr><tr><td>Automated ground<br>vehicle</td><td>1700 µplots, 10 days<br>meas</td><td>68.0</td><td>17.4</td><td>3.5</td><td>5.3</td><td>1.5</td><td><strong>96</strong></td><td>18.2</td></tr><tr><td>Hand-held ground<br>vehicle</td><td>1700 µplots, 10 days<br>meas</td><td>68.0</td><td>16.7</td><td>5.3</td><td>7.1</td><td>0.7</td><td><strong>98</strong></td><td>17.1</td></tr><tr><td>UAV</td><td>1700 µplots, 10 days<br>meas</td><td>68.0</td><td>4.9</td><td>7.1</td><td>10.6</td><td>0.2</td><td><strong>91</strong></td><td>5.4</td></tr><tr><td colspan="9"><strong>High throughputfield<br>experiments,<br>'demand limited'</strong></td></tr><tr><td>Automated ground<br>vehicle</td><td>1700 µplots, 10 days<br>meas</td><td>68.0</td><td>28.4</td><td>3.5</td><td>5.3</td><td>1.5</td><td><strong>107</strong></td><td>26.6</td></tr><tr><td>Hand-held ground<br>vehicle</td><td>1700 µplots, 10 days<br>meas</td><td>68.0</td><td>16.7</td><td>5.3</td><td>7.1</td><td>0.7</td><td><strong>98</strong></td><td>17.1</td></tr></tbody></table>
<table><tr><td>UAV</td><td>1700 μplots, 10 days meas</td><td>68.0</td><td>16.6</td><td>7.1</td><td>10.6</td><td>0.2</td><td><b>103</b></td><td>16.2</td></tr><tr><td><b>Robotized platform</b><br>the cost of robot is in<br>the 'handling' column</td><td>1700 plants, 90 days</td><td>71.2</td><td>9.0</td><td>1.8</td><td>10.6</td><td>2.6</td><td><b>95</b></td><td>9.5</td></tr></table>