# Pl@ntNet Crops: merging citizen science observations and structured survey data to improve crop recognition for agri-food-environment applications

Marijn van Der Velde, Hervé Goëau, Pierre Bonnet, Raphaël d'Andrimont, Martin Yordanov, Antoine Affouard, Martin Claverie, Bálint Czúcz, Neija Elvekjær, Laura Martinez-Sanchez, et al.

<figure><img src="image_1.png" /></figure>

## To cite this version:

Marijn van Der Velde, Hervé Goëau, Pierre Bonnet, Raphaël d'Andrimont, Martin Yordanov, et al. Pl@ntNet Crops: merging citizen science observations and structured survey data to improve crop recognition for agri-food-environment applications. Environmental Research Letters, 2023, 18 (2), pp.025005. 10.1088/1748-9326/acadf3. hal-03965684

HAL Id: hal-03965684

https://hal.science/hal-03965684

Submitted on 31 Jan 2023

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_2.png" /></figure>
LETTER • OPEN ACCESS

# PI@ntNet Crops: merging citizen science observations and structured survey data to improve crop recognition for agri-food-environment applications

To cite this article: M van der Velde et al 2023 *Environ. Res. Lett.* **18** 025005

View the [article online](#) for updates and enhancements.

You may also like

- Combining satellite data and agricultural statistics to map grassland management intensity in Europe
Stephan Estel, Sebastian Mader, Christian Levers et al.

- Scenarios of climate adaptation potential on protected working lands from management of soils
Kristin B Byrd, Pelayo Alvarez, Benjamin Sleeter et al.

- Developing spatial map of landscape trees at UMK Jeli campus using GIS
S Daliman, R Tan and N A Amaludin
<figure><img src="image_3.png" /></figure>

CrossMark

OPEN ACCESS

**RECEIVED**
8 April 2022

**REVISED**
25 October 2022

**ACCEPTED FOR PUBLICATION**
22 December 2022

**PUBLISHED**
24 January 2023

Original Content from
this work may be used
under the terms of the
Creative Commons
Attribution 4.0 licence.

Any further distribution
of this work must
maintain attribution to
the author(s) and the title
of the work, journal
citation and DOI.

<figure><img src="image_4.png" /></figure>

PAPER

# Pl@ntNet Crops: merging citizen science observations and structured survey data to improve crop recognition for agri-food-environment applications

M van der Velde<fnref n="1" />,* ID, H Goeau<fnref n="2" />, P Bonnet<fnref n="3" /> ID, R d'Andrimont<fnref n="1" /> ID, M Yordanov<fnref n="1" />, A Affouard<fnref n="2" />, M Claverie<fnref n="1" /> ID, B Czucz<fnref n="1" /> ID, N Elvekjaer<fnref n="1" />, L Martinez-Sanchez<fnref n="1" /> ID, X Rotllan-Puig<fnref n="1" /> ID, A Sima<fnref n="1" />, A Verhegghen<fnref n="1" /> ID and A Joly<fnref n="2" /> ID

E-mail: marijn.van-der-velde@ec.europa.eu

**Keywords:** citizen science, deep learning, computer vision, authoritative data, crowdsourcing, common agricultural policy, monitoring

Supplementary material for this article is available [online](#)

## Abstract

We present a new application to recognize 218 species of cultivated crops on geo-tagged photos, 'Pl@ntNet Crops'. The application and underlying algorithms are developed using more than 750k photos voluntarily collected by Pl@ntNet users. The app is then enriched by data and photos coming from the European Union's (EU) Land Use and Coverage Area frame Survey (LUCAS). During five tri-annual LUCAS campaigns from 2006 to 2018, 242 476 close-up 'cover' photos of crops were collected. The survey protocol for these photos specified that 'the picture should be taken at a close distance, so that the structure of leaves can be clearly seen, as well as flowers or fruits'. This unique labelled data provides an opportunity to further generalize the Pl@ntNet computer vision algorithms to recognize crops and enlarge their geographic representivity across the EU. To include LUCAS cover photos, we semantically match Pl@ntNet species and LUCAS legends, predict the species on LUCAS cover photos with the existing Pl@ntNet algorithm, and consider the accuracy of the classification and the number of species enriched by the photos. By setting a threshold of >0.5 on the Pl@ntNet prediction probabilities, 70 170 LUCAS photos representing 101 species classified with an accuracy of 0.9 were added to the 'Crops' app. The thematic accuracy of the legacy LUCAS data was improved by distinguishing 218 species, opposed to the original 36 LUCAS levels. Official and publicly financed LUCAS datastreams can now be improved because of Pl@ntNet citizen science, photo collection, and deep learning model development. Further use of the app and policy-relevant workflows in the agri-food-environment domain are discussed.

## 1. Introduction

Specialized social networks have an unprecedented ability to scale and match observations and expertise. Nowadays, the collection of photos and observations is often combined with powerful algorithms for various classification tasks (e.g. [1, 2]). Methodologies are designed that can efficiently channel citizen science into environmental monitoring to inform policy processes, e.g. the monitoring of

the Sustainable Development Goals [3]. The synergy between expertise and citizen science is also explored in the domain of land cover and land use assessments. Citizen scientists have helped to assess the land available for biofuel production [4]. Bayas et al [5] investigated how the 'crowd' could contribute to in-situ land cover and use observations to complement the European Union's (EU) Land Use and Coverage Area frame Survey (LUCAS). For example, while LUCAS is carried out every three to

<footnote n="1">¹ European Commission, Joint Research Centre (JRC), Ispra, Italy</footnote>

<footnote n="2">² INRIA Sophia-Antipolis—ZENITH Team, LIRMM, Montpellier, France</footnote>

<footnote n="3">³ CIRAD, CNRS, INRAE, IRD, Montpellier, France</footnote>

<footnote>* Author to whom any correspondence should be addressed.</footnote>
four years, citizen scientists could make annual obser-
vations. Conversely, the geographic scope of LUCAS
is EU-wide, avoiding biases that may occur in volun-
tarily driven observation schemes. Large-scale biod-
iversity and protected area monitoring are increas-
ingly benefiting from citizen science observations [6].
Novice, amateur, and expert botanists have been col-
lecting and revising millions of geo-tagged and time-
stamped photographs of plant species with apps such
as Flora Incognita [7], iNaturalist [8], and Pl@ntNet
[9]. Besides contributing to biodiversity monitoring,
these apps are intrinsically motivating volunteers as
they function as modern day floras. After careful
expert-based quality control procedures these obser-
vations can contribute to platforms such as the Global
Biodiversity Information Facility (www.gbif.org/).

Synergies between demand and supply of
collection capacity and expertise are at the core of
several recent and successful examples in the agri-
food-environment domain (e.g. [10]). Farmers are
also actively participating in such activities. Although
there is a long history of public engagement in agri-
culture, specifically through extension work with
beneficial bidirectional flows of information, the
term citizen science has rarely been applied to this
[11]. Farmers and extension workers can benefit from
crop disease expertise networks to access information
on treatments [12]. Farmer citizen scientists particip-
ated in on-farm participatory trials providing insights
into variety adaptation and selection across large geo-
climatic scale [13]. Farmers also have been engaged
in documenting temporal trends in farmland biod-
iversity and relating these to agricultural practices
[14]. Accounting for the environmental performance
of farming, but also an increasing disconnect between
people and their food, warrants exploring the inter-
faces between food systems and citizen science [11].

Legacy data and photos can also improve under-
lying identification models provided proper tools
facilitating integration and annotation are available.
The massive digitization of herbarium specimens and
annotation of specimen images is one such example
[15]. Pl@ntNet provides the possibility to integrate
specific flora previously collected by (amateur) bot-
anists using dedicated semi-automatic tools. In this
letter, we explore the use of legacy *in-situ* expert pho-
tos taken and labelled during past EU-wide LUCAS
surveys. Taking advantage of the existing Pl@ntNet
species identification algorithms and its mobile and
web functionality, we present a new application in
Pl@ntNet focusing on ‘Crops’. Besides user provided
pictures of crops, the application is enriched with
suitable LUCAS cover photos of crops. Enabling
this synergy required aligning both data sources as
detailed in this manuscript. Eventually, combining
user collected photos and LUCAS surveyed pho-
tos and observations should improve the ability to
recognize crops with Pl@ntNet and deep learning

algorithms in general. These developments could
give rise to various applications in the agri-food-
environment domain. Detailed objectives are to:

(a) Present the Pl@ntNet Crops application focusing on cultivated crops.

(b) Demonstrate how after legend matching and inference with the existing Pl@ntNet algorithms the Crops application is enriched with LUCAS cover photos.

(c) Demonstrate the synergy of voluntary and LUCAS collected photos and observations for deep learning model development to recognize crops.

(d) Suggest potential uses of the Pl@ntNet Crops application and openly published crop recognition models in the agri-food-environment domain.

## 2. Data and methods

2.1. Pl@ntNet

Pl@ntNet is an existing smartphone and web-based
application that allows identifying plant species on
images [16]. More broadly, Pl@ntNet is a collabor-
ative system operating since 2010 [9] and involving
a large community of expert and amateur contribut-
ors worldwide [17]. Available in 36 languages and in
200+ countries, about 200 000 to 600 000 users use
Pl@ntNet each day. Experiencing nearly exponential
growth since its conception, more than 200 million
plant occurrences have been recorded with Pl@ntNet
around the world. Pl@ntNet provides generic func-
tionality to recognize plants that can be used by any-
one, but also dedicated applications that focus on spe-
cific flora in e.g. natural parks, from herbaria, or on
activities with an educational character. Pl@ntNet is
straightforward to use. After selecting a flora, a user
takes up to four close-up photos of the organs of a
plant following Pl@ntNet guidelines. The user spe-
cifies if the organ is e.g. a flower, fruit, leaf, bark, the
so-called view. In return, Pl@ntNet provides a ranked
list of the most probable predicted species. Each spe-
cies is illustrated by pictures of that species for com-
parison, grouped by their view, that may also include
a ‘habitat’ view if the photo was not close-up but
rather provided a landscape view. See figure 1 on the
left side for typical photos submitted to Pl@ntNet.
Additional information and links to descriptive web
pages of the species are also provided.

## 2.2. LUCAS cover photos

The LUCAS has collected *in-situ* data on land use and
cover across the EU in 2006, 2009, 2012, 2015 and
2018. During those surveys, observations are done on
>300 000 points for a sample that is statistically rep-
resentative with respect to land cover. Along with data
on a set of relevant variables, photos are taken in the
<figure><img src="image_5.png" /></figure>

four cardinal directions from the point. A detailed methodological description, as well as the harmonised data and photos are published in [18]. Besides the photos mentioned above, previously unpublished ‘cover’ photos were also taken. The protocol specified that these cover photos ‘should be taken at a close distance, so that the structure of leaves can be clearly seen, as well as flowers or fruits’. During the five campaigns, 874 646 LUCAS cover photos were collected, out of which 242 476 were of crops. After a two-step anonymization process relying on computer vision and visual inspection (detailed in [19]), these photos will be published in 2022. The Crops app ingests the cover photos of crops. See supplementary figure 1 and figure 1 on the right side for typical LUCAS cover photos of crops. As can be seen, these contain full plant views, but also more distant views, fences

that may block the view, and occasionally the presence of objects to mark the point.

## 2.3. Pl@ntNet crop recognition models

Pl@ntNet is an automated visual identification system that integrates various novel approaches in handling and classifying imagery [9]. Pl@ntNet is built on the combination of a deep learning image classification model and a generalist content-based image retrieval method (for details see [9, 16]). Besides computer vision based identification, the Pl@ntNet framework is itself synchronized with observations that are validated by a network of expert botanists. This allows the recognition performances to improve with an ever increasing amount of training data. Pl@ntNet datasets and algorithms are also benchmarked in community-steered identification challenges such as
LifeCLEF [1]. This type of long-term evaluation makes it possible to quickly detect major advances in terms of performance and to accelerate their integration into operational production systems such as Pl@ntNet.

## 2.4. Model performance and including LUCAS photos in the app

First, before including the LUCAS crop photos into the application, the accuracy of the Pl@ntNet algorithm to identify crops is evaluated on the photos contributed by citizen scientists. When training Pl@ntNet's recognition algorithm, a fraction of the data (randomly selected) that has been evaluated by experts is removed from the training set in order to evaluate the performance of the crop identification. Second, to decide which LUCAS cover photos of crops can be included in the application, considerations are made regarding the probability of the prediction and the accuracy of the crop identification by comparing the Pl@ntNet classification with the LUCAS label. This requires matching the LUCAS legend labels with the Pl@ntNet species reference list. Legend matching is also needed for the future use of the photos to train and improve the current Pl@ntNet species identification algorithm. In the Pl@ntNet procedure, while users need to specify the type of view when submitting the photo, the algorithm also classifies the view (e.g. flower, leave, bark, habitat) before classifying the species. After inclusion in the Crops app, the established expert driven Pl@ntNet quality control mechanism continues to take place.

# 3. Results and discussion

## 3.1. Model performance

A total of 218 cultivated crop species are included in the Pl@ntNet Crops application (see supplementary table 1 and the Crops application). Along with scientific names, common names translated into 36 languages are also provided. Species covered include major crops (maize, wheat, rice, yam), but also vegetables (asparagus, eggplant), tree crops (olives, coconut), fruits (kiwi, apple), nuts (hazelnuts, pistachio), spices (cinnamon, black pepper), and cover or N-fixing crops (clover). Pl@ntNet users have already contributed 605 242 crop photos (as of 16 November 2021) from around the world to the Crops application. Out of these 605 242, the classification of 260 610 of those have also been evaluated by experts. The dataset that was used to train the model was selected from these.

Before including the LUCAS cover photos, the performance of the current Pl@ntNet Crops algorithm is evaluated with the randomly selected validation set that is removed from the training set. This set contains 2654 images and allows to measure the average identification performance of unseen

<table><caption>**Table 1.** Performance evaluation of Pl@ntNet's crop recognition application.</caption><thead><tr><td>Mean average precision</td><td>Top-1 accuracy</td><td>Top-5 accuracy</td></tr></thead><tbody><tr><td>0.927</td><td>0.891</td><td>0.972</td></tr></tbody></table>

crop images. As reported in table 1, the correct species is returned in first position 89% of the time, and among the top 5 predicted species 97% of the time. This provided a satisfactory performance of the algorithm that is used in the Pl@ntNet Crops application. A threshold on the prediction probability of the algorithm will determine whether a LUCAS cover photo can be included in the app.

## 3.2. Legend matching and species list

Pl@ntNet uses up to date Latin taxonomy to classify at species level. The LUCAS legend level 3, which is used to label the LUCAS cover photos, contains a mix of family and species names with synonyms and alternative spellings that needed to be semantically mapped to the Pl@ntNet reference species list (see supplementary table 1). The 218 cultivated crops in the application were matched to 36 LUCAS legend level 3 classes. Following the LUCAS class definition, the frequency distribution across the different crop types is shown in supplementary figure 2. This reveals the long-tail problem [9], for only a few of the crop species—the major crops—a large amount of LUCAS observations with photos exist. However, if the Pl@ntNet observations are classified following the LUCAS legend, a large group of observations are included in the ‘other fruit trees and berries’ class. Of course, a much richer differentiation exists at species level for the Pl@ntNet photos. This is illustrated in figure 2. Here each bar indicates a Pl@ntNet species, the size of the bar the number of Pl@ntNet citizen scientist observations for that species, while colour indicates the LUCAS legend level 3. Unfortunately, when using the original LUCAS label to evaluate the accuracy of the Pl@ntNet classification of the LUCAS cover photos, species will be grouped that may also be similar (e.g. ‘dry pulses’).

## 3.3. Spatial and temporal distribution of Pl@ntNet and LUCAS photos

Pl@ntNet users have already contributed 605 242 crop photos (as of 16 November 2021) to the Crops application. Approximately 400 000 photos contain GPS geolocation information and 260 610 of them are both present in Europe and have been validated by experts. The potential contribution of LUCAS cover photos of crops to the Pl@ntNet application amounts to a total of 242 476 photos and observations taken across the EU following the LUCAS classification (see supplementary figure 3 mapping the distribution of LUCAS cover photos across the EU for seven crops). The type of view classified by the algorithm
<figure><img src="image_6.png" /><figcaption>**Figure 2.** Distribution of the Pl@ntNet observations with the LUCAS class equivalent (Pl@ntNet data have been converted from species to LUCAS legend level 3). Only species with a minimum of ten observations are shown.</figcaption></figure>

(table 2) highlights the difference between Pl@ntNet user and LUCAS surveyor provided photo. While most Pl@ntNet user photos are recognized to have a view of flowers or leaves, most LUCAS photos are classified to be of habitat type, i.e. a full plant view, or a more distant photo of a cropped field. This is despite the LUCAS protocol, which referred to pictures to be taken at a close distance.

The relative contribution of Pl@ntNet and LUCAS observations to the Crops app is evaluated by comparing the 260 620 evaluated Pl@ntNet observations and the 242 476 potential LUCAS observations. Geographically speaking, the relative contribution of LUCAS to Pl@ntNet observations across Europe is especially pronounced in Eastern Europe and away from populated coastal and metropolitan areas (see figure 3). Following the launch of Pl@ntNet in 2010 and the direct uptake in the existing French Tela Botanica network (www.tela-botanica.org), it is no surprise that most Pl@ntNet user made observations have been made in France. The LUCAS cover photos are geographically distributed more uni-

formly, a drawback is that generally only a single cover photo is taken at a LUCAS point. Some of these LUCAS points will have been revisited only once, while others may have been visited during all five past surveys [18].

Summarized across a year, the temporal distribution of observations contributed by Pl@ntNet users and LUCAS surveyors is seen in figure 3. For Pl@ntNet this encompasses the time period between 2010 and 2021, while this covers the five survey years for LUCAS. Clearly, most photos are provided during the growing season, when Pl@ntNet users are out and about, and when the LUCAS surveys are carried out. However, the contribution of Pl@ntNet users is more stable throughout the year, while the LUCAS surveys clearly peak with the campaign time. These campaigns are also more or less staggered according to phenological development from South to North. Since 2018 the number of evaluated Pl@ntNet observations of cultivated crops have increased dramatically, reaching more than 80 000 in 2020, illustrating one of the strengths of citizen science.
<table><caption>**Table 2.** Type of views of Pl@ntNet user and LUCAS cover pictures.</caption><thead><tr><th></th><th>Total</th><th>Flower</th><th>Fruit</th><th>Leaf</th><th>Bark</th><th>Habit</th><th>Other</th></tr></thead><tbody><tr><td>Pl@ntNet user (n)</td><td>605 242</td><td>231 669</td><td>62 541</td><td>260 005</td><td>17 099</td><td>26 666</td><td>7262</td></tr><tr><td>Pl@ntNet user (%)</td><td></td><td>38</td><td>10</td><td>43</td><td>3</td><td>4</td><td>1</td></tr><tr><td>LUCAS cover total (n)</td><td>242 476</td><td>24 401</td><td>25 333</td><td>67 488</td><td>1692</td><td>98 234</td><td>25 328</td></tr><tr><td>LUCAS cover total (%)</td><td></td><td>10</td><td>10</td><td>28</td><td>1</td><td>41</td><td>10</td></tr><tr><td>LUCAS cover inference > 0.5 (n)</td><td>70 170</td><td>12 516</td><td>7662</td><td>23 193</td><td>55</td><td>26 235</td><td>509</td></tr><tr><td>LUCAS cover inference > 0.5 (%)</td><td></td><td>18</td><td>11</td><td>33</td><td>0</td><td>37</td><td>1</td></tr></tbody></table>

### 3.4. Including LUCAS cover photos

Several considerations are made when deciding to include LUCAS cover photos. This includes the probability of the prediction, the accuracy of the prediction against the original LUCAS classification of the crop on the photo (distinguishing 36 classes), but also the added value the photos provide. Considerations include maximizing the total number of photos that could be added to the app, minimizing the risk of confusion with the fuzzy LUCAS legend, or increasing the number of photos for species that have few or no photos associated with them. Since human expert review is part of the Pl@ntNet workflow, erroneous classifications may be corrected. Figure 4 quantifies this. Using the legend matching, the accuracy of the species classified with the highest probability by Pl@ntNet is calculated (upper left panel of figure 4). The accuracy (against the independent LUCAS labels) of the classification increases with probability threshold, and even the lowest Pl@ntNet probabilities correspond to an accuracy of 0.62. Figure 1 illustrates this further: LUCAS cover photos with a probability of classification >0.9 (L1) are included in the app, while photos predicted with a probability <0.9 but >0.4 (L2) and <0.4 (L3) are excluded. LUCAS cover photos classified with the lowest probability (L3) include photos that are very distant (B81), have little contrast (B11), depict harvested bare soil (B21), agricultural plastic (B45), and flattened crops (B51).

There are 112 445 LUCAS cover photos that could be included. The reduction from 242 476 occurs for two major reasons. First, 63 878 have been rejected as a classification could not be made by Pl@ntNet for various reasons, e.g. this happens for many of the photos with a habitat view, or there is an imposing object (e.g. a marker, notebook, ...). Second, of the 178 598 for which a prediction could be made, the first species predicted is not part of the Crops reference list of 218 crops. This can be due to various reasons, again, the wider view of the habitat photos does not correspond to the close-up view usually provided to the algorithm which may hamper the classification, but also there is a lack of training data for the crop species in the reference list. Logically, the number of photos that could be included decreases with increasing probability threshold (upper right panel). A significant number of species could benefit

from including the LUCAS cover photos. However, with an increasing probability threshold, that number decreases dramatically as well, from 133 to 26 species. Finally, the number of photos added to each species is specified in the boxplot in the lower right panel.

Following the considerations specified above, a reasonable compromise was found by setting the threshold for the probability for inclusion of LUCAS cover photos at >0.5. Hereby, 70 170 LUCAS photos classified with an accuracy of 0.9 are added to the Crops app enriching 101 species. The LUCAS Cover photos currently contributing to the app can be found here: https://identify.plantnet.org/partners/lucas-survey. In the near future, species below the 0.5 threshold will be visually evaluated as these LUCAS cover photos could improve the Pl@ntNet algorithms the most as they have been the most difficult to predict.

### 3.5. Pl@ntNet crops and application context

The interface and functionality of the mobile Pl@ntNet Crops application is illustrated in supplementary figure 4. The app includes more than 842 320 photos and more than 675 000 observations (as of 19 October 2022) and is available as an Android and iOS app as well as through a web interface (https://identify.plantnet.org/eu-crops). Various application contexts can be considered, but we consider three here. For more examples of other Pl@ntNet activities, please explore https://identify.plantnet.org/ and related literature.

#### 3.5.1. *Collecting in-situ observations*

The increasing volume of freely accessible Earth Observation (EO) data through e.g. the Copernicus program, has not been matched with proportional amounts of *in-situ*, reference, and training data, hampering novel applications. Using automated workflows, the Pl@ntNet Crops app can contribute to collecting *in-situ* data for such EO applications. Since the app is global and includes 218 species cultivated around the world, this may be particularly relevant for data-poor environments. In food insecure regions, *in-situ* crop type data are generally missing [20]. In those regions the combination of *in-situ* data with EO data is crucial to better inform early warning
<figure><img src="image_7.png" /><figcaption>**Figure 3.** (a) The geographic contribution of Pl@ntNet user and LUCAS cover provided photos. (b) The monthly distribution of Pl@ntNet user and LUCAS cover provided photos. (c) Annual totals since 2005.</figcaption></figure>

systems and guide humanitarian responses, ensuring market transparency and stability, and informing national agricultural policies [21]. The use of citizen science tools such as the Pl@ntNet Crops app could

also encourage individual farmers or surveyors to report on crops grown on specific parcels without the need to have expertise in geo-spatial technologies. Crop type recognition on geo-referenced images
<figure><img src="image_8.png" /><figcaption>**Figure 4.** Including inferred LUCAS cover photos in Pl@ntNet. Accuracy (upper left), correctly predicted images (upper right), number of species added to Pl@ntNet (lower left), and images per predicted species added to Pl@ntNet (lower right), as a function of the Pl@ntNet probability of the identification.</figcaption></figure>

for *in-situ* data collection using computer vision has been successfully tested [20, 22]. Since the position of the camera may not be in the field, smart ways to link the precise position of the crop located on the photo and the field are needed. Pictures are taken directly in the field in campaigns such as those done for the Copernicus4GEOGLAM component (https://land.copernicus.eu/global/about-copernicus4geoglam) of the Copernicus Global Land Service. However, these protocols do not necessarily ensure a smooth integration in computer vision and machine learning applications. Metrics on the positional accuracy of the *in-situ* data collection will be extracted by comparisons against spatially explicit parcel-level farmers’ declarations of crops (from e.g. France and the Netherlands) as gathered in the context of the Integrated Administration and Control System of the Common Agricultural Policy (CAP).

### 3.5.2. CAP

Automation and new technologies can reduce administrative burden and strengthen evidence provision for the European Union's CAP. When receiving specific subsidies, the responsible administration

requests farmers to provide a georeferenced photo proving that they grow certain crops (see [23]), e.g. besides crops supported by voluntary coupled support, for cover, catch, or N-fixing crops. The deep learning model developed as part of the Pl@ntNet Crops application could be ported elsewhere and implemented to take advantage of this automation. For instance, should a photo taken by a farmer result in a classification with low probability (below a threshold accepted by the administration), he/she would be immediately alerted to pay attention to the protocol and retake the photo, e.g. from a more optimized viewpoint or distance to the plant, to ensure sufficient probability of a correct crop classification. The foreseen benefits are threefold: (a) for the farmer—ensuring that his/her duty of providing the evidence was fulfilled correctly and avoiding extra efforts of coming back to the field to retake the photo at a later date; (b) for the administration handling the subsidies—receiving high quality proof captured according to the protocol enabling automatic processing and thus decreasing the need for costly expert photo evaluation; (c) for further algorithm development—the misclassified
first photo could be added to the training set to improve the classification probability. Such approaches are already implemented by certain administrations to improve the environmental performance of the CAP. To receive support in Thuringia, Germany, farmers need to evidence maintenance of environmentally sensitive and biodiverse grasslands by providing the occurrence of at least six key species out of list using Flora Incognita (https://marswiki.jrc.ec.europa.eu/wikicap/images/5/57/09_Detection_of Bradyrhizobium plants.pdf). These approaches improve monitoring of practices, but also directly contribute to near real-time monitoring of ecological patterns and can thus help to assess progress towards the EU biodiversity targets in managed agricultural land.

### 3.5.3. Food system awareness raising

Engaging citizen scientists in food system research keeps on gaining momentum and now covers diverse applications [24]. At the same time, Ryan et al [11] note that the term citizen science has rarely been applied to a long history of public engagement in agriculture and food science. Consumers increasingly care about the origin and sustainability of the food they buy. In the context of the EU's Farm to Fork strategy, citizen science activities with consumers or schools could use the Crops app to explore local agricultural produce and in this process gain knowledge about the crops that are cultivated in their surroundings, the sustainability of the production process, and the interaction with the local environment. The transparency of the production chain for agricultural commodities (oil palm, cocoa, coffee, ...) produced outside of Europe is also of concern to consumers. If geolocated crop type is combined with additional information, the development of targeted applications could improve monitoring of environmental externalities such as biodiversity impact [25] of e.g. traditional and agro-forestry-based systems. These developments could strengthen the relationship between producers and consumers and encourage agronomic and environmental awareness with positive impacts on food systems, especially at the production stages. Nevertheless, Mourad et al [26] found that many citizen and farmer science projects tend to focus on academic research outcomes, and not on sustainable solutions in practice. Here we propose that image recognition apps and deep learning models originating in the citizen science domain can also more indirectly feed work flows and policies implementing sustainable farm practices.

## 3.6. Lessons learned

Opportunistic use was made of legacy LUCAS cover photos. There was enough overlap between the visual requirements of the Pl@ntNet algorithms and the

LUCAS cover photos so that 31% of the photos could be included in the app after a first inference. Inclusion of the LUCAS cover photos that are predominantly of 'habitat' type, improves the Pl@ntNet algorithm for this type of view. For 70 170 legacy LUCAS observations, the thematic accuracy was improved by distinguishing 218 species opposed to the original 36 levels. As expert evaluation improves the training set and thus the crop identification, LUCAS cover photos will increasingly be added as the algorithm improves.

Photos and observations that are voluntarily collected by apps such as Pl@ntNet and by statistically representative surveys such as LUCAS have different but complementary strengths. Now that the legends are matched, LUCAS surveyors could (voluntarily) use the app for the crop classification task, especially if they are not agronomists. While generally only a single photo is taken of a crop at a LUCAS point (i.e. similar to the one-shot problem, very few images per species and type of view, see [9]), multiple photos of the same species, and of different organs, can be used for the identification by Pl@ntNet. By combining photos from volunteers and surveys, as done here, increasingly rich training datasets are created. While noise in the imagery is a drawback, a positive aspect is the diversity of such datasets. Over time, this should lead to the development of more robust identification models. Such models are needed for real-world applications.

The LUCAS protocol was not built for use in computer vision and machine learning. Taking a photo of a cropped field with many closely spaced individuals is different compared to taking a close-up photo of an organ of a single plant. Furthermore, even though a protocol for close-up photos was specified, rather heterogeneous imagery is present in the LUCAS cover set. Following on from these experiences, we have improved the instructions for the 2022 LUCAS survey protocol. Background markers are no longer included during the acquisition of LUCAS cover photos to reduce the noise in the photo for further computer vision applications. Furthermore, in the training of surveyors emphasis was placed on the need of photos taken at a close distance, that have sufficient contrast, and where individual crops could be distinguished. Direct use of the app with requirements on the probability of the classification will be suggested for the next LUCAS survey. The cover photos of the LUCAS 2022 survey will also be included in the app once available. Besides photos of crops, the LUCAS cover photo-dataset also includes photos of trees (leaves) and grasses. These may be included within other Pl@ntNet floras. Other LUCAS legacy photos, such as those taken in the four cardinal directions, may prove useful in computer vision challenges related to land cover and use and landscape analysis [27].
A proliferation of apps for plant recognition has appeared in recent years (e.g. LeafSnap, PictureThis, PlantStory, Seek, etc). Jones [28] compared ten automated image recognition apps to identify British flora. Here, overall Pl@ntNet was a mid-level performer. Otter et al [29] concluded that such apps cannot yet be trusted to identify toxic plants, with two out of three apps tested (including Pl@ntNet) correctly predicting about half of plants. The extension service of Michigan State University (USA) evaluated eight different apps, with Pl@ntNet ranking second (www.canr.msu.edu/news/plant-identification-theres-an-app-for-that-actually-several). Clearly, the results of such evaluations depend strongly on the species chosen for the test set. In the future a stronger differentiation may appear between specialised systems and those oriented towards more common species.

One specific complication for certain crops is that varieties of the same species are very different. For example, *Brassica oleracea* L. includes cultivars such as cabbage, broccoli, cauliflower, kale, and Brussels sprouts. In Pl@ntNet, being one species, these cultivars are all in the same class, notwithstanding their different visual appearances. Traditionally in the LifeCLEF challenge [30], the leading activity on benchmarking new algorithms for automated plant recognition apps, accuracy is evaluated at species, genus, and family level. The particular difficulty of very visually distinct crop varieties has not been solved yet. Different phenological stages provide another challenge [22]. In fact, plant identification can be difficult, even for a trained botanist, and benchmarked datasets are needed to improve algorithms (see for example the public dataset created by Pl@ntNet on anemones, https://plantnet.org/en/2021/03/30/plantnet-dataset-for-machine-learning-researchers/).

## 4. Conclusions

A new Pl@ntNet app including more than 842k photos (as of 19 October 2022) of 218 crop species around the world is presented. The application and underlying algorithms are built on voluntarily collected photos of crops and enriched by legacy LUCAS survey cover photos of crops. The comparative strengths of merging such networked collaborative and survey-based approaches relate to temporal and geographic representivity of the sampling. The application can benefit various agri-food-environment activities and can contribute to the further development of computer vision algorithms to recognize crops on photos.

## Data availability statement

No new data were created or analysed in this study.

## Acknowledgments

This project has received funding from the European Union's Horizon 2020 research and innovation program under Grant Agreement No. 863463 (Cos4Cloud project).

## ORCID iDs

M van der Velde https://orcid.org/0000-0002-9103-7081

P Bonnet https://orcid.org/0000-0002-2828-4389

R d'Andrimont https://orcid.org/0000-0002-7326-7684

M Claverie https://orcid.org/0000-0001-9479-3205

B Czucz https://orcid.org/0000-0002-6462-7633

L Martinez-Sanchez https://orcid.org/0000-0002-1354-1792

X Rotllan-Puig https://orcid.org/0000-0003-2046-0621

A Verhegghen https://orcid.org/0000-0002-4234-0586

A Joly https://orcid.org/0000-0002-2161-9940

## References

[1] Joly A, Goëau H, Glotin H, Spampinato C, Bonnet P, Vellinga W P, Lombardo J C, Planqué R, Palazzo S and Müller H 2019 Biodiversity information retrieval through large scale content-based identification: a long-term evaluation *Information Retrieval Evaluation in a Changing World* (Switzerland: Springer International Publishing) pp 389–413

[2] Lee S H, Goëau H, Bonnet P and Joly A 2020 *Comput. Electron. Agric.* 170 105220

[3] Fritz S et al 2019 *Nat. Sustain.* 2 922–30

[4] Fritz S et al 2013 *Environ. Sci. Technol.* 47 130128103203003

[5] Bayas J C L, See L, Bartl H, Sturn T, Karner M, Fraisl D, Moorthy I, Busch M, van der Velde M and Fritz S 2020 *Land* 9 446

[6] Bonnet P et al 2020 *Ecol. Solut. Evid.* 1 e12023

[7] Mäder P, Boho D, Rzanny M, Seeland M, Wittich H C, Deggelmann A and Wäldchen J 2021 *Methods Ecol. Evol.* 12 1335–42

[8] Unger S, Rollins M, Tietz A and Dumais H 2020 *J. Biol. Educ.* 55 1–11

[9] Joly A et al 2014 *Ecol. Inform.* 23 22–34

[10] Minet J, Curnel Y, Gobin A, Goffart J-P, Mélard F, Tychon B, Wellens J and Defourny P 2017 *Comput. Electron. Agric.* 142 126–38

[11] Ryan S F et al 2018 *Proc. R. Soc. B* 285 20181977

[12] Hughes D P and Salathe M 2016 An open access repository of images on plant health to enable the development of mobile disease diagnostics (arXiv:1511.08060 [cs.CY])

[13] van Etten J et al 2019 *Proc. Natl Acad. Sci.* 116 4194–9

[14] Billaud O, Vermeersch R-L, Porcher E and Pocock M 2020 *J. Appl. Ecol.* 58 261–73

[15] Pearson K D et al 2020 *BioScience* 70 610–20

[16] Affouard A, Goëau H, Bonnet P, Lombardo J C and Joly A 2017 Pl@ntNet app in the era of deep learning *ICLR: Int. Conf. on Learning Representations*

[17] Bonnet P et al 2020 *Ecol. Solut. Evid.* 1 e12023

[18] d'Andrimont R et al 2020 *Sci. Data* 7 352
[19] d'Andrimont R, Yordanov M, Martinez-Sanchez L, Haub P, Buck O, Haub C, Eiselt B and van der Velde M 2022 *Earth Syst. Sci. Data* 14 4463–72

[20] Paliyam M, Nakalembe C, Liu K, Nyiawung R and Kerner H 2021 Climate change AI workshop (Joint Research Centre) (available at: https://marswiki.jrc.ec.europa.eu/wikicap/images/f/ff/Geotagged_JRC_ReportV5b.pdf)

[21] Becker-Reshef I et al 2020 *Remote Sens. Environ.* 237 111553

[22] d'Andrimont R, Yordanov M, Martinez-Sanchez L and van der Velde M 2022 *Comput. Electron. Agric.* 196 106866

[23] Sima A, Loudjani P and Devos W 2020 Use of geotagged photographs in the frame of Common Agriculture Policy checks, Ispra, JRC120223

[24] Oakden L et al 2021 *Front. Sustain. Food Syst.* 5 596594

[25] August T A, Pescott O L, Joly A and Bonnet P 2020 *Patterns* 1 100116

[26] Mourad K A, Hosseini S H and Avery H 2020 *Sustainability* 12 10375

[27] Martinez-Sanchez L, Borio D, d'Andrimont R and van der Velde M 2022 *Ecol. Inform.* 70 101757

[28] Jones H G 2020 *AoB PLANTS* 12 plaa052

[29] Otter J, Mayer S and Tomaszewski C A 2020 *J. Med. Toxicol.* 17 42–47

[30] Bonnet P, Goëau H, Hang S T, Lasseck M, Šulc M, Malécot V, Jauzein P, Melet J C, You C and Joly A 2018 Plant identification: experts vs. machines in the era of deep learning *Multimedia Tools and Applications for Environmental & Biodiversity Informatics* (Switzerland: Springer International Publishing) pp 131–49