# Curation and analysis of historical wheat phenotypic data

Erwan Le Floch

<figure><img src="image_1.png" /></figure>

## To cite this version:

Erwan Le Floch. Curation and analysis of historical wheat phenotypic data. AGENT Final Project Meeting, INRAE, Apr 2025, Versailles, France. hal-05035460

HAL Id: hal-05035460

https://hal.inrae.fr/hal-05035460v1

Submitted on 15 Apr 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_3.png" /></figure>

AGENT

Global Access to Plant Genetic Resources

# Curation and analysis of historical
wheat phenotypic data

AGENT Final Meeting - 02/04/2025

E. Le Floch (INRAE)

<figure><img src="image_4.png" /></figure>

<figure><img src="image_5.png" /></figure>

The AGENT project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 862613.
Content

- Historical phenotypic data flow

- Data curation & analysis methods

- Results
Phenotyping Data Flow

<figure><img src="image_6.png" /></figure>

https://doi.org/10.5281/zenodo.12625359
Phenotyping Data Flow

<figure><img src="image_7.png" /></figure>
Data collection

460,399 Data points
campaigns from 1967 to 2022

<figure><img src="image_8.png" /></figure>

<figure><img src="image_9.png" /></figure>

<figure><img src="image_10.png" /></figure>

<figure><img src="image_11.png" /></figure>

<figure><img src="image_12.png" /></figure>

43,292 Accessions
Data collection

<table><thead><tr><td>Genebank</td><td>Phenotyping campaigns</td><td>Locations</td><td>Accessions tested</td><td>Data points</td></tr></thead><tbody><tr><td>Agroscope</td><td>73</td><td>51</td><td>1186</td><td>76392</td></tr><tr><td>CREA</td><td>13</td><td>3</td><td>1353</td><td>28704</td></tr><tr><td>ICARDA</td><td>28</td><td>7</td><td>22609</td><td>145296</td></tr><tr><td>IHAR</td><td>52</td><td>3</td><td>7607</td><td>128936</td></tr><tr><td>INIA</td><td>22</td><td>1</td><td>1260</td><td>6331</td></tr><tr><td>IPGR-Sadovo</td><td>31</td><td>2</td><td>504</td><td>27398</td></tr><tr><td>NARDI F.</td><td>40</td><td>18</td><td>750</td><td>23406</td></tr><tr><td>NPPC</td><td>40</td><td>2</td><td>5784</td><td>18362</td></tr><tr><td>WUR</td><td>21</td><td>10</td><td>2239</td><td>5574</td></tr></tbody></table>
# Semi-automated data curation in FAIRDOM

## Phenotyping Data Flow

<figure><img src="image_13.png" /></figure>
# Semi-automated data curation in FAIRDOM

## Data file validator

- developed by Matthijs B. (WUR)

- initially for AGENT

- now very configurable (to use with
  MIAPPE template for example)

- available since February on
  [pypi.org/project/excel-validator](https://pypi.org/project/excel-validator)

### Validation of template based Excel files

**Navigation**

<figure><img src="image_14.png" /></figure>

Project description

<figure><img src="image_15.png" /></figure>

Release history

<figure><img src="image_16.png" /></figure>

Download files

**Verified details**

<figure><img src="image_17.png" /></figure>

These details have been verified by PyPI

**Maintainers**

<figure><img src="image_18.png" /></figure>

matthijsbrouwer

**Unverified details**

These details have not been verified by
PyPI

**Project links**

<figure><img src="image_19.png" /></figure>

Homepage

**Meta**

o License: MIT License
Copyright (c) 2024 EU H2020
AGENT project Permission is
hereby granted, free of
charge...

o Author: Matthijs Brouwer

o Requires: Python >=3.11

**Classifiers**

**Operating System**

o OS Independent

**Project description**

**Excel Validator**

Excel Validator is a Python package designed to validate Excel files (.xlx) based on configured schemas. The tool ensures your Excel files adhere to specified schemas and generates detailed reports in case of validation errors. Built on the robust Frictionless library, Excel Validator also allows for dynamic schema creation, where fields are included based on row data from other sheets.

**Features**

• Validate Excel files against predefined schemas.

• Generate detailed reports highlighting any validation issues.

• Dynamic schema creation based on data from other sheets.

• Easy integration with your existing data processing workflows.

• Built on top of the Frictionless library for reliable and extensible validation.

• Integrated webservice for online validation.

**Installation**

The software requires at least Python version 3.11. It is recommended to create and activate a dedicated conda environment for the installation of this software:

```
conda create -n excel-validator python=3.11
conda activate excel-validator
```

Now you can install Excel Validator via pip:

```
pip install excel-validator
```

**Usage**
# Data analysis

## Phenotyping Data Flow

<figure><img src="image_20.png" /></figure>
# Data analysis, ideally

## Phenotyping Data Flow

<figure><img src="image_21.png" /></figure>

decentralized:
each genebank analyses
its own data
## Data analysis, ideally

WP5-6 only provide the tools

For example:

* computational notebook template

* computational environment specifications

<figure><img src="image_22.png" /></figure>
Data analysis, in practice

The recommended analysis workflow
uses Asreml, which requires a licence

→ analysis had to be centralized

<figure><img src="image_23.png" /></figure>
# Inception of the data analysis methods used

## In 2018 in Gatersleben, Germany

ORIGINAL RESEARCH article

Front. Plant Sci., 08 May 2018

Sec. Plant Breeding

Volume 9 - 2018 | https://doi.org/10.3389/fpls.2018.00609

## Method

Leveraging the Use of Historical Data Gathered During Seed Regeneration of an *ex Situ* Genebank Collection of Wheat

<figure><img src="image_24.png" /></figure>

Norman Philipp<fnref n="1" />

<figure><img src="image_25.png" /></figure>

Stephan Weise<fnref n="2" />

<figure><img src="image_26.png" /></figure>

Markus Oppermann<fnref n="2" />

<figure><img src="image_27.png" /></figure>

Andreas Börner<fnref n="2" />

<figure><img src="image_28.png" /></figure>

Andreas Graner<fnref n="2" />

<figure><img src="image_29.png" /></figure>

Jens Keilwagen<fnref n="3" />

<figure><img src="image_30.png" /></figure>

Benjamin Kilian<fnref n="4" />

<figure><img src="image_31.png" /></figure>

Yusheng Zhao<fnref n="1" />

<figure><img src="image_32.png" /></figure>

Jochen C. Reif<fnref n="1" />*

<figure><img src="image_33.png" /></figure>

Albert W. Schulthess<fnref n="1" />

Genebanks are a rich source of genetic variation. Most of this variation is absent in breeding programs but may be useful for further crop plant improvement. However, the lack of phenotypic information forms a major obstacle for the educated choice of genebank accessions for research and breeding. A promising approach to fill this information gap is to exploit historical information gathered routinely during seed regeneration cycles. Still, this data is characterized by a high non-orthogonality hampering their analysis. By examining historical data records for flowering time, plant height, and thousand grain weight collected during 70 years of regeneration of 6,207 winter wheat (*Triticum aestivum* L.) accessions at the German Federal ex situ Genebank, we aimed to elaborate a strategy to analyze and validate non-orthogonal historical data in order to charge genebank information platforms with high quality ready-to-use phenotypic information. First, a three-step quality

https://doi.org/10.3389/fpls.2018.00609

## Data paper

Data Descriptor

[Open access](https://doi.org/10.3389/fpls.2018.00609)

Published: 29 July 2019

# Historical phenotypic data from seven decades of seed regeneration in a wheat *ex situ* collection

Norman Philipp, Stephan Weise, Markus Oppermann, Andreas Börner, Jens Keilwagen, Benjamin Kilian, Daniel Arend, Yusheng Zhao, Andreas Graner, Jochen C. Reif & Albert W. Schulthess

*Scientific Data* **6**, Article number: 137 (2019) | [Cite this article](https://doi.org/10.3389/fpls.2018.00609)

3801

Accesses

15

Citations

61

Altmetric | [Metrics](https://doi.org/10.3389/fpls.2018.00609)

## Abstract

Genebanks are valuable sources of genetic diversity, which can help to cope with future problems of global food security caused by a continuously growing population, stagnating yields and climate change. However, the scarcity of phenotypic and genotypic characterization of genebank accessions severely restricts their use in plant breeding. To warrant the seed

https://doi.org/10.1038/s41597-019-0146-y

<footnote n="1">¹ Department of Breeding Research, Leibniz Institute of Plant Genetics and Crop Plant Research (IPK), Gatersleben, Germany</footnote>

<footnote n="2">² Department of Genebank, Leibniz Institute of Plant Genetics and Crop Plant Research (IPK), Gatersleben, Germany</footnote>

<footnote n="3">³ Institute for Biosafety in Plant Biotechnology, Julius Kühn-Institut (JKI) – Federal Research Centre for Cultivated Plants, Quedlinburg, Germany</footnote>

<footnote n="4">⁴ Global Crop Diversity Trust, Bonn, Germany</footnote>
# Summary of the workflow I used

- Visualise data

- Trim raw values, with lax thresholds

- Exclude years having a single data point, and accessions tested in a single year

- Remove outlier years } with only stats: no human judgement needed

- Calculate heritability on outlier corrected data

- Fit final linear mixed model on outlier corrected data and extract BLUEs

## Outcomes:

- outlier-corrected data

- heritability: indicator of phenotypic data quality

- BLUEs: robust data aggregation at the accession level
# Summary of the workflow I used

See the repo: https://github.com/AGENTproject/historic_pheno_data_analysis

Essentially scaling a script for made for 1 trait to many genebanks, crops and traits

## Proper tooling

- R Tidyverse: easy to learn & excellent docs in https://r4ds.hadley.nz/

- computational notebooks, for reproducibility

- JupyterLab IDE

- Conda packages: controlled environment

- Git version control system

## Automation vs Debuggability tradeoff

→ one custom function per logical step
Main issue faced: A lot of unreplicated data

<figure><img src="image_34.png" /></figure>

Not all accessions were regenerated
each year: non-orthogonal data
structure

Makes fitting models difficult

Trick: *remove_lonely* function
that removes lonely data points
A broad phenotypic diversity

<figure><img src="image_35.png" /><figcaption>Dataset</figcaption></figure>

<figure><img src="image_36.png" /></figure>

<figure><img src="image_37.png" /></figure>
High-quality phenotypic data obtained after outlier correction

<table><thead><tr><th></th><th>Data points</th><th>Accessions</th><th>Data points kept</th><th>Accessions kept</th><th>Heritability</th></tr></thead><tbody><tr><td colspan="6"><strong>Agroscope spelt</strong></td></tr><tr><td>days to heading</td><td>3,946</td><td>226</td><td>62.5%</td><td>94.7%</td><td>98.9%</td></tr><tr><td>plant height</td><td>1,797</td><td>308</td><td>86.3%</td><td>55.2%</td><td>92.0%</td></tr><tr><td>thousand kernel weight</td><td>696</td><td>149</td><td>86.4%</td><td>67.1%</td><td>66.6%</td></tr><tr><td colspan="6"><strong>Agroscope wheat</strong></td></tr><tr><td>days to heading</td><td>15,286</td><td>810</td><td>72.2%</td><td>50.7%</td><td>97.5%</td></tr><tr><td>plant height</td><td>14,039</td><td>810</td><td>74.3%</td><td>50.5%</td><td>99.5%</td></tr><tr><td>thousand kernel weight</td><td>8,791</td><td>201</td><td>82.8%</td><td>88.6%</td><td>97.0%</td></tr><tr><td colspan="6"><strong>CREA wheat</strong></td></tr><tr><td>days to heading</td><td>5,937</td><td>1,353</td><td>80.6%</td><td>77.6%</td><td>88.3%</td></tr><tr><td>plant height</td><td>5,927</td><td>1,353</td><td>91.3%</td><td>80.4%</td><td>94.7%</td></tr><tr><td>thousand kernel weight</td><td>1,552</td><td>363</td><td>92.8%</td><td>75.5%</td><td>87.2%</td></tr><tr><td colspan="6"><strong>ICARDA wheat</strong></td></tr><tr><td>days to heading</td><td>57,512</td><td>22,606</td><td>47.8%</td><td>27.8%</td><td>65.7%</td></tr><tr><td>plant height</td><td>54,418</td><td>22,267</td><td>83.7%</td><td>64.2%</td><td>67.6%</td></tr><tr><td>thousand kernel weight</td><td>33,366</td><td>15,416</td><td>63.9%</td><td>35.4%</td><td>82.2%</td></tr></tbody></table>
<table><thead><tr><th></th><th>Data points</th><th>Accessions</th><th>Data points kept</th><th>Accessions kept</th><th>Heritability</th></tr></thead><tbody><tr><td colspan="6"><strong>IHAR spring wheat</strong></td></tr><tr><td>plant height</td><td>5,376</td><td>2,340</td><td>73.7%</td><td>54.6%</td><td>92.6%</td></tr><tr><td>thousand kernel weight</td><td>5,376</td><td>2,340</td><td>76.5%</td><td>55.0%</td><td>82.2%</td></tr><tr><td colspan="6"><strong>IHAR winter wheat</strong></td></tr><tr><td>plant height</td><td>10,673</td><td>5,267</td><td>81.9%</td><td>64.7%</td><td>93.2%</td></tr><tr><td>thousand kernel weight</td><td>10,673</td><td>5,267</td><td>81.7%</td><td>64.8%</td><td>72.4%</td></tr><tr><td colspan="6"><strong>INIA wheat</strong></td></tr><tr><td>days to heading</td><td>1,601</td><td>1,260</td><td>30.7%</td><td>16.1%</td><td>89.3%</td></tr><tr><td>plant height</td><td>1,601</td><td>1,260</td><td>34.0%</td><td>17.1%</td><td>66.0%</td></tr><tr><td>thousand kernel weight</td><td>1,512</td><td>1,258</td><td>28.9%</td><td>15.8%</td><td>72.0%</td></tr><tr><td colspan="6"><strong>IPGR-Sadovo wheat</strong></td></tr><tr><td>days to heading</td><td>2,491</td><td>504</td><td>98.5%</td><td>99.8%</td><td>96.6%</td></tr><tr><td>plant height</td><td>2,489</td><td>504</td><td>98.6%</td><td>99.8%</td><td>96.7%</td></tr><tr><td>thousand kernel weight</td><td>2,491</td><td>504</td><td>98.4%</td><td>100.0%</td><td>93.3%</td></tr></tbody></table>
<table><thead><tr><th></th><th>Data points</th><th>Accessions</th><th>Data points kept</th><th>Accessions kept</th><th>Heritability</th></tr></thead><tbody><tr><td colspan="6"><strong>NARDI wheat</strong></td></tr><tr><td>days to heading</td><td>4,267</td><td>750</td><td>42.2%</td><td>18.5%</td><td>72.4%</td></tr><tr><td>plant height</td><td>3,706</td><td>748</td><td>59.1%</td><td>18.6%</td><td>92.2%</td></tr><tr><td>thousand kernel weight</td><td>1,662</td><td>111</td><td>64.8%</td><td>59.5%</td><td>91.2%</td></tr><tr><td colspan="6"><strong>NPPC spring wheat</strong></td></tr><tr><td>days to heading</td><td>897</td><td>883</td><td>NA</td><td>NA</td><td>NA</td></tr><tr><td>plant height</td><td>940</td><td>891</td><td>NA</td><td>NA</td><td>NA</td></tr><tr><td>thousand kernel weight</td><td>564</td><td>535</td><td>NA</td><td>NA</td><td>NA</td></tr><tr><td colspan="6"><strong>NPPC winter wheat</strong></td></tr><tr><td>days to heading</td><td>5,572</td><td>4,850</td><td>16.4%</td><td>6.4%</td><td>75.2%</td></tr><tr><td>plant height</td><td>6,002</td><td>4,885</td><td>29.6%</td><td>15.2%</td><td>93.6%</td></tr><tr><td>thousand kernel weight</td><td>4,387</td><td>3,319</td><td>30.6%</td><td>16.8%</td><td>94.7%</td></tr><tr><td colspan="6"><strong>WUR wheat</strong></td></tr><tr><td>days to heading</td><td>794</td><td>678</td><td>15.0%</td><td>8.7%</td><td>75.0%</td></tr><tr><td>plant height</td><td>3,057</td><td>2,231</td><td>48.2%</td><td>29.7%</td><td>69.4%</td></tr><tr><td>thousand kernel weight</td><td>1,063</td><td>1,032</td><td>NA</td><td>NA</td><td>NA</td></tr></tbody></table>
Current tasks linked with data publication

1. Publication of data in e!DAL-PGP

2. Publication of code in Zenodo, through GitHub

3. Data paper submission
Back-propagation of curation work by e!DAL team to Fairdom

Phenotyping Data Flow

<figure><img src="image_38.png" /></figure>

Curation work done by
Mounika & Jay could be
used to improve data
quality in AGENT DB &
Portal
Don't hesitate to reuse the code or contribute improvements

CC-BY-4.0 (permissive license)

<figure><img src="image_39.png" /></figure>
Conclusion

High quality (heritability) phenotypic
data is now available for 23,282 wheat
accessions

Data is being published in e!DAL-PGP
with high quality (FAIR) as well

<figure><img src="image_40.png" /></figure>
## Acknowledgments

<figure><img src="image_41.png" /></figure>

- AGENT genebanks, for sharing their data !

- AGENT WP5&6 colleagues

- CREA-CI & INIA for hospitality

- e!DAL-PGP team

- ICARDA for the datathon

- IPK QG group for the guidance