SG×P : A Sorghum Genotype × Phenotype Prediction Dataset and Benchmark
Authors: Zeyu Zhang
Robert Pless
Nadia Shakoor
Austin Carnahan
Abby Stylianou
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: benchmark
dataset
plant science
ml for science
sorghum
genetics
plant phenotyping
TL;DR: This paper presents SGxP, a multimodal dataset and set of benchmarks focused on the discovery of genotype x phenotype relationships in bioenergy sorghum.

Abstract: Large scale field-phenotyping approaches have the potential to solve important questions about the relationship of plant genotype to plant phenotype.  Computational approaches to measuring the phenotype (the observable plant features) are required to address the problem at a large scale, but machine learning approaches to extract phenotypes from sensor data have been hampered by limited access to (a) sufficiently large, organized multi-sensor datasets, (b) field trials that have a large scale and significant number of genotypes, (c) full genetic sequencing of those phenotypes, and (d) datasets sufficiently organized so that algorithm centered researchers can directly address the real biological problems.  To address this, we present SGxP, a novel benchmark dataset from a large-scale field trial consisting of the complete genotype of over 300 sorghum varieties, and time sequences of imagery from several field plots growing each variety, taken with RGB and laser 3D scanner imaging.  To lower the barrier to entry and facilitate further developments, we provide a set of well organized, multi-sensor imagery and corresponding genomic data.  We implement baseline deep learning based phenotyping approaches to create baseline results for individual sensors and multi-sensor fusion for detecting genetic mutations with known impacts.  We also provide and support an open-ended challenge by identifying thousands of genetic mutations whose phenotypic impacts are currently unknown.  A web interface for machine learning researchers and practitioners to share approaches, visualizations and hypotheses supports engagement with plant biologists to further the understanding of the sorghum genotype x phenotype relationship. The full dataset, leaderboard (including baseline results) and discussion forums can be found at http://sorghumsnpbenchmark.com.

SGxP A Sorghum Genotype x Phenotype Prediction Dataset and Benchmark
Zeyu Zhang, Robert Pless George Washington University zeyu pless@gvu. edu
Nadia Shakoor
Donald Danforth Plant Science Center
Nilazhor@dmmforaacattee..
Austin Carnahan, Abby Stylianou
Saint Louis University
austin. carnahan labby styylianouSslu.ed
Abstract
Large scale penddphenoty approaches have the potential to solve important questions about the relationship of plant genotype plant phenotype. Computa- tional approaches to measuring the phenotype (the observable plant features) are required address the problem large scale, but machine learning approaches extract phenotypes from sensor data struggle without access to (a) sufficiently large, organized multi-sensor datasets, (b) field trials that have large scale and significant number of genotypes, (c) full genetic sequencing of those phenotypes, and (d) datasets sufficiently organized that algorithu centered researchers can directly address the real biological problems. Here, we present SG novel benchmark dataset from a large-scale scale field trial consisting of the complete genotype of over 300 sorghum varieties, and time sequences of imagery from several field plots growing each variety, taken with RGB and laser 3D scanner imaging. To lower the barrier to entry and facilitate further developments, we provide set of well organized, multi-sensor imagery and corresponding genomic data. We implement baseline deep learning based phenotyping approaches to create baseline results for individual sensors and multi-sensor fusion for detecting genetic mutations with known impacts. We also provide and support open-ended challenge by identifying thousands genetic mutations whose phenotypic impacts are currently unknown. A web interface for machine learning researchers and practitioners share approaches, visualizations and hypotheses supports engagement with plant biologists to further the understanding of the sorghum genotype x phenotype rela- tionship. The full dataset, leaderboard (including bascline results) and discussion forums can be found httttp::/soohhhuapppecccaaaa
Introduction
A plant's phenome is defined by its physical and biochemical characteristics, and is the result of the interaction of its genome and its environment. Finding large scale associations between genetic data and measured phenotypes in realistic conditions has the potential to transform real world agricultural systems, by enabling knowledge-driven breeding to improve crop output or performance. This is especially important as there an ever increasing need for crops that are, for example, more capable of feeding the world, more drought tolerant or otherwise resistant to climate change, or less in need of energy intensive inputs like synthetic fertilizers. In this work promote research in this area by creating benchmark datasets with large scale, multimodal imagery and genetic data from hundreds of sorghum crop variants grown in real-world field conditions. Sorghum is immensely important for both food and energy purposes, as it is used extensively as a grain crop and as source of biofuel, and has proven to be productive in variety of environments and resilient an ever changing climate.
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

Better understanding the relationship between sorghum genotypes and their expressed phenotypes has the potential to significantly impact the use of sorghum in food and energy production systems, Plant breeding pipelines classically measure plant features that are thought to be important, and then breed new varieties from those that had desirable features. Features may include appearance, structure and biochemistry (e.g. how much starch is in the stem). Here, our focus is on physical char- acteristics that are observable from visual imaging modalities. These characteristics, or phenotypes, include things like leaf shape, leaf counts, stalk height, width, and tillering (the count of how many stalks one plant produces). Tools like PlantCV [17] define explicit approaches to measuring these phenotypes under specific, typically indoor, imaging conditions. Learning based approaches to plant phenotyping are improving over time, and new tools like Segment Anything (26). suggest that deep learning advances also improve classic computer vision approaches estimating these features.
When there are very large datasets, modern deep learning offers an approach for automatically discovering features that best support learning tasks, rather than defining priori the most relevant features to then measure. Dubbed "Latent Space Phenotyping" [18 54), this data driven approach uses deep learning to compute features optimized to help solve learning problem, and then explores the use of that learned feature space as digital phenotype. The output feature vector produced by such a model a by-product of the deep learning network finding feature abstractions to correctly predict data labels. If the prediction task is complex enough, the learned features will, ideally, reflect semantically interesting phenotypes that are in the data, and perhaps even phenotypes, or combinations of phenotypes, that are not already known to be relevant.
Our contribution is the Sorghum Genotype x Phenotype (SGxP) Benchmark, a large scale, multimodal benchmark support discovery of genotype phenotype relationships. It consists of: carefully organized RGB and 3D imagery from a large field site growing over 300 different sorghum varieties and genetic data derived from full genome sequencing of those varieties, benchmarks focused on the classification of images based on small number of single nucleotide polymorphisms (SNPs) known have phenotypic impact, and over 2,700 SNPs which the impacts are currently unknown,
implementation and evaluation of baseline deep learning based latent phenotyping, to support comparison with alternative approaches, and a web platform supporting comparison of approaches with algorithm performance leader- boards, and discussion forums for interaction between machine learning practitioners and sorghum-focuu
plant biologists in understanding the SNPs with unknown expression. The images and dataset files that comprise this benchmark, along with the leaderboards and discussion forums, can be found at htthtt///soxx http:///sorghumsspbnncccooe com
2 Background
2.1
Large Scale Datasets and Deep Learning for Plant Phenotyping
Plant and agricultural datasets have been adopted both for general machine learning development such as the Iris Dataset, a flower sub-species classification dataset, initially introduced by (20) in 1936 and still used today for prototyping and testing simple machine learning algorithms as well as for more significant plant biology studies that utilize machine learning. In the last decade, deep learning has yielded huge improvements on a variety of computational tasks, including many in plant science. To achieve this performance, deep learning based approaches require very large datasets. Early machine learning focused plant datasets explored variation in the shape and appearance of leaves from large numbers of trees, including the Flavia dataset [64], the MalayaKew Plant Leaf dataset (30] and LeafSnap
(28). In recent years, the scale of the datasets has grown. For example, the
Herbarium 2022 Challenge, focused on the fine grained classification of plant species, is comprised of over 1.05 million images of 15,501 vascular plants (33).
While species identification is a common deep learning task in plant science, there are a large number of datasets and papers that focus on more specific plant phenotyping tasks measuring or characterizing visual features of plants that have been defined by biologists as being interesting or relevant. Most work in this space focuses on performing a specific task such as fruit or flower
detection and segmentation (4) 27 31 44 58
15
22). cultivar and species identification [3 5 24
31
42
43 57]. plant disease classifiction 6
19
52 61) leaf counting or shape analysis [I
16
21
35
55
60). yield prediction [22 34 41
59], and stress detection (2 10 36), among
other tasks. There has also been work to develop broader datasets and benchmarks that include
multiple phenotyping tasks, such as (37 . large scale dataset of Arabidopsis and tobacco imagery with significant annotations, focused on eight different phenotyping tasks.
2

---

In general, this prior work focuses on developing models and algorithms to extract phenotypes that are defined by biologists, as opposed to discovering novel phenotypes or incorporating plant genetics in any way. Latent Space Phenotyping [54) is notable in that it uses deep learning to discover and represent unknown plant phenotypes. Those characteristics of plant are a complex function of
its genetics and environmental factors, factors not often incorporated into these deep learning models.
Some example work that does include these factors include [32 45). which develops approaches to
predict traits from genetic information, (53) which generates 3D reconstructions to identify leaf-angle
related loci in the sorghum genome, or, most related to this benchmark, [62] which uses deep learning to predict the relative functional importance of specific genetic markers and mutations in plants.
In the x Benchmark presented in this paper, we seek to provide an unprecedented scale of curated visual and genomic data and tasks focused specifically on discovering unknown relationships between the sorghum genotype and its expressed phenotypes.
2.2 TERRA-REF
The image and genomic data in the SGxP Benchmark was collected as part of the TERRA-REF project (9 29] The TERRA-REP
gantry system shown in Figure can monitor over an acre of crops with imaging sensors including stereo-RGB, thermal, short- and long-wave hyper- spectral cameras, and laser 3D-scanner sensors. The TERRA-REF data includes imagery from sev- cral scasons where plants in the sorghum Bioen- ergy Association Panel (BAP) (8)) were grown. The BAP is a collection of 390 sorghum lines which have shown potential for bio-energy usage, and all 390 lines were fully sequenced.
The TERRA-REF Field and Gantry- based Field Scanner in Maricopa, Arizona.
Figure 1:
The SGxP Benchmark includes imagery and genetic information curated from the original, full TERRA TERRA-REF&oe dataset to be broadly accessible by the machine learning community. It is organized to (a) simplify challenges that arose in capturing data across multiple sensors from differing capture times, sensor failures, and weather conditions, and (b) construct a dataset and evaluation protocol to make the genotype x phenotype comparison accessible to those without a strong biological background. The original TERRA-REF dataset and the derived datasets used in the SGxP Benchmark are released under CCO 1.0 Universal Public Domain Dedication license.
2.3 Sorghum and the Genotype Phenotype Relationship
Sorghum is genus of grass plants, and critical crop globally for human and livestock consump- tion, and for bioenergy. Sorghum's 's drought tolerance and ability to grow in harsh environments make it valuable crop for food security in regions with unreliable rainfall, or regions being impacted by climate change, and currently provides vital nutrition in many developing countries. Furthermore, sorghum has high potential for biofuel production its high biomass yield and ability thrive in varying environmental conditions make it a sustainable alternative to fossil fuels. A deeper under- standing of the genotype phenotype relationship - the relationship between the plant's genetics and its actual expressed traits is is crucial for breeding improved sorghum crops for food and energy pur- poses. The complex interplay between genes and expressed phenotypic traits in sorghum can impact various characteristics such as yield, drought tolerance, nutrient content, and biomass accumulation. Identifying the impact of specific genes on plants and their interaction with the environment is crucial area of research in plant biology 66. 13 39 46). and the SG x P dataset and benchmark seeks to advance the understanding of the sorghum genotype x phenotype relationship, and the ways that machine learning and computer vision can be used to uncover these relationships.
3 SGXP X Benchmark
A high level overview of the plant and sorghum genetics helps to understand the data and labels in this benchmark. Sorghum has 10 chromosomes, each of which contains genetic instructions the form of DNA these genetic instructions drive both the form and function of the plant. Sorghum is a diploid species, meaning that it has two copies of each of these 10 chromosomes. The DNA that comprises these chromosomes is composed of nucleotides (either A,C,T,G), which form sequences that direct the plant to produce specific proteins. Variations at specific locations in these sequences, can be defined by change of a single nucleotide (e.g. A is changed to G). These variations
3

---

Chromosome
Gene
Position
Known Controlled Phenotype
0UTG269200
51588525
001 G269200
51588838
Wax composition [56]
0016269200
51589143
G269200
51589435
6
006GG66770
42805319
Plant height and structure, stem length and internode length 66
25
006GG067700
42804037
006G147400
50898459
006G147400
50898536
006G147400
50898315
Plant height and structure, and sugar composition
66
006G147440
50898231
006G147400
50898523
006G147400
50898525
0U6G057866
40312463
006G004400
2697734
Flowering time and maturity 14
63
40
009G229800
57040680
Pigmentation and tannin production 63
Table 1: Details about the SNPs in the dataset with known phenotypic expression.
called single nucleotide polymorphisms (SNPs), and they are important they can alter the proteins produced by the plant, affecting its structure and performance.
Because Sorghum is diploid, a particular line of sorghum may have variation on one both copies of its chromosome. If the variation is on only one copy, that line of sorghum is termed heterozygous for that SNP; if the variation is both copies, it is called homozygous. We only consider homozygous cases to simplify the machine learning task and the biological interpretation. Additionally, while in principle there could be different variations at the same location, our case each location is unique (so there is no case where at one SNP location an A is changed to a C in some lines, and the same A is changed
G in other lines).
3.1 Tasks
The overarching task in this benchmark is a binary classification task: to predict whether plant has the reference or alternate version of a SNP based on an image or images of that plant from or more imaging modalities. The benchmark includes datasets with RGB and 3D im- ages for 15 SNPs with known genotype xecotyppeppeneeee phenooype relationships, and 2,717 SNPs with unknown genotype x phenotype relationships (the specifics of the data are detailed in Section 3.2).
Details about the 15 SNPs with known phenotypic expression can be found in Table -- these SNPs are known to control phenotypes such as leaf wax composition, plant height and structure, pigmentation, and flowering time. In prior work (50) 68). we showed that simple classification models trained on RGB images can accurately predict whether an image shows a plant with reference or alternate version of many of these SNPs. Additionally, heatmap-style explanatory visualizations from these models often focused on the same traits that the SNPs were known to control. An example of this is shown in Figure 2 (right). which shows an image of a sorghum plant that has the reference version of a SNP that controls leaf wax composition, and the corresponding visualization for why a neural network trained on the binary classification task predicted the reference label, highlighting that the network learned to focus on the waxy build-up on the plant.
We encourage machine learning researchers engaging with this benchmark to first develop and evaluate approaches on these markers with known phenotypic expression - all of these markers have visual traits that are detectable and machine learning approaches to perform binary classification task on these datasets can achieve significantly above chance performance. This makes these SNPs useful for getting comfortable with the data in the benchmark and checking the validity of approaches.
These 15 SNPs, however, are perhaps less exciting to consider than the 2,717 SNPs in the SGxP Benchmark that have unknown phenotypic expression. While many SNPs are "synonmyous" meaning that the nucleotide change does not result in an amino acid change in the protein sequence, and so no change is made to the structure, function interaction of the encoded protein with other molecules these 2,717 SNPs were identified as potential "high impact" polymorphisms. When a SNP is referred to as potentially high impact, means that the nucleotide change in that SNP has the potential to significantly alter the protein coding sequence or its function. Many of these changes may not result in obvious changes to the phenotype- - but some certainly will. Finding those SNPs and uncovering their phenotypic expression is exactly the goal of this benchmark, and will be key to better understanding the genotype x phenotype relationship in sorghum and developing new lines of sorghum that have desirable properties.

---

Figure 2: (left-top) Example RGB imagery from the dataset. (left-bottom) false color 3D line scanner image from approximately the same location and date as the RGB image. The line scanner data shown with false coloring that highlights surface normal. (right) An image of a sorghum plant that has the reference version of SNP that controls leaf wax composition, and the class activation map showing the model trained on this task learned to focus on wax buildup.
3.2 Data
The entire SGxP Benchmark consists of 543,574 RGB images and 536,153 false color 3D line scanner images from the 2017 TERRA-REF growing season. There are 105 days with data captured over the course of the 140 day growing season; some days did not have any data capture. Each sorghum cultivar in the SGxP Benchmark was grown in two spatially separated plots within the field. We provide images from both plots in our datasets, organized by cultivar.
To construct simple image dataset, RGB images that cross plot boundaries are split into multiple images, that every image contains pixels of plants from only single plot. This guarantees that all images in our benchmark only show images of single cultivar. The 3D line scanner output is also cropped to plot boundaries. We convert the raw line scanner output from a three dimensional voxel representation to two dimensional false color image, where the hue encodes the surface normal at each pixel location, and the value encodes distance from the camera (completely black pixels are locations that were occluded). An example of this false color representation of the 3D line scanner output can be seen in Figure (bottom). For convenience, we will refer to these false color images as the 3D images in the remainder of the paper. Researchers interested in using other representations of the 3D data are welcome to download it from the original TERRA-REF data (29 but should include information about their processing and data format when submitting results the leaderboard.
We use these images across all of the SNP prediction datasets. There are 15 datasets focused on SNPs with known phenotypic expression and 2.717 SNPs with unknown phenotypic expression. For every one these SNPs, we provide six datasets: RGB training and testing datasets, 3D training and testing datasets, and multimodal training and testing datasets that contain 3D>RRGB pairs. The reason we consider a multimodal benchmark in addition to benchmarks for each separate modality, is that we believe there may be interesting and complex phenotypes that observable using multiple modalities that may not be observable using only one modality (e.g., a variation in a particular SNP might produce taller and greener leaves).
For every image including in any dataset in the SGxP Benchmark, we also provide additional metadata: the sorghum cultivar and subpopulation shown the image, the TERRA-REF plot from which the image was captured, and the timestamp that the image was captured. Additional information about this metadata can found in the Appendix. Benchmark participants are encouraged to explore ways to incorporate this metadata information in their approaches.
5

---

In order to generate training and testing splits for each SNP, we find whether there are more reference or alternate cultivars. We take a randomly selected third of the cultivars from whichever label has fewer cultivars and put their images in the SNP-specific test set. We then complete the test set with images from an equal number of randomly selected cultivars of the other label. In order for a SNP to be included in the SGxP x Benchmark, we require that there are at least 20 reference and 20 alternate cultivars. This means that there is a minimum of reference and 6 alternate cultivars in each test set. We then put all other cultivars in the training set, maximizing the amount of data available for training, and guaranteeing that there is no overlap in cultivars between the training and testing sets, Figure 3 shows distributions of images and labels for different subsets of the SGxP dataset. Notably, Figure (c) shows the training data distribution of reference minus alternate labels per SNP, normalized by the total number of images in the dataset. Points that are farther to the left have more alternate images than reference images, and points that have more reference images than alternate images. This plot shows that the training datasets often have significant label imbalance, typically with more reference images than alternate images. The test sets are all balanced to have an equal number of reference and alternate images (guarantecing that the average random model achieves 50% accuracy on the binary classification task). We make sure there is no overlap in the cultivars included in the training and testing datasets. This ensures that models cannot overfit on trivial, non-biologically relevant features (such as unique patterns in the dirt or in the lighting conditions). Instead, models that generalize from the cultivars in the training datasets to the entirely different cultivars in the testing datasets are learning biologically relevant visual features.
Figure 2 shows an example pair of multimodal images. Although reasonably well aligned, the alignment between sensors is not perfect because the RGB camera and the 3D line scanner camera are physically in different parts of the gantry sensor box, and because the RGB images and 3D line scanner images are typically captured at different times of day. This lack of spatial and temporal alignment is a challenge for multi-sensor fusion approaches, and different approaches are possible.
Our approach creating simple multimodal training and testing datasets is to provide pre- computed 3D-RGB data pairs. To create the dataset we took each of the 3D images from the SNP-specific datasets and selected random RGB image from the closest date in time and the same plot, to create 3D-RGB pairs. We exclude any 3D images that do not have nearby RGB image from within 10 days. In this construction, some RGB images that are included in the SNP-specific RGB datasets may not be used, some 3D images may not have a valid RGB match, and some RGB images may be the match for multiple 3D images. More details on the multimodal dataset construction are in the Appendix. While we believe these RGB-3D pairs are useful data structure for initial exploration of multimodal sensor fusion for genotype x xphenotype discovery, we encourage practitioners utilizing this benchmark to consider other approaches that may be more complex.
Baseline Approach
We provide baseline deep learning models and results for all SNPs in the SGxP Benchmark. Prior to training task specific baseline models for each of the SNPs, we pre-train a ResNet-50 model [23) for RGB and 3D imagery, to learn useful, generalizable sorghum features. To do this pre-training, we leverage additional publicly available data from the TERRA- -EEF project - 1 10,638 RGB images and 141,113 3D scanner images from 188 cultiavrs grown in 2019, processed in the same way as the data in this benchmark. The 188 cultivars that are part of this 2019 dataset have no overlap with the ones in the SGxP P Benchmark, but have significant phenotypic variation, meaning models trained on this data learn useful visual features for tasks relating to sorghum.
The pre-trained ResNet-50 models are trained classify which of 375 field plots an image comes from using cross-entropy loss. We train one model with RGB imagery and one with 3D imagery. The models have a 1148-dimensional global average pooling layer, followed by a 128 228-dimensional fully connected layer, and then the plot level classification output. During this pre- traininigg images (RGB or 3D) are resized 512x512 pixels and randomly flipped (horizontally and vertically). We use PyTorch, training for 25 epochs with the SGD optimizer with an initial learning rate of 0.01, and a learning rate decay of 0.1 every 10 epochs. The pre-trained model takes roughly one day to train on a single 1 16GB P100 Nvidia GPU with a batch size of 30.
After pre-training, we chop off the final plot classification layer. For each SNP prediction task in the SG xP Benchmark, we then take this pre-trained model and train a single fully connected layer with two outputs, corresponding to the Reference and Alternate labels, using the SG Benchmark training datasets (because the pre- trained model was trained data from an entirely different growing season, there is no risk of data leakage or overfitting). For the modal tasks, we first concatenate the 128-dimensional output features extracted by each of the RGB and 3DD pre-trained models and

---

Images per SNP (Training)
Images per SNP (Testing)
Training Data Class Imbalance
1000
RGB
RGB
RGB
600
800
600
Requency
300
Peqquecc
400
Frequancy
400
400
200
200
200
100000
2000000
300000
400000
500000
20000
40000
80400
Bo00g
-10
Images
Images
Ref Images)
(# Ref + Images)
(h)
(c)
Training Label Distributions (RGB)
Training Label Distributions (3D)
Reference
Reference
800
Alterate
800
Altemate
frequency
600
Frequency
600
400
400
200
0
100000
200et0
300000
4p0ng9
so0000
100000
201000
300000
4000no
50c0n0
# Images
Images
Figure 3: These plots show distributions of number of images by dataset, by sensor and by label. Figure (c) in particular conveys the class imbalance in the training data, showing the distribution of reference images minus alternate images in the training data (normalized by the total number of images for that dataset). Datasets with more reference images are farther to the right, and tasks with more alternate images are farther to the left.
then train a single fully connected layer on top of those concatenated features, In order to facilitate fast training of these SNP-specific models, we first extract the 128-D output features for every image in the dataset, and then train only the final fully connected layer for each SNP classification task. We use batch size of 256 when training the SNP-specific models. Due to significant class imbalance in many of the datasets, we use a balanced sampler to guarantee a roughly equal number of Reference and Alternate examples in each training batch. We discuss the results of this baseline in Section 5 and provide links to the data used for pre-training, as well as our pre- pre-trained model and training code, at the benchmark website htot:://sorghuusnpben..
com).
5 Evaluation
For every SNP in the SGxP Benchmark, we consider two different evaluation paradigms for each of the imaging modalities: one that is per-image, and one that is per-cultivar. The evaluation metric is binary classification accuracy and per-image evaluation is straight-forward for every image in the test dataset, is the predicted SNP label correct or not? This is averaged over the entire test set (every test set in the SGxP P Benchmark has an equal number of Reference and Example images, meaning that the average random model would achieve 50% accuracy each task). We report on the results of the baseline approach described in DactionFoox the SNPs with known phenotypic expression the middle three columns of Table 2 In all cases, the baseline models achieve significantly above chance performance, with the multimodal approach often outperforming the single-modality approaches.
This per-image evaluation is valuable for three reasons. First, many baseline MI. classification approaches are designed to ingest a single image or image pair and produce a classification. Second, many variations of explainable image classification (47 48 49 51 69] operate on single images at time to highlight visual features that affected the classification result. This approach to explainability can be useful when trying to understand genotype x phenotype interactions.
The per-image evaluation, however, is not necessarily always the most appropriate. Some genetic variations may only impact the plant during a small portion of the growing season for example, causing changes to the shape, size or timing of flowers that the plant produces. These sorts of variations would not be meaningfully correlated with visual features in images from other parts of the growing season. The goal of the SGxP Benchmark is to uncover meaningful and ys- unknown
7

---

By Image
By Cultivar
SNP
RGB Only
3D Only
RGB+3D
RGB Only
3D Only
RGB+3D
Chromosome
Gene
Position
U0IG269200
51588525
0.615
0.369
0.640
0.640
0,700
0.700
00I
51588838
0.601
0.564
0.582
0.652
0.696
0.609
00IG269200
51589143
0.622
0.597
0.637
0.662
0.662
0.676
001G269200
51589435
0.640
0.606
0.665
0.781
0.750
0.812
6
006G067700
42805319
0.604
0.584
0.630
0.654
0,654
0.744
6
006G067700
42804037
0.597
0.571
0.620
0.653
0.681
0.667
6
006G147400
50898459
0,539
0,536
0,578
0,553
0.605
0,684
6
006G147400
50898536
0.505
0.569
0.578
0.500
0.684
0.658
6
006G147400
50898315
0.527
0,584
0.600
0.553
0.658
0.711
6
006G147400
50898231
0.617
0.578
0.650
0.735
0.618
0.765
006G147400
50898523
0.578
0,568
0.626
0.553
0,579
0.632
006G147400
50898525
0.625
0.576
0.650
0.763
0.658
0.842
006G057866
40312463
0.670
0.543
0.659
0.850
0,650
0.850
6
006G004400
2697734
0.624
0.630
0.709
0.763
0.816
0.842
9
009G229800
57040680
0,587
0,590
0.636
0.611
0,704
0.648
Table 2: Binary classification performance of the baseline approach on the SNPs with known
genotype x phenotype relationships. Performance is reported for each sensor separately, and when using sensor fusion, for both individual images and when aggregated by cultivar.
venotypexphenotype relationships, including those that may not be observable in every image in the benchmark. To support this goal, we provide more flexible, per-cultivar evaluation.
In the per-cultivar evaluation, researchers provide single prediction for each of the cultivars in the test set (as opposed to each image). As in the per-image evaluation, the number of cultivars that have the Reference versus Alternate labels is the same, so the average random model would achieve 50% accuracy. We note that the number of cultivars per label for some SNPs can be relatively small - e.g. about 10 meaning that expect there to be more noise in the average accuracy of models on this evaluation. In the results shown in the last three columns of Table 2] we used the simplest approach to per-cultivar evaluation: we decided whether cultivar is Reference or Alternate for particular SNP by taking the mode of the predictions from all of the images that belong that cultivar. Researchers have flexibility in how they produce the per-cultivar prediction. They could, for example, only consider images from a particular portion of the growing season, or weight images from different parts of the growing season differently. They could develop model architectures that consider multiple images simultancously. In general. the per- per-cultivar evaluation provides researchers with the ability develop models that explain complex genotype xppenotype relationships sorghum.
One way that the SGxP Benchmark varies from traditional benchmarks is that it contains significant number of tasks where is unknown if above-chance performance is even possible. While we have structured the XG Benchmark to include SNPs that are already well understood for proof of concept and validation, in addition to the SNPs with unknown genotype x phenotype interations that are potentially "high impact", in reality, many of these unknown SNPs will have minimal impact. While this uncertainty makes our benchmark unique in machine learning datasets and competitions, it is not unique in science the goal of this benchmark is discovery.
Web Page, Leaderboards & Discussion Forums
To support that discovery, we provide web interface that has not only traditional leaderboards where researchers can compare their numerical performance to others, but also discussion forums where they can interact with plant biologists, sharing about their approaches, what they think their models are learning, and participate in back and forth dialogues to drive this discovery of unknown genotype x phenotype relationships in sorghum. The website sorghunsnpben chmark....cccc..ooooo....mmmkk.k com. Screenshots of this web interface can be found in Figures 4 and Figure (left) shows the main page for the SGxP Benchmark. This page includes leaderboards for each of the SNPs in the Benchmark, with their current top accuracy by image and by cultivar, as well as the attribution for those leading results. Users can switch between leaderboards for each of the different imaging modalities. By clicking on the SNP name on these leaderboards, users are taken to detail page for that SNP (seen Figure ( (right)). This page includes metadata for the SNP (its chromosome, gene and position on the chromosome), details about each of the training and testing datasets including links to download the data, SNP-specific leaderboards for each of the imaging modalities, and links to any Discussion Board posts that are about that SNP.
be found at http::/ /7
8

---

Welcome Sorghum Benchmark Leaderboard
Traiting
erok Co Ciboriroodd
CHEED
Mellerend R0044 DOW?
sou4)
Londorborrd
Whi0 C.50 ClAcs
Figure 4: (left) The overall leaderboard for the SG Benchmark, showing the current top results for different SNPs. (right) The detail page for one of the SNPs, including metadata, dataset information, the current SNP-specific leaderboards, and discussion board posts about the SNP.
4 (right). In Figure 5 we show one such example discussion for the SNP highlighted in Figure This SNP interesting because the baseline model achieved well above chance accuracy in all modalities, but also saw a significant improvement from using the multimodal data. We show an example discussion, where we posted about this interesting behavior and one of the biologists on our team replied with information from the sorghum literature about the SNP and a follow up asking about visual features that the machine learning model was focusing on. This back- and-forth engagement is precisely the goal of the SGxP Benchmark to drive the discovery of genotype x phenotype relationships that can lead new understanding about sorghum and direct development of new and improved sorghum crops.
Limitations
There are several possible limitations of this work. The focus on sorghum may limit the generaliz- ability of the findings. While we believe the structure of the machine learning approaches that on this dataset would generalize, other crops may exhibit unique genetic and phenotypic characteristics that require different approaches. The data represents a snapshot of sorghum variants grown specific field conditions, which may introduce biases and limitations in capturing the full range of genetic and environmental factors that influence sorghum phenotypes. The benchmark is also limited to focusing on above ground traits that are visible in RGB and 3D data this ignores any traits that are expressed below group or are not visible in these imaging modalities. It also is setup to consider the genetic impact of individual SNPs and does not consider more complex genetic interactions.
The SNPs chosen were ones that have a potential biological impact because they change the amino acid that the DNA encodes. However, that amino acid may not be important the function of the plant, so many. perhaps most of these SNPs will have no biological impact. This limits the value of these tasks for researchers focussed on demonstrating that generic machine learning algorithms work in this domain. These tasks may be more compelling for researchers that want to be involved in discovering biological function; in this case we provide the discussion forum because an machine learning algorithm that scores well only goes part of the way to discovering the phenotype changed by that SNP, and additional work with visualizations or other explainability approaches may be needed to discover the genotype phenootyp relationship.
Conclusion
This Sorghum Genotype Phenotype (SGxP) Benchmark provides a resource for exploring the relationships between genetic data and phenotypic expression sorghum. By integrating large-scale, multimodal imagery and genetic information from large number of sorghum species grown in real-world field conditions, this benchmark enables researchers to uncover significant insights into the use of sorghum in food and energy production systems. The benchmark includes baseline deep learning approaches, allowing for comparison to future algorithms that learn data-driven features relevant to understanding sorghum phenotypes. We create combined leaderboard and discussion

---

SNP Benchmark Forum
Interesting does better with multimodal data
General Ohooss on pneecrtaeto oo
A
The SMP adtic D0000044000 2, 28007744 ananwoding usnng our bood no asprouds we gat 50.1 acconaty using IIC8 caiks
12.5 uting dats, and jumo up 073M using musimmccl data.
pharsdyp'
Vira Lan rots ooe
Paply
Dudra
Homte thakeer warroisse
N
eestliannu Posstay: The you've identitedd localed kranee azagrum batartt (VB)) leck Theae Vi lsbelle through
eongrae
when the brn.
Chis makes serght more
wigh and holca off
Powerity ring by carsing the act diy of So5rd', a gene that ancoureges fovering, and other diviler genes Ir' worth noding that the actiety
the stochol gene adfianced by light allgns the flythrr a the plants inte biologiial joinaslarr clock.
sartope
somnthing
Freb
Dodt
Figure 5: A discussion board post about the SNP shown in Figure
forum to further facilitate collaboration and knowledge-sharing among machine learning practitioners and sorghum-focused plant biologists.
The SGxP Benchmark focuses specifically on sorghum, but we believe that algorithms and methodologies that effective on this benchmark have the potential to apply to other crops and organisms with similar scales of imagery and genetic data. In building the benchmark, we have worked to make the tasks directly relevant understanding some part of the relationship between genotypes and phenotypes, to simplify and catalyze interdisciplinary interactions within this research domain. In doing so, we hope that the SG Benchmark thus serves as catalyst to unraveling the intricate connections between genetic information and observable traits in sorghum and beyond, with the ultimate goal of addressing global challenges food production, ecosystem resilience, and sustainable development.
Acknowledgments and Disclosure of Funding
This work was funded by the Advanced Research Projects Agency Energy (ARPA-E) awards DE-AR0000594 and DE-AR0001 101, by seed grant from the Agricultural Genome to Phenome Initiative (AG2PI) (USDA-NIFA awards: 0022-70412-3844
0021-70412-35233 and 2020-70412.
32615), and by Google's 's Research Scholar Program ("Using Deep Learning Uncover Unknown mmnotppexphenotee Relationships in Crops").
References
[1] Shubhra Aich and Ian Stavness. "Leaf counting with deep convolutional and deconvolutional networks". In: JEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). 2017, pp. 2080-2089.
[2] Basavaraj S Anami, Naveen N Malvade, and Surendra Palaiah. "Deep learning approach for recognition and classification of yield affecting paddy crop stresses using field images". In: Artificial Intelligence in Agriculture 4 (2020), pp. 12-20.
[3] Belal AM Ashqar, Bassem S Abu-Nasser, and Samy Abu-Naser. "Plant seedlings classi- fication using deep learning". In: International Journal of Academic Information Systems Research (IJAISR) (2019).
[4] Suchet Bargoti and James Underwood. "Deep fruit detection in orchards". In: IEEE Interna- tional Conference Robotics and Automation (ICRA). IEEF, 2017, pp. 3626-3633.
[5] Pierre Barré, Ben C Stöver, Kai F Müller, and Volker Steinhage. "LeafNet: A computer vision system for automatic plant species identification.. In: Ecological Informatics 40 (2017), pp. 50-56.
[6] Barry R Bochner. "New technologies assess genotype-phenotype relationships". In: Nature Reviews Genetics 4.4 (2003), pp. 309-314.
[7] Richard E Boyles, Zachary W Brenton, and Stephen Kresovich. "Genetic and genomic re- sources of sorghum to connect genotype with phenotype in contrasting environments". In: The Plant Journal 97.1 (2019), pp. 19-39.
10

---

[8] Zachary W Brenton, Elizabeth A Cooper, Mathew T Myers, Richard E Boyles, Nadia Shakoor, Kelsey J Zielinski, Bradley Rauh, William C Bridges, Geoffrey P Morris, and Stephen Kresovich. "A genomic resource for the development, improvement, and exploitation of sorghum for bioenergy". In: Genetics 204.1 (2016), pp. 21-33.
[9] Maxwell Burnette, Rob Kooper, D. Maloney, Gareth S. Rohde, Jeffrey A. Terstriep, Craig Willis, Noah Fahlgren, Todd Mockler, Maria Newcomb, Vasit Sagan, Pedro Andrade-Sanchess Nadia Shakoor, Paheding Sidike, Rick Ward, and David LeBauer. "TERRA-REF Data Pro- cessing Infrastructure". In: Proceedings of the Practice and Experience on Advanced Research Computing. Ed. by Sergiu Sanielevici. ACM, 2018, 27:1-27:7. DOT: 10. 1145/ 3219104
3219152
[10] Sujata Butte, Aleksandar Vakanski, Kasia Duellman, Haotian Wang, and Amin Mirkouei. "Potato Crop Stress Identification in Aerial Images using Deep Saaninggbaseed Object Detec- tion". In: arXiv preprint arXiv: /06. 07770 (2021).
[11] Narendra Singh Chandel, Subir Kumar Chakraborty, Yogesh Anand Rajwade, Kumkum Dubey, Mukesh K Tiwari, and Dilip Jat. "Identifying crop water stress using deep learning models".
In: Neural Computing and Applications 33.10 (2021), pp. 5353-5367.
[12] Yang Chen, Won Suk Lec, Hao Gan, Natalia Peres, Clyde Fraisse, Yanchao Zhang, and Yong He. "Strawberry yield prediction based on deep neural network using high nigh-resolution acrial orthoimages". In: Remote Sensing 11.13 (2019), p. 1584.
[13] Joshua N Cobb, Genevieve DeClerck, Anthony Greenberg, Randy Clark, and Susan McCouch. "Next-generation phenotyping; requirements and strategies for enhancing our understanding of genotype-phenotype relationships and its relevance to crop improvement". In: Theoretical and Applied Genetics 126.4 (2013), pp. 867-887.
[14] Hugo E. Cuevas, Chengbo Zhou, Haibao Tang, Prashant P. Khadke, Sayan Das, Yann-Rong Lin, Zhengxiang Ge, Thomas Clemente, Hari Upadhyaya, C. Thomas Hash, and Andrew H. Paterson. "The Evolution of Photoperiod Insensitive Flowering in Sorghum. A Genomic Model for Panicoid Grasses". In: Molecular Biology and Evolution 33.9 (June 2016). pp. 2417-2428. ISSN: 0737-4038. DOT: 10. 1tw.1099/mmoo htt//mmlbbvevbe/ wwwpb///mmwwwmmm
snppperiod-Insensitive
[15] Etienne David, Mario Serouart, Daniel Smith, Simon Madec, Kaaviya Velumani, Shouyang Liu, Xu Wang. Francisco Pinto, Shahameh Shafiee, Izzat SA Tahir, et al. "Global wheat head detection 2021: An improved dataset for benchmarking wheat head detection methods". In: Plant Phenomics (2021).
[16] Andrei Dobrescu, Mario Valerio Giuffrida, and Sotirios A Tsaftaris. "Leveraging multiple datasets for deep leaf counting". In: IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). 2017, pp. 2072-2079)
[17] Noah Fahlgren, Maximilian Feldman, Malia A Gchan, Melinda S Wilson, Christine Shyu, Douglas W Bryant, Steven T Hill, Colton| J McEntee, Sankalpi N Warnasooryya, Indrajit Kumar, al. "A versatile phenotyping system and analytics platform reveals diverse temporal responses to water availability in Setaria". In: Molecular plant 8.10 (2015), pp. 1520-1535.
[18] Mitchell J Feldmann, Joseph L Gage, Sarah D Turner- Turner-Hisson,, and Jordan R Ubbens. "Images carried before the fire: The power, promise, and responsibility of latent phenotyping in plants". In: The Plant Phenome Journal 4.1 (2021), c20023.
[19] Konstantinos P Ferentinos. "Deep learning models for plant disease detection and diagnosis". In: Computers and Electronics in Agriculture 145 (2018), pp. 311-318.
[20] Ronald A Fisher. "TThe use of multiple measurements in taxonomic problems". In: Annals of eugenics 7.2 (1936), pp. 179-188.
[21] Mario Valerio Giuffrida. Peter Doerner, and Sotirios A Tsaftaris. "Pheno-deep counter: unified and versatile deep learning architecture for leaf counting". In: vol. 96. 4. Wiley Online Library, 2018, pp. 880-890.
[22] Nicolai Häni, Pravakar Roy, and Volkan Isler. "MinneApple: a benchmark dataset for apple detection and segmentation". In: IEEE Robotics and Automation Letters 5.2 (2020), pp. 852- 858.
[23] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. "Deep Residual Learning for Image Recognition". In: Proceedings of the Conference on Computer Vision and Pattern Recognition. 2016, pp. 770-778.
11

---

[24] Ahmad
Mohsen Sardari Zarchi, Sima Emadi, and Gholamreza Zarei. "An efficient deep learning model for cultivar identification of pistachio tree". In: British Food Journal 11 (2021), pp.
[25] Josie L. Hilley, Brock D. Weers, Sandra K. Truong, Ryan F. McCormick, Ashley J. Mattison,
Brian A. McKinley, Daryl T. Morishige, and John E. Mullet. "Sorghum Dw2 Encodes a Protein Kinase Regulator of Stem Internode Length". In: Scientific Reports 7.1 (July 2017).
DOI: 101:10..038///
[26]
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. "Segment Anything". In: arXiv:2304. 33044044302 (2023).
[27] Anand Koirala, KB Walsh, Zhenglin Wang, and C McCarthy. "Deep learning for real-time fruit detection and orchard fruit load estimation: Benchmarking of 'MangoYOLO"". In: Precision Agriculurre 20.6 (2019), pp. 1107-1135.
[28] Neeraj Kumar, Peter N Belhumeur, Arijit Biswas, David W Jacobs, W John Kress, Ida C Lopez, and João VB Soares. "Leafsnap: A computer vision system for automatic plant species identification". In: Computer Aision-ECCV 2012: 12th European Conference on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part II 12. Springer. 2012, pp. 502- 516.
[29] David LeBauer, Maxwell A. Burnette, Jeffrey Demieville, Noah Fahlgren, Andrew N. French, Roman Garnett, Zhenbin Hu, Kimberly Huynh, Rob Kooper, Zongyang Li, Maitiniyazi Maimaitijiang Jerome Mao, Todd C. Mockler, Geoffrey Morris, Maria Newcomb, Michael J Ottman, Philip Ozersky, Sidike Paheding, Duke Pauli, Robert Pless, Wei Qin, Kristina Riemer, Gareth Scott Rohde, William I.. Rooney, Vasit Sagan, Nadia Shakoor, Abby Stylianou, Kelly Thorp, Richard Ward, Jeffrey W White, Craig Willis, and Charles S Zender. TERRA-REF, An Open Reference Data Set From High Resolution Genomics, Phenomics, and Imaging Sensors. 2020.
dry.d. 4b8gtht99
[30] S.H. Lee, C.S. Chan, S. Mayo, and P. Remagnino. "How Deep Learning Extracts and Learns Leaf Features for Plant Classification". In: Pattern Recognition 71 (2017), pp. 1-13.
[31] Marcus Guozong Lim and Joon Huang Chuah. "Durian Types Recognition Using Deep Learning Techniques". In: 2018 9th IEEE Control and System Graduate Research Colloquium (ICSGRC). IEEE. 2018, pp. 183-187.
[32] Yang Liu, Duolin Wang, Fei Hc, Juexin Wang, Trupti Joshi, and Dong Xu. "Phenotype prediction and genome-wide association study using deep convolutional neural network of soybean". In: Frontiers in Genetics 10 (2019), p. 1091.
[33] Riccardo Lutio, John Y Park, Kimberly Watson, Stefano D' Aronco, Jan Wegner, Jan J Wieringa, Melissa Tulig, Richard L Pyle, Timothy J Gallaher, Gillian Brown, al. "The Herbarium 2021 Half-Earth Challenge Dataset and Machine Learning Competition". In: Frontiers in Plant Science 12 (2022). p. 3320.
[34] Maitiniyazi Maimaitijiang, Vasit Sagan, Paheding Sidike, Sean Hartling, Flavio Esposito, and Felix B Fritshii. "Soybean yield prediction from UAV using multimodal data fusion and deep learning". In: Remote Sensing of Environment 237 (2020), p. 111599.
[35]
Chenyong Miao, Alice Guo, Addie M Thompson, Jinliang Yang. Yufeng Ge, and James C Schnable. "Automation of leaf counting in maize and sorghum using deep learning". In: The Plant Phenome Journal 4.1 (2021), e20022.
[36] Krzysztof Mikolajczak, Piotr Ogrodowiez, Hanna Kwwiek-Kupzyynsee Kathleen Weigelt- Fischer, Srinivasa Reddy Mothukuri, Astrid Junker, Thomas Altmann, Karolina Krystkowiak, Tadeusz Adamski, Maria Surma, et al. "Image phenotyping of spring barley (Hordeum vulgare L.) RIL population under drought: selection of traits and biological interpretation". In: Frontiers in Plant Science (2020), p. 743.
[37] Massimo Minervini, Andreas Fischbach, Hanno Scharr, and Sotirios A Tsaftaris. "Finely- grained annotated datasets for image-based plant phenotyping". In: Pattern recognition letters 81 (2016), pp. 80-89.
[38] Sharada P Mohanty, David P Hughes, and Marcel Salathé. "Using deep learning for image- based plant disease detection"". In: Frontiers in Plant Science 7 (2016), P. 1419.
12

---

[39] Ravi V Mural, Marcin Grzybowski, Chenyong Miao, Alyssa Damke, Sirjan Sapkota, Richard E Boyles, Maria Salas Fernandez, Patrick S Schnable, Brandi Sigmon, Stephen Kresovich, et al. "Meta-analysis identifies pleiotropic loci controlling phenotypic trade-offs in sorghum". In: Genetics 218.3 (2021), iyab087.
[40] Rebecca L. Murphy, Daryl T. Morishige, Jeff A. Brady, William L. Rooney, Shanshan Yang. Patricia E. Klein, and John E. Mullet. "Ghd7 (Ma6) Represses Sorghum Flowering in Long
Days: Ghd7 Alleles Enhance Biomass Accumulation and Grain Production". In: The Plant
Genome 7.2 (2014). DOT: 1.....835/ppp. 2083//p.anttgenom2221....3. 11. 0040
[41] Petteri Nevavuori, Nathaniel Narra, and Tarmo Lipping. "Crop yield prediction with deep convolutional neural networks". In: Computers and electronics in agriculture 163 (2019),
p. 104859.
[42] Yutaro Osako. Hisayo Yamane, Shu- Yen Lin, Po-An Chen, and Ryutaro Tao. "Cultivar dis- crimination of litchi fruit images using deep learning", In: Scientia Horticulturae 269 (2020),
p. 109360.
[43] C. Ren, Justin Dulay, Gregory Rolwes, D. Pauli. N. Shakoor, and Abby Stylianou. "Multi- resolution Outlier Pooling for Sorghum Classification". In: Agriculture-Visicee Workshop in IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) (2021).
[44] Inkyu Sa, Zongyuan Ge, Feras Dayoub, Ben Uperoft, Tristan Perez, and Chris McCool. "Deepfruits: A fruit detection system using deep neural networks". In: Sensors 16.8 (2016),
p. 1222.
[45] Karansher S. Sandhu, Dennis N. Lozada, Zhiwu Zhang, Michael O. Pumphrey, and Arron H. Carter. "Deep Learning for Predicting Complex Traits in Spring Wheat Breeding Program". In: Frontiers in Plant Science 11 (2021), P. 2084. ISSN: 1664-462X.
DOI: 10. 3389/ fpls
2020.613332
[46] Jennifer Schweitzer, Joseph K Bailey, Dylan G Fischer, Carri LeRoy, Eric V Lonsdorf, Thomas G Whitham, and Stephen C Hart. miimt-solllmiccrooraaiinm interactions: heritable relationship between plant genotype and associated soil microorganisms", In: Ecology 89.3 (2008), pp. 773-781.
[47] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. "Grad-cam: Visual explanations from deep networks via gradient- based localization". In: Proceedings of the International Conference on Computer Vision. 2017, pp. 618-626.
[48] Avanti Shrikumar, Peyton Greenside, Anna Shcherbina, and Anshul Kundaje. "Not Just a Black Box: Learning Important Features Through Propagating Activation Differences". In: CoRR abs/16055 1605.017111 (2016). arXiv: 16055. 01713 URL: http:://a//// org/ abs/1605 01713 [49] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. "Deep inside convolutional net- works: Visualising image classification models and saliency maps". ". In: In Workshop at Inter- national Conference on Learning Representations. Citeseer. 2014.
[50] Abby Stylianou, Robert Pless, Nadia Shakoor, and Todd Mockler: "Classification and Visual- ization of Genotype x Phenotype Interactions in Biomass Sorghum".. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, Computer Vision Problems in Plant Phenotyping and Agriculture Workshop. Oct. 2021.
[51] Mukund Sundararajan, Ankur Taly, and Qiqi Yan. "Axiomatic Attribution for Deep Networks". In: ed. by Doina Precup and Yee Whye Teh. Vol. 70. Proceedings of Machine Learning Research. PMLR, June 2017, pp. 3319-3328. URL: /ttt:://proceeeeee 770/sundararaa aa....nnnnaa..............
mlr press/
[52] Edna Chebet Too, Li Yujian, Sam Njuki, and Liu Yingchun. "A comparative study of fine- tuning deep learning models for plant disease identification". In: Computers and Electronics in Agriculture 161 (2019), pp. 272-279.
[53] Michael Tross, Mathieu Gaillard, Mackenzie Zwiener, Chenyong Miao, Ryleigh Grove, Bosheng Li, Bedrich Benes, and James C Schnable. "3D reconstruction identifies loci linked to variation in angle of individual sorghum leaves". , In: PeerJ 9 (2021), e12628.
[54] Jordan Ubbens, Mikolaj Cieslak, Przemyslaw Prusinkiewicz. Isobel Parkin, Jana Ebersbach, and Ian Stavness. "Latent space phenotyping: automatic maage-based phenotyping for treatment studies".. In: Plant Phenomics 2020 (2020).
13

---

[55] Jordan Ubbens, Mikolaj Cieslak, Przemyslaw Prusinkiewicz, and lan Stavness. "The use of plant models in deep learning: an application leaf counting in rosette plants". In: Plant methods 14.1 (2018), pp. 1-10.
[56] Anurag Uttam, Praveen Madgula, Yechuri Rao, Vilas Tonapi, and Ragimasalawada Mad- husudhana. "Molecular mapping and candidate gene analysis of a new epicuticular wax locus in sorghum (Sorghum bicolor L. Moench)". In: Theoretical and Applied Genetics 130 (Oet. 2017).
[57] Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. "The iNaturalist species classification and detection dataset". In: Proceedings of the Conference on Computer Vision and Pattern Recognition. 2018. pp. 8769 8778.
[58] Shaohua Wan and Sotirios Goudos. "Faster R-CNN for multi-class fruit detection using a robotic vision system". In: Computer Networks 168 (2020), p. 107036.
[59] Anna X Wang, Caelin Tran, Nikhil Desai, David Lobell, and Stefano Ermon. "Deep transfer learning for crop yield prediction with remote sensing data". In: Proceedings of the 1st ACM SIGCAS Conference on Computing and Sustainable Societies. 2018, pp. 1-5.
[60] Bin Wang, Yongsheng Gao, Xiaohui Yuan, Shengwu Xiong, and Xianzhong Feng. "From species to cultivar: Soybean cultivar recognition using joint leaf image patterns by multiscale sliding chord matching". In: Biosystems Engineering 194 (2020), pp. 99-111.
[61] Guan Wang, Yu Sun, and Jianxin Wang. "Automatic image-based based plant disease severity estimation using deep learning". In: Computational Intelligence and Neuroscience 2017 (2017).
[62] Hai Wang, Emre Cimen, Nisha Singh, and Edward Buckler. "Deep learning for plant genomics and crop improvement". In: Current Opinion Plant Biology 54 (2020), pp. 34 41. ISSN: 1369-5266. DOT: 10. 1016/j pbi 2019. 12. 010
[63] Yan Wang. Lubin Tan, Yongcai Fu, Zuofeng Zhu, Fengxia Liu, Chuanqing Sun, and Hongwei Cai. "Molecular evolution of the sorghum maturity gene Ma3". In: Plos one 10.5 (2015), e0124435.
[64] Stephen Gang Wu, Forrest Sheng Bao, Eric You Xu, Yu-Xuan Wang, Yi-Fan Chang, and Qiao-Liang Xiang. "A Leaf Recognition Algorithm for Plant classification Using Probabilistic Neural Network". In: IEEE 7th International Symposium on Signal Processing and Information Technology. Dec. 2007.
[65] Yuye Wu, Xianran Li, Wenwen Xiang. Chengsong Zhu, Zhongwei Lin, Yun Wu, Jiarui Li, Satchidanand Pandravada, Dustan D. Ridder, Guihuaa Bai, Ming L. Wang, Harold N. Trick, Scott R. Bean, Mitchell R. Tuinstra, Tesfaye T. Tesso, and Jianming Yu. "Presence of tannins in sorghum grains is conditioned by different natural alleles of Tannin 1". In: Proceedings of the National Academy of Sciences 109.26 (2012), pp. 10281-
ISSN: 0027-8424. DOI:
20.1073/pmms pnas. 1201700109
[66] Jingnu Xia, Yunjun Zhao, Payne Burks, Markus Pauly, and Patrick J. Brown. "A sorghum NAC gene is associated with variation in biomass properties and yield potential". In: Plant Direct 2.7 (2018), e00070. DOI: 10. 1002 1002/p1d3...p/ .0.
[67] Miki Yamaguchi, Haruka Fujimoto, Ko Hirano, Satoko Araki-Nakamura, Kozue Ohmae- Shinohara. Akihiro Fujii, Masako Tsunashima, Xian Song. Yusuke Ito, Rie Nagae, Jianzhong wu, Hiroshi Mizuno, Jun-Ichi Yonemaru, Takashi Matsumoto, Hidemi Kitano, Makoto Mat- suoka, Shigemitsu Kasuga, and Takashi Sazuka. "Sorghum 1w1, an agronomically important gene for lodging resistance, encodes novel protein involved in cell proliferation". In: Scientific Reports 6 (June 2016), P. 28366. DOI: / 1038 /008///rep2366
[68] Zeyu Zhang, Madison Pope. Nadia Shakoor, Robert Pless, Todd C. Mockler, and Abby Stylianou. "Comparing Deep Learning Approaches for Understanding Genotype x Phenotype Interactions in Biomass Sorghum". In: Frontiers in Artificial Intelligence 5 (June 2022).
[69] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. "Learn- ing Deep Features for Discriminative Localization", In: Proceedings of the Conference on Computer Vision and Pattern Recognition. June 2016.
14

---

Checklist
1.For all authors...
(a)Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) you describe the limitations of your work? [Yes] See Section 7
(c)Did you discuss any potential negative societal impacts of your work? [No] Due to
nature of this benchmark (focusing on understanding the genotype menotypexphenotype rela- tionship in sorghum), potential negative societal impacts are limited.
(d)Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2.If you are including theoretical results...
(a)Did you state the full set of assumptions of all theoretical results? [N/A] (b)Did you include complete proofs of all theoretical results? [N/A]
3.If you experiments (e.g. for benchmarks)...
(a) you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] We reference the benchmark website soorghumnmbenchheg paper. It includes pointers to all code, data and instructions.
com) multiple times throughout the
(b)Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 4
Di) you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [No] For the purpose of the baseline, and because we report results on many related problems that give some intuition about the variance, we did not judge the benefits of retraining the models to be worth the power consumption.
(d)Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section 4
4.1f you are using existing assets (c.g., code, data, models) or curating/releasing new assets... (a)If your work uses existing assets, did you cite the creators? [Yes] See Sec- tions and 3.2
(b)Did you mention the license of the assets? [Yes] See Section 2.2
(c) you include any assets either in the supplemental material URL? [Yes]
Our curated datasets, pretrained models, leaderboard and discussion forums can be found at soorghunsnpbenchhark. com
(d)Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
De) you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A]
5.1f you used crowdsourcing or conducted research with human subjects...
(a)Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
(b)Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
Did you include the estimated hourly wage paid to participants and the total amount
spent participant compensation? [N/A]
15


Reviews:
[{"review_id": "tjSSf942zu", "reviewer": "Reviewer_VdiJ", "summary": "", "questions": "", "limitations": "1. Some SNPs might have no phenotypic impact but be correlated with another SNP that does have a phenotypic output. For example, a mutation at SNP #123 might co-occur with a mutation at SNP #2144 because they both happen to occur in the same cultivars. These correlations will cause spurious conclusions about of the impact of some of the SNPs. (This limitation is not acknowledged in the paper.)\n\n2. Once a model achieves a high accuracy, the task of determining which phenotypic features it has detected is non-trivial. Although having a model achieve high-performance at classifying an SNP is a step towards determining the phenotypic changes that the SNP presents, determining what the phenotype is from this model is a significant task. The complexity of this is not really acknowledged in the paper --- trying to go from a performant model to the phenotype that is presented is a serious undertaking that will require the manual labour of a data scientist; it will have to be applied to selected models on selected SNPs only a handful of times, and can not be automatically rolled across the entire dataset of thousands of SNPs. In order to maximize the utility of this manual post-hoc model analysis, it is important that the benchmark is very good at finding models and SNPs that are most worthy of manual investigation.\n\n3. Some genotypes may result in phenotypic changes which are not visible and hence can not be discovered by this dataset. This may include phenotypes of high interest, such as changes to the root structure that improve the aridity-resistance of the plant. (This limitation is not acknowledged in the paper.)\n\n4. While the goals are noble, I do not anticipate ML researchers participating in the discussion forum. The discussion forum may add value to biologists trying to understand sorghum SNPs, but not as much to ML researchers, hence I do not expect them to participate.\n\n5. In general, whilst there is value to the sorghum biology community, I'm struggling to see the utility of this dataset to machine learning researchers. Why would an ML researcher use this dataset instead of an existing fine-grained image classification dataset? Just because ML can be applied to a dataset does not inherently mean that dataset is of interest to the ML community (and the ML community is the target audience of this conference track).", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. In regards to the ethical and social implications, this is a strong positive point for the dataset. A better understanding of the relationship between sorghum genotypes and phenotypes would clearly be of societal benefit.\n\n2. The dataset has clear relevance to those interested in studying genotype/phenotype relationships, especially in the context of plants.\n\n3. The dataset combines RGB images, 3D scans, and DNA information --- an interesting combination of modalities. There are over half a million RGB images and half a million 3D scan images, which is a substantial amount of annotated data.\n\n4. Using binary tests with balanced test sets makes it simple to implement an evaluation and understand the resulting accuracy of an individual test.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "qLuMf1FKJG", "reviewer": "Reviewer_aqR6", "summary": "", "questions": "", "limitations": "- The authors do point out the limitations of the dataset being constrained to a single crop in a single season in a single field site. While this constrained dataset could still be useful, it's not clear how the models or dataset could be useful beyond this single dataset. The impact could be more convincing if it were clear that there was a large community of researchers who would be ready to interact with this discussion forum beyond their research group (or the discussion forum connected to some existing community of researchers). ", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "- The authors did a good job explaining the plant science background necessary to understand the dataset in a concise way that doesn't overwhelm the reader with domain-specific details. \n- The paper is overall well organized and easy to read. \n- I like the idea of providing a discussion forum to try to promote collaborative discussions between users and beneficiaries of the benchmark, and I appreciated the anecdote about a researcher from their team using the discussion forum in the way it was intended.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "yBCRYUccUr", "reviewer": "Reviewer_J6ZQ", "summary": "", "questions": "", "limitations": "See above. The other limitations raised are good to highlight and reasonable, and certainly do not preclude publication.", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "There are many positives to this submission:\n\n* The dataset provided here is extremely rich in imagery and genetics, providing many opportunities for vision and multimodal research.\n* The data is extremely permissively licensed (CC0).\n* The leaderboard and forum enable engagement from researchers working on the dataset.\n* Improving crop yields and resistance to climate-induced stressors is a problem of great importance.\n* The tasks are clearly described, and data is described in a way that is accessible to non-geneticists.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "1twFDKd58Z", "reviewer": "Reviewer_XBbG", "summary": "", "questions": "", "limitations": "One limitation of the dataset is that the 3D and RGB images are not completely temporally and spatially aligned. The authors specifically address this limitation in section 3.2 and explain their rationale for choosing the closest possible image pairs. This rationale seems appropriate and reasonable, and the approach is made very clear.\n\nI don’t see any other clear limitations which the authors have not addressed.", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "This is a rich and interesting dataset with the potential to drive the use of machine learning to make interesting discoveries about previously undiscovered phenotypes expressed by SNPs. I am not familiar with with the TERRA-REF project from which the dataset is drawn, so I’m unable to make a judgement as to how this dataset differs from other related datasets released from the same project, but the dataset and problem as stated are interesting an appear to be very relevant to the community.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "seDPG6FuAE", "reviewer": "Reviewer_w226", "summary": "", "questions": "", "limitations": "The main limitation of this dataset is constraint of genomic-level information to SNP. Better reasoning of the use of SNP compared to other genetic markers would increase the quality of this work. Release of the whole gene sequence data would add more value to this dataset, but not required.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "The dataset is incredibly unique and contain rich, fine-grained information for target plants, with RGB, 3D, and SNP data. The availability of data is good, with a dedicated webpage for the dataset. The dataset is available for commercial and academic usage under the Creative Commons Attribution 1.0 license, making it ideal for a wide range of research and applications in academia and industry.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: This paper introduces SG × P (Sorghum Genotype) × The Phenotype Prediction Dataset and Benchmark benchmark provides a large-scale field trial dataset, including large scale high-precision hyper-spectral images and corresponding genomic data. Researchers can use this benchmark test to evaluate phenotype prediction methods based on deep learning. The paper also introduces the organizational structure, evaluation indicators, and performance baseline of benchmark testing.
Decision: Accept (Poster)