SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Our approach to creating simple multimodal training and testing datasets is to provide precomputed 3D-RGB data pairs. To create the dataset we took each of the 3D images from the SNP-specific datasets and selected a random RGB image from the closest date in time and the same plot, to create 3D-RGB pairs. We exclude any 3D images that do not have a nearby RGB image from within 10 days. In this construction, some RGB images that are included in the SNP-specific RGB datasets may not be used, some 3D images may not have a valid RGB match, and some RGB images may be the match for multiple 3D images. More details on the multimodal dataset construction are in the Appendix. While we believe these RGB-3D pairs are a useful data structure for initial exploration of multimodal sensor fusion for genotype  $\times$  phenotype discovery, we encourage practitioners utilizing this benchmark to consider other approaches that may be more complex.
4 Baseline Approach
We provide baseline deep learning models and results for all SNPs in the  $\mathrm{SG}\times \mathrm{P}$  Benchmark. Prior to training task specific baseline models for each of the SNPs, we pre-train a ResNet-50 model [23] for RGB and 3D imagery, to learn useful, generalizable sorghum features. To do this pre-training, we leverage additional publicly available data from the TERRA-REF project - 110,638 RGB images and 141,113 3D scanner images from 188 cultiavs grown in 2019, processed in the same way as the data in this benchmark. The 188 cultivars that are part of this 2019 dataset have no overlap with the ones in the  $\mathrm{SG}\times \mathrm{P}$  Benchmark, but have significant phenotypic variation, meaning models trained on this data learn useful visual features for tasks relating to sorghum.
The pre-trained ResNet-50 models are trained to classify which of 375 field plots an image comes from using cross-entropy loss. We train one model with RGB imagery and one with 3D imagery. The models have a 2048-dimensional global average pooling layer, followed by a 128-dimensional fully connected layer, and then the plot level classification output. During this pre-training, images (RGB or 3D) are resized to  $512 \times 512$  pixels and randomly flipped (horizontally and vertically). We use PyTorch, training for 25 epochs with the SGD optimizer with an initial learning rate of 0.01, and a learning rate decay of 0.1 every 10 epochs. The pre-trained model takes roughly one day to train on a single 16GB P100 Nvidia GPU with a batch size of 30.
After pre-training, we chop off the final plot classification layer. For each SNP prediction task in the  $\mathrm{SG}\times \mathrm{P}$  Benchmark, we then take this pre-trained model and train a single fully connected layer with two outputs, corresponding to the Reference and Alternate labels, using the  $\mathrm{SG}\times \mathrm{P}$  Benchmark training datasets (because the pre-trained model was trained on data from an entirely different growing season, there is no risk of data leakage or overfitting). For the multi-modal tasks, we first concatenate the 128-dimensional output features extracted by each of the RGB and 3D pre-trained models and
(a)
(d)
Figure 3: These plots show distributions of number of images by dataset, by sensor and by label. Figure (c) in particular conveys the class imbalance in the training data, showing the distribution of reference images minus alternate images in the training data (normalized by the total number of images for that dataset). Datasets with more reference images are farther to the right, and tasks with more alternate images are farther to the left.
(b)
(c)
(e)
then train a single fully connected layer on top of those concatenated features. In order to facilitate fast training of these SNP-specific models, we first extract the 128-D output features for every image in the dataset, and then train only the final fully connected layer for each SNP classification task. We use a batch size of 256 when training the SNP-specific models. Due to significant class imbalance in many of the datasets, we use a balanced sampler to guarantee a roughly equal number of Reference and Alternate examples in each training batch. We discuss the results of this baseline in Section 5, and provide links to the data used for pre-training, as well as our pre-trained model and training code, at the benchmark website (http://sorghumsnpbenchmark.com).
For every SNP in the  $\mathrm{SG} \times \mathrm{P}$  Benchmark, we consider two different evaluation paradigms for each of the imaging modalities: one that is per-image, and one that is per-cultivar. The evaluation metric is binary classification accuracy and per-image evaluation is straight-forward – for every image in the test dataset, is the predicted SNP label correct or not? This is averaged over the entire test set (every test set in the  $\mathrm{SG} \times \mathrm{P}$  Benchmark has an equal number of Reference and Example images, meaning that the average random model would achieve  $50\%$  accuracy on each task). We report on the results of the baseline approach described in Section 4 for the SNPs with known phenotypic expression in the middle three columns of Table 2. In all cases, the baseline models achieve significantly above chance performance, with the multimodal approach often outperforming the single-modality approaches.
This per-image evaluation is valuable for three reasons. First, many baseline ML classification approaches are designed to ingest a single image or image pair and produce a classification. Second, many variations of explainable image classification [47, 48, 49, 51, 69] operate on single images at a time to highlight visual features that affected the classification result. This approach to explainability can be useful when trying to understand genotype  $\times$  phenotype interactions.
The per-image evaluation, however, is not necessarily always the most appropriate. Some genetic variations may only impact the plant during a small portion of the growing season – for example, causing changes to the shape, size or timing of flowers that the plant produces. These sorts of variations would not be meaningfully correlated with visual features in images from other parts of the growing season. The goal of the  $\mathrm{SG} \times \mathrm{P}$  Benchmark is to uncover meaningful and as-yet unknown
SNP	By Image	By Cultivar
Chromosome	Gene	Position	RGB Only	3D Only	RGB+3D	RGB Only	3D Only	RGB+3D
1	001G269200	51588525	0.615	0.569	0.640	0.640	0.700	0.700
1	001G269200	51588838	0.601	0.564	0.582	0.652	0.696	0.609
1	001G269200	51589143	0.622	0.597	0.637	0.662	0.662	0.676
1	001G269200	51589435	0.640	0.606	0.665	0.781	0.750	0.812
6	006G067700	42805319	0.604	0.584	0.630	0.654	0.654	0.744
6	006G067700	42804037	0.597	0.571	0.620	0.653	0.681	0.667
6	006G147400	50898459	0.539	0.536	0.578	0.553	0.605	0.684
6	006G147400	50898536	0.505	0.569	0.578	0.500	0.684	0.658
6	006G147400	50898315	0.527	0.584	0.600	0.553	0.658	0.711
6	006G147400	50898231	0.617	0.578	0.650	0.735	0.618	0.765
6	006G147400	50898523	0.578	0.568	0.626	0.553	0.579	0.632
6	006G147400	50898525	0.625	0.576	0.650	0.763	0.658	0.842
6	006G057866	40312463	0.670	0.543	0.659	0.850	0.650	0.850
6	006G004400	2697734	0.624	0.630	0.709	0.763	0.816	0.842
9	009G229800	57040680	0.587	0.590	0.636	0.611	0.704	0.648
Table 2: Binary classification performance of the baseline approach on the SNPs with known genotype  $\times$  phenotype relationships. Performance is reported for each sensor separately, and when using sensor fusion, for both individual images and when aggregated by cultivar.
genotype  $\times$  phenotype relationships, including those that may not be observable in every image in the benchmark. To support this goal, we provide a more flexible, per-cultivar evaluation.
In the per-cultivar evaluation, researchers provide a single prediction for each of the cultivars in the test set (as opposed to each image). As in the per-image evaluation, the number of cultivars that have the Reference versus Alternate labels is the same, so the average random model would achieve  $50\%$  accuracy. We note that the number of cultivars per label for some SNPs can be relatively small – e.g. about 10 – meaning that we expect there to be more noise in the average accuracy of models on this evaluation. In the results shown in the last three columns of Table 2, we used the simplest approach to per-cultivar evaluation: we decided whether a cultivar is Reference or Alternate for a particular SNP by taking the mode of the predictions from all of the images that belong that cultivar.
Researchers have flexibility in how they produce the per-cultivar prediction. They could, for example, only consider images from a particular portion of the growing season, or weight images from different parts of the growing season differently. They could develop model architectures that consider multiple images simultaneously. In general, the per-cultivar evaluation provides researchers with the ability to develop models that explain complex genotype  $\times$  phenotype relationships in sorghum.
One way that the  $\mathrm{SG} \times \mathrm{P}$  Benchmark varies from traditional benchmarks is that it contains a significant number of tasks where it is unknown if above-chance performance is even possible. While we have structured the  $\mathrm{SG} \times \mathrm{P}$  Benchmark to include SNPs that are already well understood for proof of concept and validation, in addition to the SNPs with unknown genotype  $\times$  phenotype interations that are potentially "high impact", in reality, many of these unknown SNPs will have minimal impact. While this uncertainty makes our benchmark unique in machine learning datasets and competitions, it is not unique in science - the goal of this benchmark is discovery.
6 Web Page, Leaderboards & Discussion Forums
To support that discovery, we provide a web interface that has not only traditional leaderboards where researchers can compare their numerical performance to others, but also discussion forums where they can interact with plant biologists, sharing about their approaches, what they think their models are learning, and participate in back and forth dialogues to drive this discovery of unknown genotype  $\times$  phenotype relationships in sorghum. The website can be found at http:// sorghumsnpbenchmark.com. Screenshots of this web interface can be found in Figures 4 and 5.
Figure 4 (left) shows the main page for the  $\mathrm{SG} \times \mathrm{P}$  Benchmark. This page includes leaderboards for each of the SNPs in the Benchmark, with their current top accuracy by image and by cultivar, as well as the attribution for those leading results. Users can switch between leaderboards for each of the different imaging modalities. By clicking on the SNP name on these leaderboards, users are taken to a detail page for that SNP (seen in Figure 4 (right)). This page includes metadata for the SNP (its chromosome, gene and position on the chromosome), details about each of the training and testing datasets including links to download the data, SNP-specific leaderboards for each of the imaging modalities, and links to any Discussion Board posts that are about that SNP.
Figure 4: (left) The overall leaderboard for the  $\mathrm{SG} \times \mathrm{P}$  Benchmark, showing the current top results for different SNPs. (right) The detail page for one of the SNPs, including metadata, dataset information, the current SNP-specific leaderboards, and discussion board posts about the SNP.
In Figure 5, we show one such example discussion for the SNP highlighted in Figure 4 (right). This SNP was interesting because the baseline model achieved well above chance accuracy in all modalities, but also saw a significant improvement from using the multimodal data. We show an example discussion, where we posted about this interesting behavior and one of the biologists on our team replied with information from the sorghum literature about the SNP and a follow up asking about visual features that the machine learning model was focusing on. This back-and-forth engagement is precisely the goal of the  $\mathrm{SG} \times \mathrm{P}$  Benchmark - to drive the discovery of genotype  $\times$  phenotype relationships that can lead to new understanding about sorghum and direct development of new and improved sorghum crops.
There are several possible limitations of this work. The focus on sorghum may limit the generalizability of the findings. While we believe the structure of the machine learning approaches that on this dataset would generalize, other crops may exhibit unique genetic and phenotypic characteristics that require different approaches. The data represents a snapshot of sorghum variants grown in specific field conditions, which may introduce biases and limitations in capturing the full range of genetic and environmental factors that influence sorghum phenotypes. The benchmark is also limited to focusing on above ground traits that are visible in RGB and 3D data – this ignores any traits that are expressed below group or are not visible in these imaging modalities. It also is setup to consider the genetic impact of individual SNPs and does not consider more complex genetic interactions.
The SNPs chosen were ones that have a potential biological impact because they change the amino acid that the DNA encodes. However, that amino acid may not be important to the function of the plant, so many, perhaps most of these SNPs will have no biological impact. This limits the value of these tasks for researchers focussed on demonstrating that generic machine learning algorithms work in this domain. These tasks may be more compelling for researchers that want to be involved in discovering biological function; in this case we provide the discussion forum because an machine learning algorithm that scores well only goes part of the way to discovering the phenotype changed by that SNP, and additional work with visualizations or other explainability approaches may be needed to discover the genotype  $\times$  phenotype relationship.
This Sorghum Genotype  $\times$  Phenotype (SG×P) Benchmark provides a resource for exploring the relationships between genetic data and phenotypic expression in sorghum. By integrating large-scale, multimodal imagery and genetic information from a large number of sorghum species grown in real-world field conditions, this benchmark enables researchers to uncover significant insights into the use of sorghum in food and energy production systems. The benchmark includes baseline deep learning approaches, allowing for comparison to future algorithms that learn data-driven features relevant to understanding sorghum phenotypes. We create a combined leaderboard and discussion
Figure 5: A discussion board post about the SNP shown in Figure 4.
forum to further facilitate collaboration and knowledge-sharing among machine learning practitioners and sorghum-focused plant biologists.
The  $\mathrm{SG} \times \mathrm{P}$  Benchmark focuses specifically on sorghum, but we believe that algorithms and methodologies that are effective on this benchmark have the potential to apply to other crops and organisms with similar scales of imagery and genetic data. In building the benchmark, we have worked to make the tasks directly relevant to understanding some part of the relationship between genotypes and phenotypes, to simplify and catalyze interdisciplinary interactions within this research domain. In doing so, we hope that the  $\mathrm{SG} \times \mathrm{P}$  Benchmark thus serves as a catalyst to unraveling the intricate connections between genetic information and observable traits in sorghum and beyond, with the ultimate goal of addressing global challenges in food production, ecosystem resilience, and sustainable development.
This work was funded by the Advanced Research Projects Agency - Energy (ARPA-E) awards DE-AR0000594 and DE-AR0001101, by a seed grant from the Agricultural Genome to Phenome Initiative (AG2PI) (USDA-NIFA awards: 2022-70412-38454, 2021-70412-35233, and 2020-70412-32615), and by Google's Research Scholar Program ("Using Deep Learning to Uncover Unknown genotype  $\times$  phenotype Relationships in Crops").
[1] Shubhra Aich and Ian Stavness. "Leaf counting with deep convolutional and deconvolutional networks". In: IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). 2017, pp. 2080-2089.
[2] Basavaraj S Anami, Naveen N Malvade, and Surendra Palaiah. "Deep learning approach for recognition and classification of yield affecting paddy crop stresses using field images". In: Artificial Intelligence in Agriculture 4 (2020), pp. 12-20.
[3] Belal AM Ashqar, Bassem S Abu-Nasser, and Samy S Abu-Naser. "Plant seedlings classification using deep learning". In: International Journal of Academic Information Systems Research (IJAISR) (2019).
[4] Suchet Bargoti and James Underwood. "Deep fruit detection in orchards". In: IEEE International Conference on Robotics and Automation (ICRA). IEEE. 2017, pp. 3626-3633.
[5] Pierre Barre, Ben C Stöver, Kai F Müller, and Volker Steinhage. "LeafNet: A computer vision system for automatic plant species identification". In: Ecological Informatics 40 (2017), pp. 50-56.
[6] Barry R Bochner. "New technologies to assess genotype-phenotype relationships". In: Nature Reviews Genetics 4.4 (2003), pp. 309–314.
[7] Richard E Boyles, Zachary W Brenton, and Stephen Kresovich. "Genetic and genomic resources of sorghum to connect genotype with phenotype in contrasting environments". In: The Plant Journal 97.1 (2019), pp. 19-39.
1.For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b)Did you describe the limitations of your work? [Yes] See Section 7.
(c) Did you discuss any potential negative societal impacts of your work? [No] Due to nature of this benchmark (focusing on understanding the genotype  $\times$  phenotype relationship in sorghum), potential negative societal impacts are limited.
(d)Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2.If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b)Did you include complete proofs of all theoretical results? [N/A]
3.If you ran experiments (e.g. for benchmarks)...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [Yes] We reference the benchmark website (sorghumsnpbenchmark.com) multiple times throughout the paper. It includes pointers to all code, data and instructions.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 4.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [No] For the purpose of the baseline, and because we report results on many related problems that give some intuition about the variance, we did not judge the benefits of retraining the models to be worth the power consumption.
(d)Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section 4.
4.If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a)If your work uses existing assets, did you cite the creators? [Yes] See Sections 2.2 and 3.2.
(b)Did you mention the license of the assets? [Yes] See Section 2.2.
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes] Our curated datasets, pretrained models, leaderboard and discussion forums can be found at sorghumsnpbenchmark.com.
(d)Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e)Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5.If you used crowdsourcing or conducted research with human subjects...
(a)Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b)Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c)Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]