Fine-Grained Zero-Shot Learning with DNA as Side Information
Authors: Sarkhan Badirli
Zeynep Akata
George Mohler
Christine Picard
Murat Dundar
Venue: NeurIPS 2021 Poster (2021)
Keywords: zero-shot learning
Bayesian
species classification
fine-grained classification
DNA
insects

Abstract: Fine-grained zero-shot learning task requires some form of side-information to transfer discriminative information from seen to unseen classes. As manually annotated visual attributes are extremely costly and often impractical to obtain for a large number of classes, in this study we use DNA as a side information for the first time for fine-grained zero-shot classification of species. Mitochondrial DNA plays an important role as a genetic marker in evolutionary biology and has been used to achieve near perfect accuracy in species classification of living organisms. We implement a simple hierarchical Bayesian model that uses DNA information to establish the hierarchy in the image space and employs local priors to define surrogate classes for unseen ones. On the benchmark CUB dataset we show that DNA can be equally promising, yet in general a more accessible alternative than word vectors as a side information. This is especially important as obtaining robust word representations for fine-grained species names is not a practicable goal when information about these species in free-form text is limited. On a newly compiled fine-grained insect dataset that uses DNA information from over a thousand species we show that the Bayesian approach outperforms state-of-the-art by a wide margin. 

Fine-Grained Zero-Shot Learning with DNA as Side Information
Sarkhan Badirli
Department of Computer Science
Purdue University, West Lafayette, IN, USA
Indiana University-Purdue University Indianapolis, IN, USA
Zeynep Akata
George Mohler
University of Tübingen Max Planck Institute for Informatics Max Planck Institute for Intelligent Systems
Computer and Information Science Department Indiana University Purdue University
Indianapolis, IN, USA
Christine J. Picard
Murat Dundar
Department of Biology
Computer and Information Science Department Indiana University Purdue University Indianapolis, IN, USA Sondarr@uppi..
Indiana University Purdue University Indianapolis, IN, USA
Abstract
Fine-graindd zero-shot learning task requires some form of side-information to transfer discriminative information from seen unseen classes. As manually annotated visual attributes are extremely costly and often impractical to obtain for large number of classes, in this study we use DNA side information for the first time for fine-grained zero-shot classification of species. Mitochondrial DNA plays an important role as a genetic marker in evolutionary biology and has been used to achieve near perfect accuracy in species classification of living organisms. We implement simple hierarchical Bayesian model that uses DNA information to establish the hierarchy in the image space and employs local priors to define surrogate classes for unseen ones. On the benchmark CUB dataset we show that DNA can be equally promising, yet in general more accessible alternative than word vectors as side information. This especially important as obtaining robust word representations for fine-grained species names is not practicable goal when information about these species free-form text limited. On a newly compiled fine-grained insect dataset that uses DNA information from over a thousand species we show that the Bayesian approach outperforms state-of-the-arr by wide margin.
1 Introduction
Fine-grained species classification is essential in monitoring biodiversity. Diversity of life is the central tenet to biology and preserving biodiversity is key to a more sustainable life. Monitoring biodiversity requires identifying living organisms at the lowest taxonomic level possible. The traditional approach to identification uses published morphological dichotomous keys identify the collected sample. This identification involves a tedious process of manually assessing the presence or absence of a long list of morphological traits arranged hierarchical levels. The analysis is often performed in a laboratory setting by well-trained human taxonomist and is difficult to do at scale. Fortunately, advances in technology have addressed this challenge to some extent through the use of
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

DNA barcodes. DNA barcoding is a technique that uses a short section of DNA from a specific gene, such as cytochrome C oxidase I (COI), found in mitochondrial DNA, and offers specific information about speciation in living organisms and can achieve nearly perfect classification accuracy at the species level (66 17
As it is costly to obtain the label information for fine-grained image classification of species, Zero-Shot Learning (ZSL) that handles missing label information is suitable task. In ZSL, side information is used to associate seen and unseen classes. Popular choices for side-information are manually annotated attributes 21 12,, word embeddings 41 27) derived from free-form text or the WordNet hierarchy (28 2 It is often assumed that an exhaustive list of visual attributes characterizing all object classes (both seen and unseen) can be determined based only on seen classes. However, taking insects as our object classes, if no seen class species have antennae, the attribute list may not contain antenna, which may in fact be necessary to distinguish unseen species. In the United States alone, more than 40% of all insect species (>70,000) remain undescribed (42), which is a clear sign of the limitations of existing identification techniques that rely on visual attributes. Similarly, free-form text is unlikely to contain sufficiently descriptive information about fine-grained objects to generate discriminative vector embeddings. For example, tiger beetle is a class in the ImageNet dataset. However, the tiger beetle group itself contains thousands of known species and the Wikipedia pages for these species either do not exist or are limited to short text that does not necessarily contain any information about species' morphological characteristics. WordNet hierarchy may not be useful either as most of the species names do not exist in WordNet.
Given that DNA information can be readily available for training 35
species-level DNA 36
information can be used as highly specific side information to replace high-level semantic information in ZSL. For seen classes, species-level DNA information can be obtained by finding the consensus nucleotide sequence among samples of a given species or by averaging corresponding sequence embeddings of samples. For unseen classes, species-level DNA information can be obtained from actual samples, if available, in the same way as seen classes, or can be simulated in a non-trivial way to represent potentially existing species.
Our approach uses DNA as side information for the first time for zero-shot classification of species. In fine-grained, large-scale species classification, no other side information can explain class dichotomy better than DNA, as new species are explicitly defined based on variations in DNA. The hierarchical Bayesian model leverages the implicit inter-species association of DNA and phenotypic traits and ultimately allows us to establish a Bayesian hierarchy based on DNA similarity between unseen and seen classes. We compare DNA against word representations for assessing class similarity and show that the Bayesian model that uses DNA to identify similar classes achieves favorable results compared to the version that uses word representations dataset involving slightly less than 200 bird species. In the particular case of an insect dataset with over 1000 species, when visual attributes or word representations may not offer feasible alternatives, we show that our hierarchical model that relies on DNA to establish class hierarchy significantly outperforms all other beteedddinn-sseee methods and feature generating networks.
well-known ZSL benchmark species
Our contributions are on three fronts. First, we introduce DNA as side information for fine- one-grained ZSL tasks, implement: a Convolutional Neural Net (CNN) model to learn DNA barcode embeddings, and show that the embeddings are robust and highly specific for closed-set set classification of species, even when training and test sets of species are mutually exclusive. We use the benchmark CUB dataset as a case study to show that DNA embeddings are competitive to word embeddings as side information. Second, we propose a fine-grained insect dataset involving 21, 212 matching image/DNA pairs from 578 genera and 1, 213 species as a new benchmark dataset and discuss the limitations of current ZSL paradigms for fine-grained ZSL tasks when there is no strong association between side information and image features. Third, we perform extensive studies to show that a simple hierarchical Bayesian model that uses DNA side information outperforms fate-of-the-ar ZSL techniques on the newly introduced insect dataset by a wide margin.
2 Related Work
Zero-Shot Learning. Early ZSL. literature is dominated by methods that embed image features into a semantic space and perform various forms of nearest neighbor search to do inference (14 [1. As the dimensionality of semantic space is usually much smaller than the feature space this leads to the
2

---

Nusson dividiatus
Nysson distinguendus
Arge ciliarts
Arge usstulata
Megachile ericetorum
Megachile meritima
Myrmica fructicurnis
Myrmicaa alaskensis
Adatia
Adalia decompinceeta
Annara
Ainrru consuloris
Cieindela compestria
Vatianaerhynchits acquatuss
Lathrobium fuleipeune
Phyllopertha horticole
bipunctata
Cricotopus
Cricotopus surowstris
Aedes
Anopheles albimonus
Megosetia cirvisentris
Toxomerus amrginatus
Simndium
Platycheirus
abnizux
albopictus
callidum
neoperpalliiuu
Figure I: Image samples from the INSECT dataset. Rows represents small subset of species from three orders: Hymenoptera, Coleoptera and Diptera, respectively. The first word in names indicate genus, the two words together define the species name.
hubness problem where classes become hub and occur the nearest neighbor of many samples. In an effort to alleviate the hubness problem, (50) 44) changed the direction of the embedding from semantic to image feature space. This followed by line of work that investigates bidirectional
embedding between semantic and image spaces through a latent space (51
32 38
In (25 155, new strategy of synthesizing features for unseen classes and converting the challenging
ZSL problem into traditional supervised learning is introduced (23) 44 13 48
52
29 39 4).
Although feature generating networks (FGNs) currently achieve state-of-ther art results in ZSL, they suffer from the same problem as earlier lines of work in ZSL: hypersensitivity towards side information not strongly correlated with visual attributes. The vulnerability of both embedding and FGN-based methods toward sources of side information different than visual attributes, such as word vectors WordNet hierarchy, is investigated in (2) 339 44). Another limitation of FGNs is that features generated for unseen classes are significantly less dispersed than actual features due to the generator failing span generative models mitigate this problem by proposing different loss functions that can better explore inter-sample and inter-class relationships 3 with an increasing number of classes with an especially high inter-class similarity (24).
than small subset of modes available in the data. Recent deep
7 g 19 45). However, these methods fail scale well
Side Information in ZSL. Side information serves as the backbone of ZSL as bridges the knowledge gap between and unseen classes. Earlier lines of work (22) (D] visual attributes to characterize object classes. Although visual attributes achieve compelling results, obtaining them involves laborious process that requires manual annotation by human experts scalable to data sets with a large number of fine-graiie rreegga object classes. When dealing with fine- fine-graine grained species classification, apart from scalability, more pressing obstacle how define subtle attributes potentially characteristic of species that have never been observed.
As an alternative manual annotation, several studies (II 14 information that requires less effort and minimal expert labor such as textual descriptions, distributed text representations, like Word2Vec (27) and Glo Ve (33), learned from large unsupervised text corpora, taxonomical order built from pre-defined ontology like WordNet (28), or even human gaze reaction images 20)). The accessibility, however, majority of ZSL. methods implicitly assume strong correlation between side information and image features, which is true for handerafted attributes but less likely be true for text representations taxonomic orders. Consequently, all these methods experience significant decline in performance when side information is not based visual attributes.
2
46 34 5) proposed to learn side
the cost of performance degradation 2 39). A
Barcode of Life Data and DNA Embeddings
In this study, we present the fine-grained INSECT dataset with 21, matching image/DNA pairs from 2.3 species (see Fig. [oo sample images). Unlike existing benchmark ZSL datascts, this new

---

ACTCH
Sog ADCT-AACCGGGGA MCTTROENAA
Seq -ACCTGGG ACTTTOONAC
Cytochrome Oriidane (Coe) harondax
Sequence Aligoment
One lost Eneading
CNN Modell
UNA (nAke
Figure 2: Attribute extraction from mitochondrial DNA.
dataset uses DNA as side informatior] and can be best characterized with the high degree of similarity among classes. Among the existing benchmark datasets, SUN contains the largest number classes (717) but classes in SUN represent a wide range of scene categories related to transportation, indoor and outdoors, nature, underwater etc., and as such can be considered a relatively coarse-grained dataset compared to the INSECT dataset we are introducing in this study.
All insect images and associated DNA barcodes in dataset come from the Barcode of Life Data System (BOLD) (35 36). BOLD an open-access database in which users can upload DNA sequences and other identifying information for any living organism on Earth. The database provides approximately 658 base pairs of the mitochondrial DNA barcode extracted from the cytochrome c oxidase / (COI) gene along with additional information such country of origin, life-stage, order, family, subfamily, and genus/species names.
Data Collection. We collected image/DNA pairs of insects that originate from three orders: Diptera (true flies), Coleoptera (beetles) and Hymenopiera (sawflies, wasps, bees, and ants). While the dataset in general clean, manual effort was devoted to further curate the dataset. Only cases with images and matching DNA barcodes of adult insects are included. Images from each species were visually inspected and poor quality images were deleted. Only species with more than ten instances were included. The final dataset consisted of 21, 212 images and 1, 213 insects species of which 254 belong to Diptera (133 genera), 564 to Coleoptera (315 genera) and 395 to Hymenoptera (130 genera). We extracted image features, namely image embeddings, using a pre-trained (on ImageNet 1000 classes) ResNet101 model (16). Images are resized to 256 x 256 and cnter-cropped before fed to the ResNet model. No other pre- pre-processing is applied to the images.
Data Split. We randomly chose 10% of all species as unseen classes for the test set leading to 1. 092 seen and 121 unseen classes. Similarly, we randomly chose 10% of the 1, 092 training classes as unseen classes for the validation set. Samples from seen classes were split by 80/20 ratio in a stratified fashion to create seen portion of the train and test datasets. In the dataset there were few hundred cases where multiple image views (dorsal, ventral, and lateral) of the same insect were present. To avoid splitting these cases between train and test, we made sure all instances of the same insect are included in the training set. As a result, 12 of the 1, 092 seen classes in the training set were not represented in the test set. Our dataset splits are summarized in Table [
yoll
y
Y
#lmages #Classes
21,212
3,525
2,425
1,213
1,080
121
Table 1: ZSL split details. Y*, and Y" denote the seen and unseen test sets, whereas y represents entire data. There are 15, 262 (21,212 2, 425) samples left for the training set.
212 - 3, 525
DNA Embeddings. Although it is the first time DNA barcodes are used as side information in ZSL domain, there have been some work investigating vector embeddings for DNA sequences. Authors of (31) trained a CNN model to do binary DNA sequence classification considering sequences as a text data. Imitating amino acid structure, each triplet of base pairs are treated as a word and sequences are converted into one-hot vector representation. Taking (27) as the base, (30) trained a shallow neural network on human genome data to generate representation for k-mers. Unlike these techniques we deal with DNA Barcodes represented by nucleotide sequences and aim to convert the entire character sequence into a vector embedding useful for species classification with more than 1,000 classes.
Pleass refer to supplementary material for discussion limitations of using DNA side information

---

Most recently, DNABERT (88) adapted the powerful text transformer model (10) to genomic DNA setting and generated vector embeddings for long DNA sequences.
In this paper, we trained a CNN model to learn a vector representation of DNA barcodes in the Euclidean space. First, the consensus sequence of all DNA barcodes in the training with 658bp is obtained. Then, all sequences are aligned with respect to this consensus sequence using progressive alignment technique implemented in MATLAB R2020A (Natick, MA, USA). total of five tokens are used, one for each of the four bases, Ademine, Guanine, Cytosine, Thymine, and one for others. All ambiguous and missing symbols are included in the others token. In pre- preppesssssngg barcodes are one hot encoded into a 658x5 2D array, where 658 is the length of the barcode sequence (median of the nucleotide length of the DNA data).
To train the CNN model, a balanced subset of the training data is subsampled, where each class size is capped at 50 samples. The CNN is trained with 14, 723 barcodes from 1, 092 classes. No barcodes from the 121 unseen classes are employed during model training. The training set is further split into two as train (80%) and validation (20%) by random sampling. We used 3 blocks of convolutional layers each followed by batch normalization and 2D max-pooling, The output of the third convolutional layer is flattened and batch normalized before feeding the data into fully-connected layer with 500 units. The CNN architecture is completed by a softmax layer. We used the output of the fully-connected layer as the embeddings for DNA. Class level attributes are computed by the mean embedding of each class. The DNA- -based attribute extraction is illustrated in Figure 2 The details of the model architecture is depicted in Figure in Supplementary material. We used ADAM optimizer for training the model for five epochs with batch size of 32 (with a step-decay initial learning rate = 0.0005 and drop factor= 0.5, B1 = 0.9, B2 0.999). The model is developed in Python with Tensorflow- Keras API.
Predietive accuracy of DNA embeddings. Although the insect barcodes we used are extracted from a single gene (COI) of the mitochondrial DNA with relatively short sequence length of 658 base pairs, they are proven to have exceptional predictive accuracy: the CNN model achieves a 99.1% 1% accuracy on the held-out validation set. Note that, we only used the data from training seen classes to train the CNN model. In order to validate the generalizability of embeddings to unseen data, we trained a simple K-Nearest Neighbor classifier (K = 1) on the randomly sampled 80% of the DNA-embeddings of unseen classes and tested on the remaining 20%. The classifier had a perfect accuracy for all 121 but one classes with an overall accuracy of 99.8%.
In addition to our CNN model we have explored a DNABERT (88) model for converting DNA barcodes to vector embeddings. The pretrained DNABERT model achieves around 85% (vs 99% from CNN) top-1 KNN accuracy (averaged over 10 runs) on the unseen classes. Pretrained DNABERT can be fine-tuned for species classification however because of the vast number of parameters to tune each run takes few hours on a relatively sophisticated GPU, significantly more than CNN training. Similarly, a simple LSTM model with half of the parameters as the CNN model is almost 5 times slower than the CNN model and requires more epochs to reach a reasonable accuracy. Therefore, we use simple 3-layer CNN that trains in an hour and achieves almost perfect top-1 KNN accuracy.
To demonstrate that the approach can be easily extended larger members of the animal kingdom, we compiled approximately 26, 000 DNA barcodes from 1, 047 bird species to train another CNN model (ceteris paribus) learn the DNA embeddings for CUB dataset (see the Supp. materials for details). The CNN model achieved a compelling 95.60% on the held-out validation set.
4 Bayesian Zero-shot Learning
Object classes in nature already tend to emerge at varying levels of abstraction, but the class hierarchy more evident when classes represent species and species are considered the lowest taxonomic rank of living organisms. We build our approach on two layer hierarchical Bayesian model that was previously introduced and evaluated on benchmark ZSL datasets with promising results (6). The model assumes that there are latent classes that define the class hierarchy in the image space and uses side information to build the Bayesian hierarchy around these latent classes. Two types of Bayesian priors are utilized in the model: global and local. As the name suggests, global priors are shared across all classes, whereas local priors represent latent classes, and are only shared among similar classes. Class similarity is evaluated based on side information in the Euclidean space. Unlike
5

---

standard Bayesian models where the posterior predictive distribution (PPD) forms compromise between prior and likelihood, this approach utilizes posterior predictive distributions to blend local and global priors with data likelihood for each class. Inference for a test image is performed by evaluating posterior predictive distributions and assigning the sample to the class that maximizes the posterior predictive likelihood.
Generative Model.
The two-layer generative model is given below,
Jitk" ~ N(ps Eg); Ayt ~ (u,, ), Mj (No. Eykō'), Σ3 ~ W (Zo. m)
where j. k represent indices for local priors, classes, and image instances, respectively. We assume that image feature vectors jjtk come from a Gaussian distribution with mean Mij and covariance matrix Ej. and are generated independently conditioned not only on the global prior but also on their corresponding local priors.
Each local prior is characterized by the parameters μ4 and E.- H4 is the mean of the Gaussian prior defined over the mean vectors of local priors. Ko is scaling constant that adjusts the dispersion of the means of local priors around μ4o A smaller value for Kig suggests that means of the local priors are expected to be farther apart from each other whereas larger value suggests they are expected to be closer. On the other hand, Dg and m dictate the expected shape of the class distributions, as under the inverse Wishart distribution assumption the expected covariance is E(E[=,, m) = m- -D-1- where D the dimensionality of the image feature space. The minimum feasible value of m is equal to D and the larger the m is the less individual covariance matrices will deviate from the expected shape. The hyperparameter K1 is a scaling constant that adjusts the dispersion of the class means around the centers of their corresponding local priors. A larger #1 leads to smaller variations in class means relative to the mean of their corresponding local prior, suggesting a fine fine-grained relationship among classes sharing the same local prior. Conversely, a smaller dictates parse-grained relationships among classes sharing the same local prior. To preserve conjugacy of the model, the proposed model constrains classes sharing the same local prior to share the same covariance matrix Ej. Test examples are classified by evaluating posterior predictive distributions (PPD) of seen and unseen classes, As illustrated in Fig. the PPD in general incorporates three sources of information: the data likelihood that arises from the current class, the local prior that results from other classes sharing the same local prior as the current class, and global prior defined in terms of hyperparameters. PPDs for seen classes include the global prior and data likelihood and are derived in the form a Student-t distribution whereas for unseen classes the data likelihood does not exist as no image samples are available for these classes. We leave the details of derivations to the supplementary material.
Eo
Surrogate classes. According to the generative model in (T. groupings among classes are determined based on local priors. Thus, once estimated from seen classes, local priors can be used to define surrogate classes for unseen ones during inference. Associating each unseen class with a unique local prior forms the basis of our approach. The local prior for each unseen class is defined by finding the K seen classes most similar to that unseen class. The sim-
Surrogate Unseen clsss
Seen class
Grobel prior
Local prior estiattonn
ilarity is evaluated by computing the L (Euclidean) dis- tance between class-level attribute or embedding vectors (Φ) obtained from the side information available. Once local prior is defined for each unseen class the PPD for the corresponding surrogate class can be derived in terms of only global and local priors as in equation (2). Test exam- ples are classified based on conditionii likelihoods evaluated for both seen and surrogate classes.
K most smilar seen classes
Current class Likelhood
Surrogate-cluss Formation
Seen-clas Formation
Figure 3: Class formations for PPD dur- ing inference.
Σ Eitess (ngi Zzz + KoHOO
ny:81
P(x| (f no, Ko, K1) = T(x Eg, U,);
jj. Emery (mg)ta))
+ Ku
0 = Σ (nys 1 D--DDE 1+ Σ (Eo + Emn, (n +1)
(2)

---

where, Eyt, Sjs and noj represent sample mean, scatter matrix and size of class associated with local prior j. respectively and R is defined as in Eq. (30) in the supplementary materia
Rationale for the hierarchical Bayesian approach and limitations. We believe that the hierar- chical Bayesian model is ideally suited for fine-grained zero-shot classification of species when DNA used as side information for the following reasons. The performance of the model in identifying unseen classes depends on how robust the local priors can be estimated. This in turn depends on whether or not the set of seen classes contain any classes similar to unseen ones. As the number of seen classes increases, scen classes become more representative of their local priors, more robust estimates of local priors can be obtained, and thus, unseen classes sharing the same local priors as seen classes can be more accurately identified. On the other hand, if the class-level side information is not specific enough to uniquely characterize a large number of classes, then the model cannot evaluate class similarity accurately and local priors are estimated based potentially incorrect association between seen and unseen classes. In this case having a large number of seen classes available may necessarily help. Instead, highly specific DNA side information comes into play for accurately evaluating class similarity. If a unique local prior can be eventually described for each unseen class, then unseen classes can be classified during test time without the model having learn the mapping between side information and image features beforehand. Uniqueness of the local prior can only be ensured when the number of seen classes is large compared to the number of unseen classes. Thus, the ratio of the number of seen and unseen classes becomes the ultimate determinant of performance for the hierarchical Bayesian model. The higher this ratio is the higher the accuracy of the model will be. An experiment demonstrating this effect is performed in Section 5.3
If the same set of K classes is found to be the most similar for two different unseen classes, then these two unseen classes will inherit the same local prior and thus they will not be statistically identifiable during test time. The likelihood of such tie happening for fine-grained data sets quickly decreases as the number of classes increases. In practice we deal with this problem by replacing the least similar of the K most similar seen classes by the next most similar seen class for one of the unseen classes,
Experiments
In this section report results of experiments with two species datasets that use DNA side information. Details of training and hyperparameter tuning are provided in the supplementary material along with the source code of our methods.
5.1 Experiments with the INSECT dataset
We compare our model (BZSL) against state-of- the-art (SotA) ZSL methods proved to be most competitive on benchmark ZSL datasets that use visual attributes or word vector representations as side information. Selected SotA models rep-
Method
US
S
H
CRNet (49)
13.33
19.70
15.90
ALE (I
2.86
27.18
5.17
RelationNet (43
3.25
24.37
5.73
CADA-VAE 33
14.55
20.81
17.10
resent various ZSL categories: (1) Embedding
ESZSL B7)
3.41
18.61
5.77
methods with traditional (I) 37) and end-to-end neural network (49) approaches. (2) FGNs using VAE (39) and GAN (44), and (3) end-to-end few shot learning approach extended to ZSL. (43). Table 2
LsrGan 44
12.58
30.41
17.75
BZSL
20.83
38.30
26.99
Table 2: Generalized ZSL results on Insect data using DNA barcodes as attributes.
displays seen (S) and unseen (US) ac- curacies and their harmonic mean (H) on the
INSECT data using DNA as the side information. Results suggest that the large number of seen classes along with the highly specific nature of DNA information in characterizing classes particularly favors the Bayesian method to more accurately estimate local priors and characterize surrogate classes. The harmonic mean achieved by the proposed method is 52% higher than the harmonic mean achieved by the second best performing technique. Similar levels of improvements are maintained on both seen and unseen class accuracies. The next top performers are FGNs. CADA VAE uses a VAE whereas LsrGan utilizes GAN to synthesize unseen class features, then both train LogSofimax classifier for inference. Lower unseen class accuracies suggest that FGNs struggle to synthesize meaningful
2 The code and dataset available athttps: http://gthhabcccoeeaaaaa//////e/ Grained- Setraned-ZSL--a
7

---

Attributes
Word Vectors
DNA
Method
US
H
US
S
H
US
H
CRNet (49)
44.28
59.84
50.89
22.75
45.92
30.43
9.27
56.56
15.93
ALE I
25.15
60.80
35.59
3.95
48.57
7.31
3.50
50.18
6.54
RelationNet (43
11.66
44.81
18.50
8.67
36.16
13.99
5.33
40.83
9.42
TADA-VAE 39
47.15
53.11
49.95
26.45
41.98
32.45
19.42
37.05
25.48
ESZSL.
15.58
50.66
23.84
2.26
23.86
4.12
5.99
5.38
5.67
LsrGan 44
47.65
56.97
51.89
24.63
37.96
29.88
15.99
33.57
21.66
BZSL
31.49
50.61
38.82
22.43
45.00
29.94
27.46
48.14
34.97
Table 3: Generalized ZSL results on CUB data using original visual attributes, word vectors, and DNA attributes. US, S, and H represent unseen, seen class accuracies and harmonic mean, respectively.
features the image space. On the other hand, CRNet that uses end-to-end neural network to learn the embedding between semantic and image spaces renders slightly worse performance than FGNs. seems, non-linear embedding also works better than a linear (ESZSL) and bilinear (ALE) ones for this specific dataset. RelationNet is amongst the ones with the lowest performance, as the method is explicitly designed for Few-shot learning and expects the side information to be strongly correlated with image features. The weak association between side information and image features affects the performance of both FGNs and embedding methods, but the traditional embedding methods suffer the most.
5.2 Experiments with the benchmark CUB dataset
To demonstrate the utility of DNA-based attributes in a broader spectrum of species classification, we procured DNA barcodes, again from the BOLD system, for bird species in the CUB dataset. For this experiment, we derived 400 dimensional embeddings in order to have the same size with word vectors and eliminate the attribute size effect. There were 6 classes, seen and 2 unseen, that did not have DNA barcodes extracted from COI gene in the BOLD system. These classes were excluded from the dataset but the proposed split from (47) is preserved otherwise.
The results shown in Table 3 validate our hypothesis that when side information is not strongly correlated with visual characteristics of object classes (like in word vectors or DNA) both embedding methods and FGNs display significant performance degradation. With the exception of the proposed Bayesian model, word vector representation yields better accuracy than DNA-based attributes for all models. This phenomenon can be explained by our observation that text fragments related to common animals/birds in the Wikipediaalnternet often include some morphological traits of the underlying species. Hence, word vector representation is expected to have higher degree of correlation to visual attributes than DNA information. Our model produces the best results, 34.97% vs 32.45% when side information is not derived from visual characteristics of classes. This outcome validates the robustness of the Bayesian model to diverse sources of side information and emphasizes the need for more robust FGN or embedding based models in more realistic scenarios where hand-crafted visual attributes are not feasible.
5.3 The effect of the number of seen classes on performance
Local priors central to the performance of the hierarchical Bayesian model. Here, we perform experiments to show that as the number of seen classes increases while the number of unseen classes fixed, each unseen class can be associated with larger pool of candidate seen classes and more informative local priors can potentially be obtained, which in turn leads to more accurate identification of unseen classes. To demonstrate this effect we run two experiments. In the first experiment we use the same set of unseen classes as in Section 5.1 but gradually increase the number of seen classes used for training. In the second experiment we double the size of the unseen classes and gradually include the remaining classes into training as seen classes. The first experiment is also performed for CADA-V CADA-VAE. LsrGan skipped for this experiment due to long training time. To account for random subsampling of seen classes each experiment is repeated five times and error bars are included in each plot. There a clear trend in these results that further highlights the intuition behind the hierarchical Bayesian model and explains why this model is well-suited for fine-graine ZSL. When 10% of the

---

0.5
0.6
0.5
ADC.
Actura
24
Accurccy
0.4
03
0.2
01
0.1
0
109
70%
359
40%
60%
40%
102%
10%
18aa
30%
40%
60%
68%
100%
10%
307
30%
403
60%
100th
(189)
(220
(227)
(638)
(653)
9973)
(1992)
1336
(254)
(133)
(88)
09831
(109)
(218)
(2271
(436)
(650
0992
8 Seen cass
Seen clsss
% Seen class
(a) BZSI. results in original setup
(h) B7SL. results with Yf 983
(c) CADA-VAE in original setup
(Y 1, 092 and yd 121)
and Y 230
Y 1, 092 and Y 121)
Figure 4: The effect of the number of seen classes on the performance of BZSL and CADA-VAE. Each experiment is repeated five times to account for random subsampling of seen classes.
classes are used as unseen, unseen class accuracy improves with increasing number of seen classes until it flatlines beyond the 60% mark while seen class accuracy always maintains around the same level (see Fig. 4ab. When 20% of the classes are used as unseen no flatlining effect in unseen class accuracy is observed even at 100% mark, which suggest that there is still room for improvement in unseen class accuracy if more classes become available (see Fig. 4b). For CADA-VAL unseen class accuracy initially improves and then flatlines beyond 80% mark but this improvement comes at the expense of significant degradation in seen class accuracy, which suggest that as the number of seen classes increase generated features further confound the classifier as would be expected of an FGN for a fine-grained dataset.
6 Conclusions
For the first time in the ZSL. literature we use DNA as side information and demonstrate utility in evaluating class similarity for the purpose of identifying unseen classes in a fine-gra fine-grained ZSL setting. On the CUB dataset, despite being trained with less than 30,000 very short sequences, we find DNA embeddings to be highly competitive with word vector representations trained on massive text corpora. We emphasize the importance of DNA as side information in zero-shot classification of highly fine-grained species datasets involving thousands of species, and on the INSECT dataset, show that a simple Bayesian model that readily exploits inherent class hierarchy with the help of DNA significantly outperform highly complex models. We show that SotA ZSL methods that take the presence of an explicit association between visual attributes and image features for granted, suffer significant performance degradation when nosuvisual attributes such word vectors and WordNet used as side information. The same effect is observed with DNA embeddings as well. Although visual attributes tend be the best alternative as side information for a coarse -rarise-ggaindddde species classification task, they quickly lose their appeal with an increasing number of classes. Considering the tens of thousands of described species and even larger number of undescribed species, DNA seems to be the only feasible alternative to side information for large-scale, fine-grained zero-shot classification of species.
These favorable results by a simpler model suggest that as the number of classes increases along with inter-class similarity, the complexity of the mapping between side information and image attributes emerges as a major bottleneck at the forefront of zero-shot classification. A promising future research avenue appears be implementing hierarchically organized FGNs where each subcomponent only operates with a small subset of seen classes all sharing the same local prior.
This work does not present any foreseeable negative societal consequences beyond those already associated with generic machine learning classification algorithms.
Acknowledgements: This research was sponsored by the National Science Foundation (NSF) under Grant Number IIS-1252648 (CAREER). This work has been partially funded by the ERC (853489- DEXIM) and by the DFG (2064/1 - Project number 390727645). GM acknowledges support from NSF-ATD grant 2124313. The content is solely the responsibility of the authors and does not necessarily represent the official views of NSF

---

References
[1] Z. Akata, F. Perronnin, Z. Harchaoui, and C. Schmid. Label-embedding for image classification.
TPAMI, 2016.
[2] Z. Akata, S. Reed, D. Walter, H. Lee, and B. Schiele. Evaluation of output embeddings for
fine-grained image classification. In CVPR, 2015.
[3] M. Arjovsky, S. Chintala, and L. Bottou. Wasserstein generative adversarial networks. In International conference on machine learning, pages 214-223. PMLR, 2017.
[4] G. Arora, V. Verma, A. Mishra, and P. Rai. Generalized zero-shot learning via synthesized
examples. IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4281-
4289, 2018.
[5] J. Ba, K. Swersky, S. Fidler, and R. Salakhutdinov. Predicting deep zero-shot convolutional
neural networks using textual descriptions. ICCV, 2015.
[6] S. Badirli, Z. Akata, and M. Dundar. Bayesian zero-shot learning. In ECCV Workshops, 2020.
[7] J. Bao. D. Chen, F. Wen, H. Li, and G. Hua. Cvae-gan: fine-grained image generation through Proceedings of the IEEE international conference on computer vision, pages 2745-2754, 2017.
asymmetric training.
[8] L. Cacheux, H. L. Borgne, and M. Crucianu. Modeling inter and intra-class relations in the triplet loss for zero-shot learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10333-10342, 2019.
[9] L. Chen,, H. Zhang. Xiao, W. Liu, and S. Chang. Zero-shot visual recognition using semantics- preserving adversarial embedding networks. IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 1043-1052, 2018.
[10] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019.
[11] M. Elhoseiny, B. Salch, and A. Elgammal. Write a classifier: Zeroshot learning using purely textual descriptions. In ICCV, 2013.
[12] A. Farhadi, I. Endres, D. Hoiem, and D. Forsyth. Describing objects by their attributes. In
CVPR, 2009.
[13] R. Felix, V. Kumar, I. Reid, and G. Carneiro. Multi-modal cycle-consistent generalized zero-shot learning. In ECCV, 2018.
[14] A. Frome, G.S. Corrado, J. Shlens, S. Bengio, J.
Dean, M. A. Ranzato, and T. Mikolov. Devise: A deep visual-semantic embedding model. In NIPS, 2013,
[15] Y. Guo, G. Ding, J. Han, and Y. Gao. Synthesizing samples for zero-shot learning. In I/CAI,
2017.
[16] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR,
2016.
[17] D. Hebert, M. Y. Stoeckle, T. S. Zemlak, and C. M. Francis. Identification of birds through
dna barcodes. PLoS biology, 2, 2004.
[18] Y. Ji, Z. Zhou, H. Liu, and R. V. Davuluri. Dnabert: Pre-trained bidirectional encoder represen- tations from transformers model for dna-language in genome. Bioinformatics, 37, 2021. [19] H. Jiang. R. Wang. S. Shan, and X. Chen. Transferable contrastive network for generalized zero-shot learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9765-9774, 2019.
[20] N. Karessli, Z. Akata, B. Schiele, and A. Bulling. Gaze embeddings for zero-shot image classification. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6412-6422 2017.
[21] C. Lamperr, H. Nickisch, and S. Harmeling. Learning to detect unseen object classes by between-class attribute transfer. In CVPR, 2009.
[22] C. Lampert, H. Nickisch, and S. Harmeling. Attribute-based classification for zero-shot visual object categorization. TPAMI, 36(3):453-444
2013.
[23] J. Li, M. Jin, K. Lu, Z. Ding. L. Zhu, and Z. Huang. Leveraging the invariant side of generative
zero-shot learning. CVPR, 2019.
[24] J. Liu, Z. Zhang, and G. Yang. Cross-class generative network for zero-shot learning. Informa- tion Sciences, 555:147-163, 2021.
[25] Y. Long, L. Liu, L. Shao, F. Shen, G. Ding, and J. Han. From zero-shot learning to conventional
supervised classification: Unseen visual data synthesis. In CVPR, 2017.
[26] D. H. Lunt, D. X. ZHang, J. M. Szymura, and G. M. Hewitt. The insect cytochrome oxidase i gene: evolutionary patterns and conserved primers for phylogenetic studies. PLoS biology, 5, 1996.
[27] T. Mikolov, K. Chen, G. Corrado, and J. Dean. Efficient estimation of word representations in vector space. In ICLR, 2013.
10

---

[28] G. A. Miller. Wordnet: a lexical database for english. CACM, 38, 1995.
[29] A. Mishra, M. K. Reddy, A. Mittal, and H. Murthy. A generative model for zero shot learning
using conditional variational autoencoders. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 2269-22698, 2018.
[30] Ng. dna2vec: Consistent vector representations of variable-length k-mers. ArXiv,
abs/1701.0622
2017.
[31] N. G. Nguyen, V. A. Tran, D. L. Ngo, D. Phan, F. R. Lumbanraja. M. R. Faisal, B. Abapihi,
M. Kubo, and K. Satou. Dna sequence classification by convolutional neural network. Journal of Biomedical Science and Engineering, 9, 2016.
[32] M. Norouzi, T. Mikolov, S. Bengio, Y. Singer, Shlens, A. Frome, G. Corrado, and J. Dean. Zero-shot learning by convex combination of semantic embeddingg. In ICLR, 2014.
[33] J. Pennington, R. Socher, and C. D. Manning. Glove:
Global vectors for word representation.
In E
[34] R. L. L. C. Shen, and A. van den Hengel. Less is more: Zero-shot learning from online textual documents with noise suppression. In CVPR, 2016.
[35] Ratnasingham and D. Hebert. Bold: The barcode of life data system (htpp://ww.. barcod-
inglife. org). Molecular ecology notes, 7(3):355-364, 2007.
[36] S. Ratnasingham and P. D. Hebert. A dna-based registry for all animal species: the barcode index number (bin) system. Ploš one, 8(7):e66213, 2013.
[37] B. Romera-Parede and P. H. Torr. An embarrassingly simple approach to zero-shot learning.
In ICMI., 2015.
[38] S.Changpinyo, W.-L.Chao, B.Gong, and F.Sha. Synthesized classifiers for zero-shot learning.
In CVPR, 2016.
[39] E. Schonfeld, S. Ebrahimi, S. Sinha, T. Darrel, and Akata. Generalized zero- and few-shot learning via aligned variational autoencoders. In CVPR, 2019.
[40] Y. Shigeto, I. Suzuki, K. Hara, M. Shimbo, and Y. Matsumoto. Ridge regression, hubness, and zero-shot learning. In ECML/PKDD, 2015.
[41] R. Socher, M. Ganjoo, C. D. Manning, and A. Ng. Zero-shot learning through cross-modal transfer. NIPS, 2013.
[42] N. E. Stork. How many species of insects and other terrestrial arthropods are there on earth?
Annual review of entomology, 63:31 1- 1-45, 2018
[43] Sung. Y. Yang, L. Zhang. T. Xiang, P. H. Torr, and T. M. Hospedales. Learning to compare:
Relation network for few-shot learning. In CVPR, 2018.
[44] M. R. Vyas, H. Venkateswara, and S. Panchanathan. Leveraging seen and unseen semantic relationships for generative zero-shot learning. In ECCV, 2020.
[45] M. R. Vyas, H. Venkateswara, and S. Panchanathan. Leveraging seen and unseen semantic relationships for generative zero-shot learning. In European Conference on Computer Vision, pages 70-86. Springer, 2020.
[46] Y. Xian. Z. Akata, G. Sharma, Q. Nguyen, M. Hein, and B. Schiele. Latent embeddings for zero-shot classification. In CVPR, 2016.
[47] Y. Xian, C. Lampert, B. Schiele, and Z. Akata. Zero-shot learning comprehensive evaluation of the good, the bad and the ugly. TPAMI, 2018.
[48] Y. Xian, T. Lorenz, B. Schiele, and Z. Akata, Feature generating networks for zero-shot learning.
In CVPR, 2018.
[49] F. Zhang and G. Shi. Co-representation network for generalized zero-shot learning. In ICML,
2019.
[50] L. Zhang, T. Xiang, and S. Gong. Learning a deep embedding model for zero-shot learning. In CVPR, 2017.
[51] Z. Zhang and V. Saligrama. Zero-shot learning via semantic similarity embedding. In ICCV,
2015.
[52] Y. Zhu, M. Elhoseiny, Liu, and A. Elgammal. Learning feature-to to-feuure translator by alternating back poc-proopagttion for generative shro-shot learning. ICCV, 2019.
11


Reviews:
[{"review_id": "lstzVB4CgbE", "reviewer": "Reviewer_wkgT", "summary": "This paper proposes to use DNA as the side information for zero-shot species classification. The idea is novel and biologically sound. Experiments show that DNA embeddings achieve comparable performance as text embeddings when serving as side information.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Advantages:\n1. This paper is the first work that utilizes DNA as side information for Zero-shot species classification, and the authors clearly explain the reasons and insights behind this.\n2. This paper introduces a new dataset for Zero-shot species classification and provides codes to reproduce their experiments.\n\nWeaknesses:\n1. On the technique side, the paper's main contribution is developing a CNN model to calculate DNA embeddings. However, there are many existing works (e.g., dna2vec, Gene2vec, DNABERT, etc.) that calculate embeddings for DNA sequences. This paper doesn't mention, use or compare with them. Thus, it is unclear how novel/effective the proposed model is.\n\n2. The paper is not well-written and relatively hard to follow. \n    -  It is not well-organized. In section 3, the authors introduce data collection, data processing, DNA embedding calculation, and some experimental details and results on DNA embedding, while in Section 4, they introduce the rest part of the proposed method. It may be better to re-organize this paper.\n    - The Experimental setting and baseline models are not clearly explained. The tables are hard to understand. It can be very helpful to include more information in the table caption (e.g., explain the experimental setting, evaluation metrics, and meaning of the abbreviations).\n    - Other issues:\n        - Table 3 should appear after Table 2.\n        - Caption of a table should appear above it.\n        - The most important reference \"[5]\" that the proposed model is mainly based on is missing.\n        - ......\n        \n\n3. As mentioned in Line 189-190, the Bayesian Zero-shot Learning introduced in Section 4 is previously proposed in the missing reference. It is unclear how this paper relates to that one.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I think the proposed idea is novel and interesting. The paper can be greatly improved if the paper is re-organized and more experiments are conducts.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "giPm0vDgwWD", "reviewer": "Reviewer_2AZs", "summary": "Authors propose to use DNA as side information for zero-shot image classification and a method that can leverage this information. The method is based on a Bayessian model that captures similarity between different classes as well as between images of a single class through global and local priors. Unseen classes are represented using the statistics of a fixed number of seen classes that are the most similar according to the side-information. Authors introduce an INSECT database and compare their method on INSECT and CUB with several other zero-shot methods. When using DNA as side information, the proposed method performs the best.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "As of my knowledge this is the first paper proposing to use DNA as side information for zero-shot image classification. The hierarchical Bayessian approach to model the data is also relevant and I can imagine it becoming one of usual baselines (though it is a pity it is coded in matlab). The exposition is overall clear, find few suggestions below. An added value is also the INSECT dataset.\n\nSuggestions:\n- The main text falls short to explain how to arrive from the seen class statistics to computing PPD for an unseen class.\n- Table 3 - Include what US/S/H means in the caption.\n- Fig. 4 - Use a larger font. Put equal y-axis scale in all three plots.\n- Line 76 - \"state of the art\" => \"state-of-the-art\"\n- Line 133 - \"larger\" => \"more\"\n- Line 209 - \"farther\" => \"further\"\n- Line 325 - \"fixed\" => \"stays fixed\"\n- Line 336 - \"maintained\" => \"maintains\"\n\nFinal recommendation:\nI have read the other reviews and authors responses. I would like thank the authors for the responses and urge them to reflect the criticism in the final paper. I keep my recommendation to accept the paper (with the current score).\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "- BSZL performs better than the best SOTA only on the DNA.\n- The obvious issue of using DNA as side information is that it is not applicable to non-living objects.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "IvG3YDCnqSU", "reviewer": "Reviewer_fQEh", "summary": "The paper proposes using DNA as side information for fine-grained zero-shot learning of insects, as opposed to visual attribute labels. Their proposed approach establishes a hierarchy over the images using DNA information, via a hierarchical Bayesian model over a learned embedding of DNA barcodes. Their method particularly outshines attribute-label-based approaches on an insect identification application, where visual attributes (from public Wikipedia pages, for example) are sparsely labeled or not available. The provide as an additional contribution their paired DNA/image benchmark dataset containing 21,212 examples of 1,213 insect species.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper provides an interesting alternative, DNA, as side information for zero-shot learning.  The main drawback to the generality and scalability of this method is that it requires paired DNA information to be extracted from the specimen in question, which is not necessarily available for any given sighting - in fact it is most likely only available when digitizing and analyzing large natural history collections or within research labs that collect physical specimens (very common in entomology). That said, when that information is available it should be used, and this paper provides a clean formulation of the problem, benchmark dataset, and proposes a methodology for how to use DNA information effectively in zero-shot learning.\n\nSome insect species, for example Rove Beetles, cannot be visually identified by experts unless they are dissected and the genitalia in inspected under a microscope. The error rates in classification from DNA barcoding can be up to 30%, and museum collections are known to contain labeling errors, particularly for challenging-to-identify species. Do the authors have a sense of the accuracy of their dataset subject to these known challenges in accurate insect ID?\n\nIt would be very interesting to see a visualization of the taxonomic distribution of the data, and particularly see the resultant distributions across the 3 orders considered of the random train/test/unseen splits (I’m imagining a taxonomic tree with the different splits colored at the end node, or possibly the proportion of categories in each split at each taxonomic level colored in some way).  It would also be good to show the distribution of examples per class in each split, to see the amount of imbalance represented. Further, it would be interesting to look at the distribution of these specimens across digitizing entities contribution to BOLD, to probe whether any potential correlations exist in digitization practice (camera type, background, DNA extraction method, etc) for certain taxa.\n\nIs the DNA embedding network trained with a species classification loss? I assume so but it isn’t explicit in the text.\n\nHow much does the similarity of seen classes affect the local prior for surrogate classes (beyond just the ratio of seen to unseen)?  This would be interesting to explore. Perhaps you could probe how results shift based on the number of seen classes within some distance threshold in embedding space? Or build an explicit split based on taxonomy, with ie held out genera instead of species, to probe the robustness of the model to less “familiar” taxonomic branches?\n\nHow much is the performance gain over SOTA based on the fact that those methods were not designed with DNA in mind, but instead visual attributes or word vectors?\n\n\n\nText Suggestions & Nits:\n\nIn the introduction, your first sentence “Fine-grained species classification is essential in monitoring biodiversity.” is perhaps not well justified for a machine learning audience. It may help to provide a few examples of how it is used - what happens after you identify the species? \n\nIn line 30, “fine-grained classification of species” does this refer to classification generally across data type? Or from images specifically? Throughout these first paragraphs clarification about what data type and problem formulation you are referring to would help.\n\nIn line 53: “or can be simulated in a non-trivial way 54 to represent potentially existing species.” Can you further elaborate? How would you simulate the DNA metadata for an image of an unseen species in a robust way in order to provide that side information to your model?\n\nLine 82: give a quick high-level description of the hubness problem\n\nDefine US, S, H in the caption of Tables 2&3 so that readers do not need to search the text to understand the table, it’s also a bit odd that table 3 appears before table 2 in the document\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "One potential area of social impact would be risks associated with errors in such a system for insect ID, particularly if those errors are systematically biased in some way, in downstream biodiversity monitoring. It would be nice if the authors dug into these potential risks specifically in downstream insect biodiversity and conservation.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "1fxBGWHqcV_", "reviewer": "Reviewer_SAgn", "summary": "This paper proposed to use DNA as side information for the zero-shot learning tasks. \nA CNN is used to generate DNA embeddings, and a Bayesian model is used for zero-shot learning.\nExperiments are performed on a newly collected fine-grained insect dataset with image and DNA pairs, as well as on the CUB dataset.\nResults showed that the Bayesian model outperforms other methods on the insect dataset using DNA as attributes.\nOn the CUB dataset, using DNA is comparable with using word vectors, but pre-defined attributes still work better.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper proposed to use DNA for zero-shot learning, unlike prior works using attributes or text descriptions for side information. Results on the collected insect dataset show improvements over other methods. On the CUB dataset, using Bayesian modeling is better for modeling DNA information, but not on visual attributes or word vectors. \n\nIn general, it is interesting to see that DNA data is helpful for ZSL, and a simple Bayesian model performs well on the DNA data.\nThe paper is well-written, and all the details of training and datasets are included. Ablation studies on the number of seen classes and hyper-parameters are also presented. \n\nQuestions:\n* Why use CNN (on top of one-hot encoding) instead of RNN or other methods for encoding the DNA data?\nFor the CUB dataset, the DNA embedding is trained on an external dataset of 1047 bird species. Is it a fair comparison between using attributes and word vectors (in Table 3)?\n* As stated in L. 246-264, the performance of the Bayesian model is highly correlated to the ratio of the number of seen and unseen classes. If there are similar classes in the seen classes, then the model can perform well. Is this limitation the reason why using visual attributes performs better than using DNA on the CUB dataset?\n* Another limitation of the proposed method is the trade-off between seen and unseen classes with different hyper-parameters (Fig. 5) Is there a way to mitigate this, i.e., having good performance on both seen and unseen classes?\n\nTypo:\n- Reference [5] is missing\n- L.10 \"of\" CUB dataset", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Some limitations have been discussed in the paper, and there is no negative societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper addresses the problem of zero shot learning (ZSL) in the context of fine-grained visual categorization using images from the natural world with (for the first time) DNA as side information. Existing work in this space typically exploits alternative forms of side information such as visual attributes or textual information. Experiments are performed using an existing Bayesian ZSL method on the CUB dataset (with additional DNA information) and a newly proposed dataset of insect images, and show that the models using DNA are competitive and often better than those using only textual side information. 

Issues that should be addressed in the final text. 
* The current related work text is missing a discussion of alternative methods for learning DNA embeddings. This needs to be remedied. (wkgT)
* Include the already performed comparison to DNABert. This is important as the CNN encoder is listed as one of the contributions of the paper but no comparisons are made to alternative methods for encoding this information. 
* Discuss the limitations of DNA side-information e.g. that it is not applicable to all visual categories (wkgT).
* Fix the issues with the writing and paper structure as noted by wkgT.
* Fix the issues with the text as noted by 2AZs and fQEh. e.g. Clarify in the caption for Table 3, what US, S, and H represent. 

The reviewers raised legitimate concerns regarding the level of technical contribution, the writing quality, and lack of discussion of other approaches for learning DNA embeddings. The final two of these concerns can be addressed in a revision of the text. Despite these concerns, the reviewers were supportive of the paper, with the most critical raising their score after the discussion. The novelty of the application and the merit of the new dataset were quoted as the reasons for recommending acceptance. The authors are strongly encouraged to address the above issues in the final camera ready text. 

As a final note, the authors should be aware that how they cited [5] was not consistent with the NeurIPS guidelines. As the paper was already published it should have been cited with the authors names and treated no differently than any other paper:  
"If you need to cite one of your own papers, you should do so with adequate anonymization to preserve double-blind reviewing.  For instance, write “In the previous work of Smith et al. [1]…” rather than “In our previous work [1]...”). If you need to cite one of your own papers that is in submission to NeurIPS and not available as a non-anonymous preprint, then include a copy of the cited submission in the supplementary material and write “Anonymous et al. [1] concurrently show...”)."
https://nips.cc/Conferences/2021/CallForPapers
Decision: Accept (Poster)