Fine-Grained Zero-Shot Learning with DNA as Side Information

1 Introduction . Fine-grained species classification is essential in monitoring biodiversity . Diversity of life is the central tenet to biology and preserving biodiversity is key to a more sustainable life . Monitoring biodiversity requires identifying living organisms at the lowest taxonomic level possible . The traditional approach to identification uses published morphological dichotomous keys to identify the collected sample . This identification involves a tedious process of manually assessing the presence or absence of a long list of morphological traits arranged at hierarchical levels . The analysis is often performed in a laboratory setting by a well-trained human taxonomist and is difficult to do at scale . Fortunately , advances in technology have addressed this challenge to some extent through the use of 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . DNA barcodes . DNA barcoding is a technique that uses a short section of DNA from a specific gene , such as cytochrome C oxidase I ( COI ) , found in mitochondrial DNA , and offers specific information about speciation in living organisms and can achieve nearly perfect classification accuracy at the species level ( 26 ; 17 ) . As it is costly to obtain the label information for fine-grained image classification of species , Zero-Shot Learning ( ZSL ) that handles missing label information is a suitable task . In ZSL , side information is used to associate seen and unseen classes . Popular choices for side-information are manually annotated attributes ( 21 ; 12 ) , word embeddings ( 41 ; 14 ; 27 ) derived from free-form text or the WordNet hierarchy ( 28 ; 2 ) . It is often assumed that an exhaustive list of visual attributes characterizing all object classes ( both seen and unseen ) can be determined based only on seen classes . However , taking insects as our object classes , if no seen class species have antennae , the attribute list may not contain antenna , which may in fact be necessary to distinguish unseen species . In the United States alone , more than 40 % of all insect species ( > 70,000 ) remain undescribed ( 42 ) , which is a clear sign of the limitations of existing identification techniques that rely on visual attributes . Similarly , free-form text is unlikely to contain sufficiently descriptive information about fine-grained objects to generate discriminative vector embeddings . For example , tiger beetle is a class in the ImageNet dataset . However , the tiger beetle group itself contains thousands of known species and the Wikipedia pages for these species either do not exist or are limited to short text that does not necessarily contain any information about species ’ morphological characteristics . WordNet hierarchy may not be useful either as most of the species names do not exist in WordNet . Given that DNA information can be readily available for training ( 35 ; 36 ) , species-level DNA information can be used as highly specific side information to replace high-level semantic information in ZSL . For seen classes , species-level DNA information can be obtained by finding the consensus nucleotide sequence among samples of a given species or by averaging corresponding sequence embeddings of samples . For unseen classes , species-level DNA information can be obtained from actual samples , if available , in the same way as seen classes , or can be simulated in a non-trivial way to represent potentially existing species . Our approach uses DNA as side information for the first time for zero-shot classification of species . In fine-grained , large-scale species classification , no other side information can explain class dichotomy better than DNA , as new species are explicitly defined based on variations in DNA . The hierarchical Bayesian model leverages the implicit inter-species association of DNA and phenotypic traits and ultimately allows us to establish a Bayesian hierarchy based on DNA similarity between unseen and seen classes . We compare DNA against word representations for assessing class similarity and show that the Bayesian model that uses DNA to identify similar classes achieves favorable results compared to the version that uses word representations on a well-known ZSL benchmark species dataset involving slightly less than 200 bird species . In the particular case of an insect dataset with over 1000 species , when visual attributes or word representations may not offer feasible alternatives , we show that our hierarchical model that relies on DNA to establish class hierarchy significantly outperforms all other embedding-based methods and feature generating networks . Our contributions are on three fronts . First , we introduce DNA as side information for fine-grained ZSL tasks , implement a Convolutional Neural Net ( CNN ) model to learn DNA barcode embeddings , and show that the embeddings are robust and highly specific for closed-set classification of species , even when training and test sets of species are mutually exclusive . We use the benchmark CUB dataset as a case study to show that DNA embeddings are competitive to word embeddings as side information . Second , we propose a fine-grained insect dataset involving 21 , 212 matching image/DNA pairs from 578 genera and 1 , 213 species as a new benchmark dataset and discuss the limitations of current ZSL paradigms for fine-grained ZSL tasks when there is no strong association between side information and image features . Third , we perform extensive studies to show that a simple hierarchical Bayesian model that uses DNA as side information outperforms state-of-the-art ZSL techniques on the newly introduced insect dataset by a wide margin . 2 Related Work . Zero-Shot Learning . Early ZSL literature is dominated by methods that embed image features into a semantic space and perform various forms of nearest neighbor search to do inference ( 14 ; 41 ; 1 ) . As the dimensionality of semantic space is usually much smaller than the feature space this leads to the hubness problem where some classes become hub and occur as the nearest neighbor of many samples . In an effort to alleviate the hubness problem , ( 50 ; 40 ) changed the direction of the embedding from semantic to image feature space . This was followed by a line of work that investigates bidirectional embedding between semantic and image spaces through a latent space ( 51 ; 43 ; 2 ; 32 ; 38 ) . In ( 25 ; 15 ) , a new strategy of synthesizing features for unseen classes and converting the challenging ZSL problem into traditional supervised learning is introduced ( 23 ; 44 ; 9 ; 13 ; 48 ; 52 ; 29 ; 39 ; 4 ) . Although feature generating networks ( FGNs ) currently achieve state-of-the-art results in ZSL , they suffer from the same problem as earlier lines of work in ZSL : hypersensitivity towards side information not strongly correlated with visual attributes . The vulnerability of both embedding and FGN-based methods toward sources of side information different than visual attributes , such as word vectors or WordNet hierarchy , is investigated in ( 2 ; 39 ; 44 ) . Another limitation of FGNs is that features generated for unseen classes are significantly less dispersed than actual features due to the generator failing to span more than a small subset of modes available in the data . Recent deep generative models mitigate this problem by proposing different loss functions that can better explore inter-sample and inter-class relationships ( 3 ; 7 ; 8 ; 19 ; 45 ) . However , these methods fail to scale well with an increasing number of classes with an especially high inter-class similarity ( 24 ) . Side Information in ZSL . Side information serves as the backbone of ZSL as it bridges the knowledge gap between seen and unseen classes . Earlier lines of work ( 22 ; 1 ) use visual attributes to characterize object classes . Although visual attributes achieve compelling results , obtaining them involves a laborious process that requires manual annotation by human experts not scalable to data sets with a large number of fine-grained object classes . When dealing with fine-grained species classification , apart from scalability , a more pressing obstacle is how to define subtle attributes potentially characteristic of species that have never been observed . As an alternative to manual annotation , several studies ( 11 ; 14 ; 2 ; 46 ; 34 ; 5 ) proposed to learn side information that requires less effort and minimal expert labor such as textual descriptions , distributed text representations , like Word2Vec ( 27 ) and GloVe ( 33 ) , learned from large unsupervised text corpora , taxonomical order built from a pre-defined ontology like WordNet ( 28 ) , or even human gaze reaction to images ( 20 ) . The accessibility , however , comes at the cost of performance degradation ( 2 ; 39 ) . A majority of ZSL methods implicitly assume strong correlation between side information and image features , which is true for handcrafted attributes but less likely to be true for text representations or taxonomic orders . Consequently , all these methods experience significant decline in performance when side information is not based on visual attributes . 3 Barcode of Life Data and DNA Embeddings . In this study , we present the fine-grained INSECT dataset with 21 , 212 matching image/DNA pairs from 1 , 213 species ( see Fig . 1 for sample images ) . Unlike existing benchmark ZSL datasets , this new dataset uses DNA as side information1 and can be best characterized with the high degree of similarity among classes . Among the existing benchmark datasets , SUN contains the largest number of classes ( 717 ) but classes in SUN represent a wide range of scene categories related to transportation , indoor and outdoors , nature , underwater etc. , and as such can be considered a relatively coarse-grained dataset compared to the INSECT dataset we are introducing in this study . All insect images and associated DNA barcodes in our dataset come from the Barcode of Life Data System ( BOLD ) ( 35 ; 36 ) . BOLD is an open-access database in which users can upload DNA sequences and other identifying information for any living organism on Earth . The database provides approximately 658 base pairs of the mitochondrial DNA barcode extracted from the cytochrome c oxidase I ( COI ) gene along with additional information such as country of origin , life-stage , order , family , subfamily , and genus/species names . Data Collection . We collected image/DNA pairs of insects that originate from three orders : Diptera ( true flies ) , Coleoptera ( beetles ) and Hymenoptera ( sawflies , wasps , bees , and ants ) . While the dataset is in general clean , manual effort was devoted to further curate the dataset . Only cases with images and matching DNA barcodes of adult insects are included . Images from each species were visually inspected and poor quality images were deleted . Only species with more than ten instances were included . The final dataset consisted of 21 , 212 images and 1 , 213 insects species of which 254 belong to Diptera ( 133 genera ) , 564 to Coleoptera ( 315 genera ) and 395 to Hymenoptera ( 130 genera ) . We extracted image features , namely image embeddings , using a pre-trained ( on ImageNet 1000 classes ) ResNet101 model ( 16 ) . Images are resized to 256× 256 and center-cropped before fed to the ResNet model . No other pre-processing is applied to the images . Data Split . We randomly chose 10 % of all species as unseen classes for the test set leading to 1 , 092 seen and 121 unseen classes . Similarly , we randomly chose 10 % of the 1 , 092 training classes as unseen classes for the validation set . Samples from seen classes were split by a 80/20 ratio in a stratified fashion to create seen portion of the train and test datasets . In the dataset there were a few hundred cases where multiple image views ( dorsal , ventral , and lateral ) of the same insect were present . To avoid splitting these cases between train and test , we made sure all instances of the same insect are included in the training set . As a result , 12 of the 1 , 092 seen classes in the training set were not represented in the test set . Our dataset splits are summarized in Table 1 . DNA Embeddings . Although it is the first time DNA barcodes are used as side information in ZSL domain , there have been some work investigating vector embeddings for DNA sequences . Authors of ( 31 ) trained a CNN model to do binary DNA sequence classification considering sequences as a text data . Imitating amino acid structure , each triplet of base pairs are treated as a word and sequences are converted into one-hot vector representation . Taking ( 27 ) as the base , ( 30 ) trained a shallow neural network on human genome data to generate representation for k-mers . Unlike these techniques we deal with DNA Barcodes represented by nucleotide sequences and aim to convert the entire character sequence into a vector embedding useful for species classification with more than 1,000 classes . 1Please refer to supplementary material for discussion on limitations of using DNA as side information Most recently , DNABERT ( 18 ) adapted the powerful text transformer model ( 10 ) to a genomic DNA setting and generated vector embeddings for long DNA sequences . In this paper , we trained a CNN model to learn a vector representation of DNA barcodes in the Euclidean space . First , the consensus sequence of all DNA barcodes in the training set with 658bp is obtained . Then , all sequences are aligned with respect to this consensus sequence using a progressive alignment technique implemented in MATLAB R2020A ( Natick , MA , USA ) . A total of five tokens are used , one for each of the four bases , Adenine , Guanine , Cytosine , Thymine , and one for others . All ambiguous and missing symbols are included in the others token . In pre-processing , barcodes are one hot encoded into a 658x5 2D array , where 658 is the length of the barcode sequence ( median of the nucleotide length of the DNA data ) . To train the CNN model , a balanced subset of the training data is subsampled , where each class size is capped at 50 samples . The CNN is trained with 14 , 723 barcodes from 1 , 092 classes . No barcodes from the 121 unseen classes are employed during model training . The training set is further split into two as train ( 80 % ) and validation ( 20 % ) by random sampling . We used 3 blocks of convolutional layers each followed by batch normalization and 2D max-pooling . The output of the third convolutional layer is flattened and batch normalized before feeding the data into a fully-connected layer with 500 units . The CNN architecture is completed by a softmax layer . We used the output of the fully-connected layer as the embeddings for DNA . Class level attributes are computed by the mean embedding of each class . The DNA-based attribute extraction is illustrated in Figure 2 . The details of the model architecture is depicted in Figure 3 in Supplementary material . We used ADAM optimizer for training the model for five epochs with a batch size of 32 ( with a step-decay initial learning rate = 0.0005 and drop factor= 0.5 , β1 = 0.9 , β2 = 0.999 ) . The model is developed in Python with Tensorflow-Keras API . Predictive accuracy of DNA embeddings . Although the insect barcodes we used are extracted from a single gene ( COI ) of the mitochondrial DNA with a relatively short sequence length of 658 base pairs , they are proven to have exceptional predictive accuracy ; the CNN model achieves a 99.1 % accuracy on the held-out validation set . Note that , we only used the data from training seen classes to train the CNN model . In order to validate the generalizability of embeddings to unseen data , we trained a simple K-Nearest Neighbor classifier ( K = 1 ) on the randomly sampled 80 % of the DNA-embeddings of unseen classes and tested on the remaining 20 % . The classifier had a perfect accuracy for all 121 but one classes with an overall accuracy of 99.8 % . In addition to our CNN model we have explored a DNABERT ( 18 ) model for converting DNA barcodes to vector embeddings . The pretrained DNABERT model achieves around 85 % ( vs 99 % from CNN ) top-1 KNN accuracy ( averaged over 10 runs ) on the unseen classes . Pretrained DNABERT can be fine-tuned for species classification however because of the vast number of parameters to tune each run takes a few hours on a relatively sophisticated GPU , significantly more than CNN training . Similarly , a simple LSTM model with half of the parameters as the CNN model is almost 5 times slower than the CNN model and requires more epochs to reach a reasonable accuracy . Therefore , we use a simple 3-layer CNN that trains in an hour and achieves almost perfect top-1 KNN accuracy . To demonstrate that the approach can be easily extended to larger members of the animal kingdom , we compiled approximately 26 , 000 DNA barcodes from 1 , 047 bird species to train another CNN model ( ceteris paribus ) to learn the DNA embeddings for CUB dataset ( see the Supp . materials for details ) . The CNN model achieved a compelling 95.60 % on the held-out validation set .

Summary: Authors propose to use DNA as side information for zero-shot image classification and a method that can leverage this information. The method is based on a Bayessian model that captures similarity between different classes as well as between images of a single class through global and local priors. Unseen classes are represented using the statistics of a fixed number of seen classes that are the most similar according to the side-information. Authors introduce an INSECT database and compare their method on INSECT and CUB with several other zero-shot methods. When using DNA as side information, the proposed method performs the best.