Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposed to use DNA as side information for the zero-shot learning tasks. 
A CNN is used to generate DNA embeddings, and a Bayesian model is used for zero-shot learning.
Experiments are performed on a newly collected fine-grained insect dataset with image and DNA pairs, as well as on the CUB dataset.
Results showed that the Bayesian model outperforms other methods on the insect dataset using DNA as attributes.
On the CUB dataset, using DNA is comparable with using word vectors, but pre-defined attributes still work better.

main_review: This paper proposed to use DNA for zero-shot learning, unlike prior works using attributes or text descriptions for side information. Results on the collected insect dataset show improvements over other methods. On the CUB dataset, using Bayesian modeling is better for modeling DNA information, but not on visual attributes or word vectors. 

In general, it is interesting to see that DNA data is helpful for ZSL, and a simple Bayesian model performs well on the DNA data.
The paper is well-written, and all the details of training and datasets are included. Ablation studies on the number of seen classes and hyper-parameters are also presented. 

Questions:
* Why use CNN (on top of one-hot encoding) instead of RNN or other methods for encoding the DNA data?
For the CUB dataset, the DNA embedding is trained on an external dataset of 1047 bird species. Is it a fair comparison between using attributes and word vectors (in Table 3)?
* As stated in L. 246-264, the performance of the Bayesian model is highly correlated to the ratio of the number of seen and unseen classes. If there are similar classes in the seen classes, then the model can perform well. Is this limitation the reason why using visual attributes performs better than using DNA on the CUB dataset?
* Another limitation of the proposed method is the trade-off between seen and unseen classes with different hyper-parameters (Fig. 5) Is there a way to mitigate this, i.e., having good performance on both seen and unseen classes?

Typo:
- Reference [5] is missing
- L.10 "of" CUB dataset
limitations_and_societal_impact: Some limitations have been discussed in the paper, and there is no negative societal impact.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: Authors propose to use DNA as side information for zero-shot image classification and a method that can leverage this information. The method is based on a Bayessian model that captures similarity between different classes as well as between images of a single class through global and local priors. Unseen classes are represented using the statistics of a fixed number of seen classes that are the most similar according to the side-information. Authors introduce an INSECT database and compare their method on INSECT and CUB with several other zero-shot methods. When using DNA as side information, the proposed method performs the best.
main_review: As of my knowledge this is the first paper proposing to use DNA as side information for zero-shot image classification. The hierarchical Bayessian approach to model the data is also relevant and I can imagine it becoming one of usual baselines (though it is a pity it is coded in matlab). The exposition is overall clear, find few suggestions below. An added value is also the INSECT dataset.

Suggestions:
- The main text falls short to explain how to arrive from the seen class statistics to computing PPD for an unseen class.
- Table 3 - Include what US/S/H means in the caption.
- Fig. 4 - Use a larger font. Put equal y-axis scale in all three plots.
- Line 76 - "state of the art" => "state-of-the-art"
- Line 133 - "larger" => "more"
- Line 209 - "farther" => "further"
- Line 325 - "fixed" => "stays fixed"
- Line 336 - "maintained" => "maintains"

Final recommendation:
I have read the other reviews and authors responses. I would like thank the authors for the responses and urge them to reflect the criticism in the final paper. I keep my recommendation to accept the paper (with the current score).

limitations_and_societal_impact: - BSZL performs better than the best SOTA only on the DNA.
- The obvious issue of using DNA as side information is that it is not applicable to non-living objects.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes using DNA as side information for fine-grained zero-shot learning of insects, as opposed to visual attribute labels. Their proposed approach establishes a hierarchy over the images using DNA information, via a hierarchical Bayesian model over a learned embedding of DNA barcodes. Their method particularly outshines attribute-label-based approaches on an insect identification application, where visual attributes (from public Wikipedia pages, for example) are sparsely labeled or not available. The provide as an additional contribution their paired DNA/image benchmark dataset containing 21,212 examples of 1,213 insect species.
main_review: This paper provides an interesting alternative, DNA, as side information for zero-shot learning.  The main drawback to the generality and scalability of this method is that it requires paired DNA information to be extracted from the specimen in question, which is not necessarily available for any given sighting - in fact it is most likely only available when digitizing and analyzing large natural history collections or within research labs that collect physical specimens (very common in entomology). That said, when that information is available it should be used, and this paper provides a clean formulation of the problem, benchmark dataset, and proposes a methodology for how to use DNA information effectively in zero-shot learning.

Some insect species, for example Rove Beetles, cannot be visually identified by experts unless they are dissected and the genitalia in inspected under a microscope. The error rates in classification from DNA barcoding can be up to 30%, and museum collections are known to contain labeling errors, particularly for challenging-to-identify species. Do the authors have a sense of the accuracy of their dataset subject to these known challenges in accurate insect ID?

It would be very interesting to see a visualization of the taxonomic distribution of the data, and particularly see the resultant distributions across the 3 orders considered of the random train/test/unseen splits (I’m imagining a taxonomic tree with the different splits colored at the end node, or possibly the proportion of categories in each split at each taxonomic level colored in some way).  It would also be good to show the distribution of examples per class in each split, to see the amount of imbalance represented. Further, it would be interesting to look at the distribution of these specimens across digitizing entities contribution to BOLD, to probe whether any potential correlations exist in digitization practice (camera type, background, DNA extraction method, etc) for certain taxa.

Is the DNA embedding network trained with a species classification loss? I assume so but it isn’t explicit in the text.

How much does the similarity of seen classes affect the local prior for surrogate classes (beyond just the ratio of seen to unseen)?  This would be interesting to explore. Perhaps you could probe how results shift based on the number of seen classes within some distance threshold in embedding space? Or build an explicit split based on taxonomy, with ie held out genera instead of species, to probe the robustness of the model to less “familiar” taxonomic branches?

How much is the performance gain over SOTA based on the fact that those methods were not designed with DNA in mind, but instead visual attributes or word vectors?



Text Suggestions & Nits:

In the introduction, your first sentence “Fine-grained species classification is essential in monitoring biodiversity.” is perhaps not well justified for a machine learning audience. It may help to provide a few examples of how it is used - what happens after you identify the species? 

In line 30, “fine-grained classification of species” does this refer to classification generally across data type? Or from images specifically? Throughout these first paragraphs clarification about what data type and problem formulation you are referring to would help.

In line 53: “or can be simulated in a non-trivial way 54 to represent potentially existing species.” Can you further elaborate? How would you simulate the DNA metadata for an image of an unseen species in a robust way in order to provide that side information to your model?

Line 82: give a quick high-level description of the hubness problem

Define US, S, H in the caption of Tables 2&3 so that readers do not need to search the text to understand the table, it’s also a bit odd that table 3 appears before table 2 in the document


limitations_and_societal_impact: One potential area of social impact would be risks associated with errors in such a system for insect ID, particularly if those errors are systematically biased in some way, in downstream biodiversity monitoring. It would be nice if the authors dug into these potential risks specifically in downstream insect biodiversity and conservation.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper proposes to use DNA as the side information for zero-shot species classification. The idea is novel and biologically sound. Experiments show that DNA embeddings achieve comparable performance as text embeddings when serving as side information.
main_review: Advantages:
1. This paper is the first work that utilizes DNA as side information for Zero-shot species classification, and the authors clearly explain the reasons and insights behind this.
2. This paper introduces a new dataset for Zero-shot species classification and provides codes to reproduce their experiments.

Weaknesses:
1. On the technique side, the paper's main contribution is developing a CNN model to calculate DNA embeddings. However, there are many existing works (e.g., dna2vec, Gene2vec, DNABERT, etc.) that calculate embeddings for DNA sequences. This paper doesn't mention, use or compare with them. Thus, it is unclear how novel/effective the proposed model is.

2. The paper is not well-written and relatively hard to follow. 
    -  It is not well-organized. In section 3, the authors introduce data collection, data processing, DNA embedding calculation, and some experimental details and results on DNA embedding, while in Section 4, they introduce the rest part of the proposed method. It may be better to re-organize this paper.
    - The Experimental setting and baseline models are not clearly explained. The tables are hard to understand. It can be very helpful to include more information in the table caption (e.g., explain the experimental setting, evaluation metrics, and meaning of the abbreviations).
    - Other issues:
        - Table 3 should appear after Table 2.
        - Caption of a table should appear above it.
        - The most important reference "[5]" that the proposed model is mainly based on is missing.
        - ......
        

3. As mentioned in Line 189-190, the Bayesian Zero-shot Learning introduced in Section 4 is previously proposed in the missing reference. It is unclear how this paper relates to that one.

limitations_and_societal_impact: I think the proposed idea is novel and interesting. The paper can be greatly improved if the paper is re-organized and more experiments are conducts.
review_title: null

Meta-review: This paper addresses the problem of zero shot learning (ZSL) in the context of fine-grained visual categorization using images from the natural world with (for the first time) DNA as side information. Existing work in this space typically exploits alternative forms of side information such as visual attributes or textual information. Experiments are performed using an existing Bayesian ZSL method on the CUB dataset (with additional DNA information) and a newly proposed dataset of insect images, and show that the models using DNA are competitive and often better than those using only textual side information. 

Issues that should be addressed in the final text. 
* The current related work text is missing a discussion of alternative methods for learning DNA embeddings. This needs to be remedied. (wkgT)
* Include the already performed comparison to DNABert. This is important as the CNN encoder is listed as one of the contributions of the paper but no comparisons are made to alternative methods for encoding this information. 
* Discuss the limitations of DNA side-information e.g. that it is not applicable to all visual categories (wkgT).
* Fix the issues with the writing and paper structure as noted by wkgT.
* Fix the issues with the text as noted by 2AZs and fQEh. e.g. Clarify in the caption for Table 3, what US, S, and H represent. 

The reviewers raised legitimate concerns regarding the level of technical contribution, the writing quality, and lack of discussion of other approaches for learning DNA embeddings. The final two of these concerns can be addressed in a revision of the text. Despite these concerns, the reviewers were supportive of the paper, with the most critical raising their score after the discussion. The novelty of the application and the merit of the new dataset were quoted as the reasons for recommending acceptance. The authors are strongly encouraged to address the above issues in the final camera ready text. 

As a final note, the authors should be aware that how they cited [5] was not consistent with the NeurIPS guidelines. As the paper was already published it should have been cited with the authors names and treated no differently than any other paper:  
"If you need to cite one of your own papers, you should do so with adequate anonymization to preserve double-blind reviewing.  For instance, write “In the previous work of Smith et al. [1]…” rather than “In our previous work [1]...”). If you need to cite one of your own papers that is in submission to NeurIPS and not available as a non-anonymous preprint, then include a copy of the cited submission in the supplementary material and write “Anonymous et al. [1] concurrently show...”)."
https://nips.cc/Conferences/2021/CallForPapers
Decision: Accept (Poster)