Prompting Scientific Names for Zero-Shot Species Recognition
Authors: Shubham Parashar
Zhiqiu Lin
Yanan Li
Shu Kong
Venue: EMNLP 2023 Main (2023)
Keywords: vision-language model
fine-grained recognition
zero-shot recognition
prompt engineering
species recognition
TL;DR: Using the vision-language model CLIP, we address zero-shot recognition of fine-grained species and propose a simple method, which achieves 2~4 times higher accuracy by translating species scientific names to common ones used in prompts.

Abstract: Trained on web-scale image-text pairs, Vision-Language Models (VLMs) such as CLIP can recognize images of common objects in a zero-shot fashion. However, it is underexplored how to use CLIP for zero-shot recognition of highly specialized concepts, e.g., species of birds, plants, and animals, for which their scientific names are written in Latin or Greek. Indeed, CLIP performs poorly for zero-shot species recognition with prompts that use scientific names, e.g., ``a photo of Lepus Timidus'' (which is a scientific name in Latin). This is because these names are usually not included in CLIP's training set. To improve performance, we explore using large-language models (LLMs) to generate descriptions (e.g., of species color and shape) and additionally use them in prompts. However, this method improves only marginally. Instead, we are motivated to translate scientific names (e.g., Lepus Timidus) to common English names (e.g., {\tt mountain hare}) and use such in the prompts. We find that common names are more likely to be included in CLIP's training set, and prompting them achieves 2$\sim$5 times higher accuracy on benchmarking datasets of fine-grained species recognition.

Prompting Scientific Names for Zero-Shot Species Recognition
Shubham Parashar', Zhiqiu Lin?, Yanan Li Shu Kong TTexss A&M University, CCanngiee Mellon University, *Zhejiang Lab, Institute of Collaborative Innovation, University of Macau
(shubhamprshr, shuj@tamu. edu, zhiqiulin@cmu.ed liynézhejianglab.om com, skongšum, skong@um.edu.ma md
Abstract
kiowificaates
Dana
Heie too photo of tose. Bax plaso of Mountan
Legus Turd
Hore paust the Wago
Trained on web-scale image-text pairs, Vision- Language Models (VLMs) such as CLIP (Rad- ford et al., 2021) can recognize images of com- mon objects in zero-shot fashion. However, it is underexplored how to use CLIP for zero- shot recognition highly specialized concepts, e.g., species of birds, plants, and animals, for which their scientific names are written in Latin Greek. Indeed. CLIP performs poorly for zero-shot species recognition with prompts that use scientific names, e.g., "a photo Lepus Timidus" (which is scientific This is because such names are usually not included in CLIP's training set. improve performance, prior works propose use large- language models (LLMs) to generate descrip- tions (e.g., of species color and shape) and ad- ditionally use them in prompts. We find that they bring only marginal gains. Differently, we are motivated to translate scientific names (e.g., Lepus Timidus) to common English names (c.g., mountain hare) and use such in the prompts. We find that common names more likely to be included in CLIP's training set, and prompting them achieves 2~5 times higher accuracy on benchmarking datasets of fine-grained species recognition.
monoonett Enghish conmmin rames
Bese
Maggee
oo sonstt prompess
Rise Hecal Neueraii Bonsa suat
We propose rather simple method that hoosts performance for zero-shot species recognition. It translates scientific names (written in Latin Greek) to common En- glish names, and use the latter in prompts. While using large language models (LLMs) can translate scientific names, they can fail for many species (Fig. 2). Instead, common names can be found in other publicly available sources such online collections and museums. By replacing scientific names with common names, simple prompt method achieves 2-5 times hetter zero-shot species recognition accuracy on challenging benchmarks. (Table I).
Figure I:
in Latin).
In other words, we study the problem zero-shot species recognition using VLMS.
Motivation. Zero-shot species recognition is practically meaningful in various applications, e.g., for education and ecological and biodiversity re- search, which desire automated recognition of species (Stork et al., 2021; Rodr'iguez et al., 2022). Building species recognition systems can hardly rely on traditional supervised learning methods which require a large set of labeled data, because labeling images with species names demands do- main expertise which is expensive to obtain. There- fore, propose to leverage pretrained VLMSs (e.g., CLIP) for zero-shot species recognition. Technical Insights. Species have scientific names written in Latin or Greek. Directly us- ing them in prompts (e.g., "a photo of Ponana Ci Citrina") does not allow OpenCLIP (an open- source CLIP) (Ilharco al., 2021) to perform well (Fig. 3). This not quite surprising, because, for the first time, we find that OpenCIIPP's training set LAION400M (Schuhmann et al., 2021) does not contain scientific names of many species. Per-
1 Introduction
Trained on large-scale image-text data, vision- language models (VLMs) such as CLIP (Radford et al., 2021; Ilharco et al., 2021) and ALIGN (Jia et al., 2021) can recognize images of common ob- jects in a zero-shot fashion, i.e., without further finetuning. Such success is typically reported on standard benchmarks such ImageNet which con- tains 1,000 common classes, c.g., CLIP achieves 76.2% zero-shot accuracy vs. 11.5% by Li et al. (2017) prior to CLIP. We are motivated to use CLIP for zero-shot recognition of highly specialized con- cepts, e.g., species of plants, birds, and animals. Corresponding to Yanan Li and Shu Kong.

---

S-name
S-name + dep.
C-name
35
39.8
15
Modil GPP
INat
Aves
Con you thei vitut attittnns of Forona Chtnn nus photo?
Figure 3: We compare methods (1) vanilla prompt (Radford et al., 2021) using scientific names which are written in Latin or Greek (S-name), (2) S-name plus descriptions (Menon and Vondrick, 2022), and (3) our prompt method using common names (C-name). Results are top-1 accuracy on two datasets (iNat and Aves detailed in Sec. 4.1). Clearly, using scientitic names in prompts yields poor zero-shot species recognition performance because pretrained VLMs do not necessarily see scientific names. Additionally using descriptions improves marginally. In contrast, simply translating them to common English names significantly boost performance.
sorny but of my trawinn ralvotf Sepptembee 2021, information "Fonara Chi no" cous
specitt:
missceel ing night torm apssica or variant that besn descrited after my training data. Vita plant, animal soject you
identiy apeetti
shape, rin,, potwens neture end yyou hawe more informetiona or contest, please provide and It took essit based pencral koowlodge
Figure 2: Top-row: four images from the species Ponana Citrina; bottom-row: GPT4 fails answer questions related this species, demonstrating limitation of LLMs un- derstanding scientific names.
2 Related Work
haps surprisingly, the recent large-language model (LLM) GPT4 can also fail to answer questions related to species scientific names (Fig. 2), such "can you describe the appearance of Ponana Citrina?"' Inspired by Menon and Vondrick (2022), to improve zero-shot recognition perfor- mance, we additionally use descriptions in prompts, hoping they provide useful contextual information. However, it only marginally improves performance. On the other hand, although scientific names are not frequently included in OpenCLIP's training set, their corresponding English common names are. Therefore, we are motivated translate sci- entific names common ones and use the latter in prompts (Fig. 1). Our embarrassingly simple method significantly boosts performance by 2~-5 times! To note, while LLMs can fail to understand scientific names' and so provide common names, common names can be found in other sources such online collections and museums.
Zero-shot recognition. Previous methods ex- ploit auxiliary semantic information such as at- tributes (Lampert et al., 2013) or distributed word embedding (Elhoseiny et al., 2017) of class names and learn mapping between them for knowledge transfer from base to novel classes (Xian et al., 2018). Recently, zero-shot VLMs pre-trained on large-scale image-text corpus such as CLIP (Rad- ford al., 2021) achieve state-of-the-art perfor- mance on a wide range of image recognition bench- marks. Specifically, these models do not require fine-tuning on downstream datasets so long as the class names are provided. In this work, we use OpenCLIP (Ilharco et al., 2021) the zero-shot VLM for recognition but focus and underexplored tasks, i.e., species recognition. Prompting approaches. To use VLMs for zero- shot recognition, it is crucial to design prompts. Initially, prompts are manually engineered (Rad- ford et al., 2021) which requires trials and errors. Some methods propose to learn prompt templates (Zhou et al., 2022; Bulat and Tzimiropoulos, 2022), which require labeled data and are hard opti- mize. Some recent work proposes to exploit LLMs (Bubeck et al., 2023; Touvron al., 2023) gen- erate prompts for given concepts (Menon and Von- drick, 2022: Pratt et al., 2023). In this work, we focus on zero-shot species recognition, for which LLMs might fail to provide useful information to help construct prompts (Fig. 2). Further, different from existing prompting approaches, rather simple method by using species common names (translated from their scientific names). To the best of our knowledge, ours the first that translates scientific names to common names in prompts for zero-shot recognition.
specialized
Contributions are two-fold.
1.
We study the underexplored problem of zero- shot species recognition using VLMs. We confirm that current prompting methods yield poor performance and find that the culprit is the scientific names written in Latin or Greek, most of which are not in VLM's training set.
2. We propose an embarrassingly simple method that translates scientific names to common names, boosting zero-shot species recognition accuracy by 2~5 times.
propose
is common that GPT4 can fail to answer human's 's ques- tions, hut its web browsing mode mitigates this by retrieving information on the fly from existing (professional) websites.

---

3 Methods
Table 1: Benchmarking results of zero-shot species recog- nition on the iNat and Aves datasets. We use two VLMs: OpenCLIP VIT-B/32 and VIT-L/14 (which is a bigger model). Clearly, using bigger model produces higher accuracy across methods. The Vanilla method that prompts scientific names (S-name) yields quite low accuracy: additionally using de- scriptions improves accuracy slightly. In contrast, simply prompting common names (C-name) by translating scientific names, we boost performance by 2X on iNat and 5X on Aves, justifying our hypothesis that OpenCLIP might not be versed in understanding scientific names (hecause its training texts do not contain such). When using common names, additionally exploiting descriptions does not necessarily improve perfor- mance. We conjecture that the common names plus descrip- tions not frequently seen by OpenCLIP during training. Furthermore, our final method F-name, which uses the more frequent name between scientific and common names for each species, is among the hest methods across methods, datasets, and OpenCLIP models.
We start with the vanilla prompts that have been used in the contemporary literature of zero-shot recognition (Radford et al., 2021).
Vanilla Prompt uses the prompt template "a photo of Lepus Timidus" to get a similarity mea- sure for given input image. Note that for this method, we use species scientific names (written in Latin or Greek) in the prompts, hence call this method S-name. Concretely, feeds prompts to CLIP's s text encoder for the text features, uses them to compute cosine similarities to the visual feature computed by the visual encoder for the given image. CLIP tends to produce a high similarity for the cor- rect prompt (corresponding to ground-trutt truth class name), hence achieving zero-shot recognition.
iNat
Aves
VLM
Prompt Method
(810-way)
(200-way)
S-name (vanilla)
6,84%
7.05%
+ descriptions
8.17%
7.75%
Prompt with additional descriptions enriches prompts with descriptions (e.g., shape, color, and size). These descriptions provide contextual infor- mation for the targeted concepts and help zero-shot recognition (Menon and Vondrick, 2022). Recent work proposes to generate such descriptions using LLMs (Pratt et al., 2023). Although using LLMs does not always work on specific species (Fig. 2), we follow this practice to generate descriptions using GPT4. When GPT4 fails to provide useful descriptions for any species, we use the simple prompts without any descriptions by default. Note using additional descriptions apply to both vanilla prompt and methods below.
C-name (ours)
13.51%
39.80%
ViT-B/32
descriptions
14.42%
39.75%
F-name (ours)
13.88%
39.95%
descriptions
14.47%
39.40%
S name (vanilla)
9.21%
11.10%
descriptions
10.15%
12.50%
C-name (ours)
20.17%
59.00%
VIT-L/14
descriptions
20.04%
59.90%
F-name (ours)
20.32%
58.45%
descriptions 20.03%
59.10%
frequency counts of both scientific and common names for all the species of interest and use the more frequent one in the prompt for the correspond- ing species.
Experiments
Prompt with common names translated from scientific names (C-name). We investigate Open- CLIP's training set, i.e., LAION400M (Schuhmann et al., 2021), and find that it does not sufficiently cover scientific names (e.g., 468 out of 810 species in the semi-iNat dataset have their scientific names in LAINON400M). Instead, many species have common names that are in LAION400M. There- fore, we turn to external resources (c.g., online collections and Wikipedia) to translate scientific names (written in Latin or Greek) common En- glish names and use the latter in prompts. For the species that do not have any common English names (only a few in the benchmarking datasets), we use their original scientific names by default. Prompt with more frequent names (F-name) between common and scientific names. We hy- pothesize that it is important to use the texts that are more frequently encountered by OpenCIII during training. Therefore, we further analyze LAION400M (Schuhmann et al., 2021) to obtain
We conduct extensive experiments to validate our methods. We start with implementation details, datasets and the evaluation metric, and then analyze benchmarking results with in-depth analyses.
4.1 Setup
Implementations. We implement all the meth- ods in PyTorch on single NVIDIA A100 GPU. We use the open-source OpenCLIP (i.e., the ViT- B/32 and ViT-L/14 models) (Ilharco al., 2021). Datasets. We use four datasets that require species-level recognition. While they were initially curated for fine-grained classification via super- vised learning, we repurpose them for zero-shot species recognition, i.e., using their validation sets for benchmarking.
The semi-iNaturalist (iNat) (Su and Maji. 2021b) contains 810 species, covering a broad spectrum of organisms including mammals, plants, birds, insects, fungi, etc.

---

Table Further analysis on Flowers (for plant recog- nition) and CUB200 (for bird species recognition) datasets. Breakdown results in Fig. 4 show that using common names in prompts improves significantly on organism types such as birds but induces a decrease on plants. Therefore, we use the two more datasets to confirm our hypothesis: OpenCLIP has already seen most of plant species names (different from birds) and hence perform well by prompting scientific names. Indeed, all the prompt methods perform well on Flowers 102, although our F- F-name method performs the best with the bigger model VIT-I/14. Differently, on CUB200, S-name that uses scientific names does not perform well (6.08% with VIT B/32), but C-name (which uses common names in prompts) boosts accuracy to 59.42% (with VIT-B/32).
MAVIVALS
41.43
BIRDS
$1.32
FISHES
26:15
REPTILES
AMPHIBIANS
10.13
ARACHNIDS
7.35
INSECTS
6.98
SEA URCHIN
FUNGI
Delta Accuracy
Figure 4: Compared with using scientific names in prompts, using common names achieves significantly higher zero-shot accuracy on all organism types except plants. We hypoth- esize that scientific names of plants are more likely to be included in OpenCLIP's training set (LAION400M (Schuh- mann et al., 2021)), such that OpenCLIP performs well when directly prompting scientific names. We investigate this by matching the scientific names and LA1ON400M texts, and find LAION400M contains 65% plant species' scientific names of iNat. To confirm this further, we use the Flowers 102 dataset for plant species recognition (Table 2).
Flowers102
CUB200
VLM
Prompt Method
(102-way)
(200-way)
S-name (vanilla)
66.43%
6.08%
+ descriptions
68.67%
7.61%
C-name (ours)
61.89%
56.01%
VIT-B/32
+ descriptions
63.33%
58, 58.44%
F-name (ours)
68.35%
58.35%
+ descriptions
66.91% %
59,42%
S-name (vanilla)
77.28%
6,92%
The semi-Aves (Aves) (Su et al., 2021) rated for 200 bird species recognition.
)) is cu-
+ descriptions
78.53%
10.44%
C-name (ours)
73.49%
76.27%
VIT-/114
descriptions
74.67%
76.30%
The Flowers 102 by Nilsback and Zisserman (2008) is a popular fine-grained classification dataset that contains 102 flower types.
F-name (ours)
78.95% %
76.70%
descriptions
78.95%
76.87%
LAION400M, generally performs the best across datasets and VLM models.
The CUB-200-2011 Bird (CUB200) by Wah et al. (2011) is popular fine-grained classifi- cation dataset that contains 200 bird species. Metric. We use the top-1 accuracy averaged over K per-class accuracy as the metric, where K is the total number of classes.
4.3 Analysis
Focusing iNat which contains multiple organ- ism types, show performance gain of using com- mon names vs. scientific names for each type in Fig. 3. Using common names performs much bet- ter on all organisms (e.g., birds) except plants. This motivates us to investigate the reason using the Flower1 02 (for plant species recognition) and CUB200 (for bird recognition) datasets. Table 2 lists the results and the caption details conclusions.
4.2 Results
Table lists benchmarking results. Please refer to the caption for detailed conclusions. We reiter- ate three salient conclusions. First, using common names in prompts performs significantly better than scientific names. It is worth noting that, on Aves, simply using common names in prompts achieves 59.0% zero-shot recognition accuracy (59.9%), which is even higher than the fully-supervised method (56.6%) that pretrains on ImageNet Su et al. (2021)! Second, additionally using descriptions, al- though paired with common/scientifie names, are not always beneficial, shown by results Aves; it can even decrease results (e.g., from 20.32% to 19.93% with the F-name method and ViT-L/14). We conjecture the reason is that OpenCLIP was not trained on texts (similar to prompts) contain- ing species names plus descriptions. Third, F- name, the method that uses the more frequent name between scientific and common names in
Conclusions
We study the problem of zero-shot species recog- nition using pretrained vision-language models (VLMs). For this problem, the typical prompt method that uses the species scientific names "by default" performs poorly, and additionally using descriptions in prompts improves only marginally. We find that scientific names written in Latin or Greek are not frequently seen in VLMs' training set, explaining why they do not work well. Yet, species English common names more likely to be seen in VLMs training set. Hence, we propose rather simple method that translates scientific names to common English names, and use the latter in prompts. This method significantly boosts aceu- racy by 2-5 times on three benchmarking datasets.
CCB2200 and Flowers provide common names. We translate them to scientific names for the vanilla method in experiments.

---

Table 4: We show top-1 accuracies (in %) by S-name (using scientific names in prompts) and C-name (using common names in prompts) methods on different species types of the iNat datasets. This table supplements Fig. 4. Using scientific names improves accuracy on all species types except plants, on which it achieved decreased accuracy 9.4%, compared to using common names (10.83%).
(a) Lepus Timidus Mountain Hare
(b) Ciconia Nigra Black Stork
Species Type
S-name
C-name
mammals (14)
20.00
61.43
birds (59)
6.10
37.62
fish (13)
6.15
32,30
reptiles (19)
4.21
24.21
snails (8)
0.00
12.5
(c) Nymphaea Water Lily
(d) Sayorna Say's Phoebe
shellfish (12)
16.67
28.34
amphibians (9)
4.45
15,56
arachnids (28)
2.14
9.29
Figure 5: Visual illustrations showcasing various class ex- amples from the benchmarking datascts. Under cach image, provide the corresponding species scientific name (top row) and English common name (bottom row). (a) iNat en- compasses broad spectrum of species, including mammals, plants, and fungi. (b) Aves comprises 200 bird species ex- tracted from the INaturalis20188 dataset. (c) Flowers 102 collection 102 distinct flower species. (d) CUB200 a bird species recognition dataset that includes 200 bird species.
insects (264)
2.88
9.84
sea urchin (3)
6.67
13.3
fungi (45)
4.45
10.67
plants (336)
10.83
9.40
Table 5: We show our examples using Pica pica, a species in the Aves dataset, known as the common magpie. We use the scientific name, Pica Pica, as part of the prompt for the first method (s-name), and the common name, Common Magpie, for the second method (c-name). The frequency of Common Magpie and Pica Pica in LAION400M (Schuhumann et al.. 2021) is then examined. Notably, the scientific name, Pica Pica, has higher frequency and is employed for the third method (f-name). Finally, descriptions for this class are obtained by prompting GPT4, forming our description based methods as in (Menon and Vondrick, 2022), and use with scientific, common, and high-frequency names.
Table 3:
We show the the counts of species scientific and com- mon names which have occurance in LAION400m (Schuh- mann et 2021). which is the training set of the VLM model OpenCIII IIP used in our work. We find that substantial portion of species scientific names (especially from iNat) are not in LATON400M, This helps explain why using scientific names in prompts does not work well for zero-shot species recogni- tion. Moreover, when also counting common names altogether with scientific names. we find that almost all species have then names in LAION40MM. This explains why our F-name (using more frequent names of each species in prompts) generally performs the best. It is worth noting that in 102. nearly all scientitic names are in LA1ON400M, explaining why the vanilla prompt method using species scientific names performs exceptionally well on this dataset (Table 2).
Prompt Method
Prompt Example
S-name
Here is a photo of the Pica pica. Pica pica has a blue tail.
descriptions
C-name
Here is photo of the common magpie.
descriptions
Cormon magpie has a blue tail. Here is photo of the common magpie. + Pica pica has blue tail.
F-name
descriptions
iNat
Aves
Flowers102
CUB200
Names Found
(810)
(200)
(102)
(200)
LAION400M (Schuhmann et al., 2021). Table 3 demonstrates that scientific names alone have lim- ited occurrence in LAION400M, particularly in more specialized datasets like iNat and Aves. For a comprehensive analysis, please refer to the caption.
S-names
468
173
98
190
+ C-names
781
200
101
200
Appendix
A Visual Examples
C Breakdown Accuracy for Different Species Types
We conducted experiments on four distinct datasets, each primarily focusing on a unique set of species. Aves (Su and Maji, 2021a) and CUB200 (Wah et al.. 2011) encompass 200 bird species each. iNat (Su and Maji, 2021b) consists of a diverse range of species, including plants, animals, and fungi, total- ing 810 species. Flowers102 (Nilsback and Zisser- man, 2008) is a relatively smaller dataset specifi- cally dedicated to 102 flower types. Fig. 5 displays some random images from these datasets.
Table 4 lists the accuracy of each species type on the iNat dataset by prompting s-name and c-name. respectively. It augments Fig. 4.
D Examples of Prompts
Table 5 shows some prompt examples. Vanilla prompts employs the prompt template "Here is a photo of the ppeccies-scienttiii---aaam When additionally using descriptions, adopt the same template as in (Menon and Vondrick, 2022). which is "species-sciente has blah-blah", where blah-blah can be shape (e.g., "a long tail"), color (e.g., "a blue tail"), etc.
B Names Covered by Pretraining Data We use the VLM model of OpenCLIP (Ilharco et al., 2021) in our work, and the training set is
name

---

Limitations
A. Li, A. Jabri, A. Joulin, and van der Maaten. 2017. Learning visual n n-grams from web data. In ICCV.
Vision-language models (VLMs) and large- language models (LLMs) could learn bias and un- fairness from their pretraining set. As our work exploits publicly available VLMs and LLMs, we do not address these issues and do not study how zero-shot species recognition suffers from them. Moreover, our work focuses on zero-shot setup for species recognition. In practice, even though species-level annotations demand expertise knowl- edge and are expensive to obtain, there are some annotated data in online libraries, online collections and museums. Leveraging such available data to improve species recognition is future work.
Sachit Menon and Carl Vondrick. 2022. Visual classi- fication via description from large language models. In ICLR.
Maria-Elena Nilsback and Andrew Zisserman. 2008, Automated flower classification over a large number of classes. In Indian Conference Computer Vision, Graphics & Image Processing.
Sarah Pratt, Ian Covert, Rosanne Liu, and Ali Farhadi. 2023. What does platypus look like? generating customized prompts for zero-shot image classifica- tion. In ICCV.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, et al. 2021. Learning transferable visual models from natural language supervision. In ICML.
Ethics Statement
We focus recognizing species of organisms (e.g.. animals, birds, fungi, plants, etc.), we do not envi- sion ethic issues in our work.
Andr'es es C. Rodr Rodr'iguez, Stefano D D'aronco, Ro- drigo Caye Daudt, Jan Dirk Wegner, and Konrad Schindler. 2022. Zero-shot bird species recog- nition by learning from field guides. ArXiv, abs/2206.01466.
A Acknowledgements
This work is supported by NSFC (No. 62.662262566.
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. 2021. Laion Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv:211.02.114
Shu Kong acknowledges the support by the Univer- sity of Macau (SRG2023-00044FFST,,, Shubham Parashar acknowledges the compute resource and travel support from the Department of Computer Science and Engineering at Texas A&M University.
Lise Stork, Andreas Weber, Jaap van den Herik, Aske Plaat, Fons Verbeek, and Katherine Wolstencroft. 2021. Large-scale zero-shot learning in the wild: Classifying zoological illustrations. Ecological Infor- matics, 62:101222.
References
Sébastien Bubeck, Varun Chandrasekaran. Ronen Eldan,
Johannes Gehrke, Eric Horvitz, Ece Kamar, et al. 2023. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv:2303.12112
Jong-Chyi Su, Zezhou Cheng, and Subhransu Maji. 2021. A realistic evaluation of supi-supe leam- ing for tne-grained classification. In CVPR.
Adrian Bulat and Georgios Tzimiropoulos. 2022. Language-aware soft prompting for vision & lan- guage foundation models. a1Xiv:2210001115.
Jong-Chi mng-Chyi Su and Subhransu Maji. 2021a. The semi-supervised inaturalist-aves challenge at FGVC7 workshop. CoRR, abs/2103.06931)
Jong-Chyi Su and Subhransu Maji. 2021b. The semi- supervised inaturalist challenge at the FGVC8 work- shop. CoRR, abs/2106.011 06.01364.
Mohamed Elhoseiny, Yizhe Zhu, Han Zhang, and Ahmed Elgammal. 2017. Link the head to the "beak": Zero shot learning from noisy text description at part precision. In CVPR.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, al. 2023. Llama: Open and efficient foundation lan- guage models. 132.17230..13971
Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, et al. 2021. Openclip.
C. Wah, S. Branson, P. Welinder, P. Perona, and S. Be- longie. 2011. The caltech-uesd birds-200-2011 dataset. Technical report.
Chao Jia. Yinfei Yang, Ye Xia. Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun-Hsuan Sung. Zhen Li, and Tom Duerig. 2021. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML.
Yongqin Xian, Christoph H Lampert, Bernt Schiele and Zeynep Akata. 2018. Zero-shot learning comprehensive evaluation of the good, the bad and the ugly. PAMI, 41(9))225--2265
Christoph H Lampert, Hannes Nickisch, and Stefan Harmeling, 2013. Auribute-based classification for zero-shot visual object categorization. PAMI, 36(3):453-465
Kaiyang Zhou, Jingkang Yang. Chen Change Loy, and Ziwei Liu. 2022. Learning to prompt for vision- language models. IJCV, 23377)23377


Reviews:
[{"review_id": "30vJrcKg8N", "reviewer": "Reviewer_Duti", "summary": "", "questions": "Have you considered the lack of a one one onto mapping between common names and scientific names? How does your method deal with ambiguities between a horse and a sea horse or a lion and a sea lion or  a pig and a guinea pig? \nUpdate: the authors have provided a convincing response.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer Duti for the insightful comments. Reviewer Duti thinks our work has an \"interesting finding backed by great improvement over the state of the art.\" Reviewer Duti has a major concern about ambiguity in common names, as well as a minor one about our \"embarrassingly simple\" method. We address them below. \n\n> **Reviewer Duti thinks \"common names on the other hand are often ambiguous and downright confusing\" because \"there may not be a bijective mapping\" between scientific names and common names. The reviewer explains this ambiguity with multiple examples and asks how we handle this issue, e.g., \"prairie dogs\" vs. \"dogs\",  \"guinea pigs\" vs. \"pigs\", \"a lion\" vs. \"a sea lion\", \"a horse\" vs. \"a seahorse\".**\n\nGreat comment! The comment is partially correct -- it is right that there might not always be one-to-one (bijective) mapping. Concretely, it is quite often that a scientific name can have multiple common names. But, importantly, a common name corresponds to only one scientific name. To quickly justify this, we can find the common names for a scientific name using wikipedia; for a common name, we can do the same but wikipedia \"translates\" the common name to a unique scientific name. That said, common names are not ambiguous. We explain further below.\n\n\nThe examples provided by the reviewer look ambiguous to a human because they have the same keywords, e.g., keyword \"lion\" in \"a lion (Panthera Leo)\" vs. \"a sea lion (Otariinae)\", keyword \"horse\" in \"a horse (Equus caballus)\" vs. \"a seahorse (Hippocampus guttulatus)\". \nFor the examples provided by the reviewers, we find their corresponding scientific names using wikipedia, shown below. Clearly, each common name corresponds to a single unique scientific name. Hence, common names are not ambiguous. \n - lion → Panthera Leo\n - sea lion → Otariinae\n - horse → Equus caballus\n - seahorse → Hippocampus guttulatus\n - pigs →  S. domesticus\n- guinea pigs → Cynomys socialis \n - prairie dogs → Cynomys socialis\n - dogs → C. familiaris\n\nImportantly, pre-trained vision-language models (e.g., OpenCLIP) have learned *the correspondence between (unambiguous) common names and visual characteristics*, hence using common names (frequently seen in the pretraining set) for zero-shot recognition is much better than using infrequently-seen scientific names.  To support our claim, we did two experiments of binary zero-shot recognition  between (1) lion vs. sea lion, and (2) pigs vs. guinea pigs. These classes are in the popular Imagenet-1K dataset and we use OpenCLIP to zero-shot recognize their images (following the standard text-prompt method described in our paper). Below are the confusion matrices between the two pairs of classes. Clearly, OpenCLIP does not confuse these common classes!\n\n- Confusion matrix of zero-shot recognition between lion and sea lion.\n  |class| lion| sea  lion|\n  |--------|---------:|------:|\n  |lion| 100.0% |  0.0%|\n  |sea lion       | 0.0%  | 100.0%|\n\n- Confusion matrix of zero-shot recognition between pig vs guinea pig.\n  |class | pig| guinea pig|\n  |--------|---------:|------:|\n  |pig     | 100.0% |  0.0%|\n  |guinea pig| 6.0%  | 94.0%|\n\n\n> **Reviewer Duti is concerned that our method is “embarrassingly simple” and suggests doing some analysis to understand \"what the implications of their result are\".**\n\nIn the literature, simple methods tend to make big impacts. To the best of our knowledge, our work is the first that shows improving the vocabulary of category names (i.e., translating scientific names to common names) significantly improves a downstream task (i.e., zero-shot species recognition). In contrast, prior work has focused on more sophisticated techniques to exploit attributes [1] and prompt templates [2] to improve zero-shot performance. Hence, our method is not only simple but also novel in this field.\n\nIn terms of more analyses, we have listed some in Table 3 in the appendix after reference. May we politely ask the reviewer to have a look and we are delighted to do more per your suggestion. Yet, towards a four-page short paper, it is difficult to include all the analyses in the main text. The implication of our results enables rich future work as briefly mentioned in Line274-280, e.g., how to leverage the frequently-seen common names to finetune / adapt such pre-trained models in downstream tasks. \n\n - [1] Menon et al., \"Visual classification via description from large language models\", ICLR, 2023.\n - [2] Radford et al., \"Learning Transferable Visual Models From Natural Language Supervision\". ICML, 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Interesting finding backed by great improvement over the state of the art.", "reasons_to_reject": "1. The technique itself is \"embarrasingly\" simple as per the authors own description. While that is not a knock against it, the authors do not dig deeper into what the implications of their result are. It is not shocking that CLIP would be trained with data that does not have scientific names in it. However, scientific names are assigned so there are unambiguous names for species. Common names on the other hand are often ambiguous and downright confusing. For example, prairie dogs are rodents while dogs are canines, Similarly guinea pigs are not porcine at all.  A lion is a feline but a sea lion is a mariine mammal. The authors are satisfied with the improvements in the results that they have achieved but do not probe for cases where there may not be a bijective mapping between scientific and common names. \nupdate: The authors have provided a convincing response to this question.", "questions_to_authors": "Have you considered the lack of a one one onto mapping between common names and scientific names? How does your method deal with ambiguities between a horse and a sea horse or a lion and a sea lion or  a pig and a guinea pig? \nUpdate: the authors have provided a convincing response.", "paper_topic_and_main_contributions": "This paper describes a way to improve retrieval of bird, plant and animal species using scientific names through a translation approach in which they translate the scientific name into a common name before prompting a large multimodal model such as CLIP. The authors achieve retrieval results that exceed the state of the art by an order of magnitude.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "No missing reference to my knowledge.", "Typos_Grammar_Style_and_Presentation_Improvements": "No typos that I could see."}, "extra_text": {}}, {"review_id": "E2BKcSmbtG", "reviewer": "Reviewer_tzwz", "summary": "", "questions": "A: Considering most of the class names from CUB were in the CLIP pre-training dataset (Table 3), can authors elaborate why there is a massive performance boost with the proposed method?", "limitations": "", "rating": 3, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer tzwz for the valuable comments. Reviewer tzwz thinks the \"proposed idea is very simple\" which \"is a bold move and can open-up an interesting research venue.\" The reviewer has three concerns and we address them below.\n\n> **Reviewer tzwz questions whether our method can generalize because, as stated by the reviewer, \"one of the reasons that the model works well on CUB dataset is that the CUB dataset is not very fine-grained\" (which has only 200 species, and each genus has <5 species), whereas \"there are more than 2000 beetles\".**\n\nThe statistics of the CUB dataset is correct, but we respectfully note that the CUB dataset is well-established in the field of fine-grained recognition although it has <5 species per genus. Importantly, another dataset we focus on (Table 1, Fig. 3-4) is iNat, which is more challenging than CUB because it has 810 species and some genera have upto 10 species. As noted by the reviewer, we evaluate our method \"on 4 benchmark datasets\"; extensive results sufficiently demonstrate that our method generalizes well across datasets. We agree that nature has way more species (e.g., >2000 beetles), but there does not exist such large-vocabulary fine-grained datasets for experiments. We hope future work will collect such a dataset because we are also curious about how typical zero-shot recognition methods and ours perform on it!\n\n\n> **Reviewer tzwz comments that \"No ZSL methods are tested as a baseline\" and recommends some ZSL papers.**\n\nThank you for the recommended papers and we will happily cite them in the camera-ready. We would like to first emphasize the crucial difference (as briefly contrasted in Line 096-112) between the classic zero-shot learning (ZSL) setup and the current zero-shot recognition setup enabled by pretrained vision-language models (VLMs).\n\n - The classic ZSL setup in computer vision refers to the study of generalizing to **unseen classes** in image classification while performing knowledge transfer from seen classes using some semantic information such as attributes. The papers you recommend fall in this setting.\n\n - The setup in our work refers to the new **zero-shot transfer** setup formulated in the CLIP paper [1] that aims to leverage a pre-trained VLM for zero-shot recognition in **unseen imagery datasets** prompted by texts, i.e., allowing methods to be based on prompting.\n\nIn the new setup, [1] shows the prompting-based approach that uses pre-trained VLMs resoundingly outperforms classic ZSL methods. Therefore, we treat the suggested prompting method [1] as our strong baseline. Furthermore, we compare the most recent prompting method [2] which uses LLMs (i.e., GPT3) to extract more semantic information to augment text prompts for better zero-shot performance. In our work, we reimplement [2] by using a more powerful LLM (i.e., GPT-4) to report its performance (Table 1). We will clarify and discuss the above in the camera-ready version.\n\n\n - [1] Radford et al., \"Learning Transferable Visual Models From Natural Language Supervision\". ICML, 2021.\n - [2] Menon et al., \"Visual classification via description from large language models\". ICLR, 2023.\n\n\n\n\n> **Reviewer tzwz asks, \"considering most of the class names from CUB were in the CLIP pre-training dataset (Table 3), can authors elaborate why there is a massive performance boost with the proposed method\" (which incorporates common names and use the most frequent name for a given species in prompting).**\n\nGreat question! The conclusive explanation is that, *aligning a downstream task's vocabulary of categorical names to the more frequently seen (scientific/common) names by the pretrained model OpenCLIP allows leveraging this model to better serve the downstream task*. Let us explain further below. \n\nIntuitively, when a pretrained model has seen common names much more frequently than scientific names, it understands the common names better. To justify this, we assemble results on the CUB dataset in the table below. The table lists the counts of scientific and common names included in OpenCLIP's pretraining dataset LAION400M, and zero-shot accuracies by prompting the two sets of names, respectively. Clearly, common names have been seen by OpenCLIP more than 10 times than scientific names, and prompting common names yields 56.01% zero-shot accuracy, 9x better than prompting scientific names (6.08%)!\n\n| #captions with that have | s-names | c-names |\n|------|------:|----:|\n| counts |  49,506     | 621,275     |\n| zero-shot accuracy using  | 6.08%  | 56.01% |\n\nLastly, our final method is to prompt more frequent names among scientific and common names, hence we call this method f-name. From Table 2 of our paper, simply doing so improves accuracy to 58.35%, notably higher than using common names only (56.01% as shown by the table above). This further demonstrates the conclusion that, *aligning a downstream task's vocabulary of categorical names to the more frequently seen (scientific/common) names by the pretrained model OpenCLIP allows leveraging this model to better serve the downstream task*.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed idea is very simple and it is validated on 4 benchmark datasets by consistently outperforming the baselines.\n\nThe idea of using VLMs for fine-grained datasets is a bold move and can open-up an interesting research venue.\n", "reasons_to_reject": "In real-world scenario with hundreds of species from the same genus or family, I wonder how this method will generalize. To illustrate, there are more than 2000 beetles and not many of them has common names. One of the reasons that the model works well on CUB dataset is that the CUB dataset is not very fine-grained. Most of bird species are well known ones and have common names. There are most 4-5 species belonging to the same genus in that dataset. \n\nNo ZSL methods are tested as a baseline. It would be very beneficial for the paper to show how the proposed method (VLM + translated class names) performs against some of traditional ZSL methods [1, 2, 3, 4]\n\n\n\n[1] Xian, Y., Lampert, C. H., Schiele, B., & Akata, Z. (2018). Zero- shot learning— A comprehensive evaluation of the good, the bad and the ugly. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(9), 2251– 2265.\n[2] E. Schonfeld, S. Ebrahimi, S. Sinha, T. Darrel, and Z. Akata. Generalized zero- and few-shot\nlearning via aligned variational autoencoders. In CVPR, 2019.\n[3] S Badirli, Z Akata, G Mohler, C Picard, M Dundar. Fine-Grained Zero-Shot Learning with DNA as Side Information. In NeurIPS, 2021.\n[4] B. Romera-Paredes and P. H. Torr. An embarrassingly simple approach to zero-shot learning.In ICML, 2015.", "questions_to_authors": "A: Considering most of the class names from CUB were in the CLIP pre-training dataset (Table 3), can authors elaborate why there is a massive performance boost with the proposed method?", "paper_topic_and_main_contributions": "The paper proposes a simple yet effective idea to use VLMs for zero-shot image classification, in particular for fine-grained datasets.\n\nThe main contribution of the method is the idea of translating the Latin names in fine-grained datasets to common language to help VLMs' language model to better accommodate the ZSL task.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "The paper is well-written and easy to follow."}, "extra_text": {"Missing_References": "a few interesting papers to add for the ZSL part. \n\nKaressli N. et al. (2016) Gaze Embeddings for Zero-Shot Image Classification. In CVPR.\nBadirli et. al. (2023) Classifying the unknown: Insect identification with deep hierarchical Bayesian learning. Methods in Ecology and Evolution."}}, {"review_id": "49SLUeyD5y", "reviewer": "Reviewer_ik9i", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer ik9i for the valuable comments and suggestions. Reviewer ik9i appreciates our method and thinks it is \"simple yet effective\". Reviewer ik9i has two concerns and we address them below. \n\n> **Reviewer ik9i worries that the method is too simple and our work \"does not appear to introduce any new or innovative techniques or algorithms\".**\n\nPerhaps Reviewer ik9i finds the simplicity of our method to prevail the novelty of our work. The literature shows that simple methods tend to make big impacts. \n\nTo the best of our knowledge, our work is the first that shows improving the vocabulary of category names (i.e., translating scientific names to common names) significantly improves a downstream task (i.e., zero-shot species recognition). In contrast, prior work has focused on more sophisticated techniques to exploit attributes [1] and prompt templates [2] to improve zero-shot performance. Our simple method is effective (as noted by reviewers). For example, it achieves 39.8% accuracy on the Aves dataset (Table 1), significantly higher than 7.05% by [1] and 7.75% by [2]! Surprisingly, our zero-shot approach even surpasses the fully supervised learning method which achieves 20.6% by [3]! Our finding reveals the significant but overlooked insight that, *aligning a downstream task's vocabulary of categorical names to the more frequently seen (scientific/common) names by the pretrained model OpenCLIP allows leveraging this model to better serve the downstream task*. Therefore, we believe our work is novel and invaluable to our community!\n\n - [1] Radford et al., \"Learning Transferable Visual Models From Natural Language Supervision\". ICML, 2021.\n - [2] Menon et al., \"Visual classification via description from large language models\". ICLR, 2023.\n - [3] Su, et al., \"A Realistic Evaluation of Semi-Supervised Learning for Fine-Grained Classification\". CVPR, 2021 \n\n\n\n> **”Reviewer ik9i suggests \"a horizontal comparison of the prompt template, which could provide valuable insights into the efficacy and accuracy of the underlying model.\"**\n\nGreat suggestion! Below, we study the zero-shot performance using different prompt templates as suggested by [1]. Clearly, our method still performs the best across prompt templates, and results show quite small deviations. That said, our study is not affected by the choice of prompt templates. We will add them to the appendix of our camera-ready paper. Thank you for the suggestion!\n\n| Prompting Method          | iNat | Aves | Flowers102 | CUB200 |\n|-----------------| ------- | ------ |-------- | ------ |\n| `itap of a {}`              |                    |                    |                     |                 |\n|                    + s-name |                6.2 |                7.1 |                67.0 |            5.7 |\n|                    + c-name |               13.2 |               35.9 |                64.6 |           54.3 |\n|                    + f-name (**ours**) |               13.1 |               36.2 |                69.4 |           55.2 |\n| `a bad photo of the {}`     |                    |                    |                     |                 |\n|                    + s-name |                6.3 |                7.6 |                65.9 |            5.8 |\n|                    + c-name |               13.6 |               39.6 |                64.3 |           56.3 |\n|                    + f-name (**ours**)  |               13.9 |               39.5 |                68.7 |           57.1 |\n| `a photo of the large {}`   |                    |                    |                     |                 |\n|                    + s-name |                6.4 |                7.4 |                66.5 |            6.3 |\n|                    + c-name |               13.6 |               38.0 |                63.5 |           56.3 |\n|                    + f-name (**ours**)  |               13.9 |               38.0 |                68.2 |           57.6 |\n| `a photo of the small {}`   |                    |                    |                     |                 |\n|                    + s-name |                6.6 |                6.8 |                66.0 |            6.2 |\n|                    + c-name |               13.3 |               38.4 |                63.4 |           56.3 |\n|                    + f-name (**ours**)  |               13.7 |               38.3 |                68.5 |           57.1 |\n| `a {} in the wild`         |                    |                    |                     |                 |\n|                    + s-name |                6.2 |                6.6 |                67.3 |            5.9 |\n|                    + c-name |               13.6 |               39.4 |                63.7 |           58.5 |\n|                    + f-name (**ours**)  |               13.8 |               39.6 |                68.8 |           59.2 |\n| `mean acc w/ std`          |                    |                    |                     |                 |\n|                    + s-name |6.35 &plusmn; 0.15    |7.1&plusmn;0.37 |66.51&plusmn;0.60|5.97&plusmn;0.27 |\n|                    + c-name |13.46 &plusmn; 0.17 |38.25&plusmn;1.33|63.92&plusmn;0.46 |56.4&plusmn;1.35|\n|                    + f-name  (**ours**) |13.68 &plusmn; 0.29 |38.29&plusmn;1.40|68.74&plusmn;0.44|57.23&plusmn;1.44|\n\n\n - [1] Radford et al., \"Learning Transferable Visual Models From Natural Language Supervision\". ICML, 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper presents a method to facilitate the conversion of scientific names into universal names  for the purpose of enhancing the accuracy of zero sample species recognition using Vision Transformer (ViT) models. \n\nThe proposed method is relatively simple yet effective. ", "reasons_to_reject": "While the researchers implemented a method for translating text during the model inference stage, the proposed approach is relatively simplistic and does not appear to introduce any new or innovative techniques or algorithms. \n\nAdditionally, the research fails to conduct a horizontal comparison of the prompt template, which could provide valuable insights into the efficacy and accuracy of the underlying model. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a simple method to convert scientific names into universal names during model inference, which improves the accuracy of zero sample species recognition based on ViT by 2-5 times.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a neat yet effective promoting method for scientific name understanding in VLMs, which is a good example of a short paper. The reviewing process is generally towards positive and the rebuttal helps to answer some of the reviewers' concern. Please help include the additional results in any final versions.
Decision: Accept-Main