SlowGuess/NeurIPS_2023_OCR huggingface dataset:
also indicate that models with stronger hierarchical structures, such as traditional convolutional networks and Swin Transformers, tend to benefit more from pretraining on the Species-U dataset.
4.3 Experiment on multimodal large language models
Recently, there has been a surge of interest in the field of Multimodal Large Language Models (MLLM) [93, 82, 82, 45, 95]. These models leverage the ability of Large Language Models (LLMs) to perform a variety of multimodal tasks effectively. However, we discovered that there are few existing benchmarks that can assess the MLLM's ability to handle fine-grained knowledge, let alone test its performance across various levels of granularity.
We designed question-and-answer tasks using images from Species196-L, which included both multiple-choice and true or false questions, and evaluated them across 9 different Multimodal Large Language Models (MLLM) [17, 73, 25, 49, 92, 50, 96, 9, 56]. Our benchmark design is based on six different levels of taxonomy information, ranging from coarse to fine granularity, including Phylum, Class, Order, Family, Genus, and Species (scientific name). Each category consists of 1000 image-based questions. For the design and evaluation metrics of true or false questions, we followed the approach of [19]. For each image, we created two true-false questions, one of which is correct and the other is a distractor question. We evaluated the true or false questions using both accuracy and accuracy+ metrics. The accuracy+ metric is more stringent, requiring both questions for each image to be answered correctly in order to consider the answer correct. Figure 6 shows our example question settings.
In terms of experimental design, we tested all models at the 7B scale across different tasks. The results in Table 6 indicate that InstructBLIP [9] achieved the best performance on the most true of false questions, however, its ACC+ score still lower than random accuracy  $25\%$  for some categories. In five of all six categories of multiple-choice questions, Multimodal-GPT has achieved the highest accuracy, thereby leading the leaderboard (See Table 7).
True or false questions
correct answer
Does the image contain any species belonging to the Mollusca? Please respond with either "yes" or "no".
distractor option
Does the image contain any species belonging to the Magnoliophyta? Please respond with either "yes" or "no"
Granularity:Phylum
Granularity: Genus
Does the image contain any species belonging to the Popillia? Please respond with either "yes" or "no".
Does the image contain any species belonging to the Eurytoma? Please respond with either "yes" or "no".
Multiple choice questions
Figure 6: Example of our questions for multimodal large language models.
Granularity: Class
Which of the following categories does the species in the image belong to?
A: Solanaceae
B:Coccidae
C:Erebidae
D: BuprestidaeAnswer:
Granularity: Order
Which of the following categories does the species in the image belong to?
A: Caryophyllales
B:Blattodea
C: Asterales
D: LepidopteraAnswer
Table 6: Leaderboard of true or false questions of Species-L multimodal benchmark.
Models	Phylum	Class	Order	Family	Genus	Species	Avg Acc
ACC(%)	ACC+(%)	ACC(%)	ACC+(%)	ACC(%)	ACC+(%)	ACC(%)	ACC+(%)	ACC(%)	ACC+(%)	ACC(%)	ACC+(%)	ACC(%)	ACC+(%)
InstructBLIP [9]	59.0	19.2	64.9	31.0	60.5	24.1	54.3	12.2	47.7	15.5	50.25	17.0	56.1	19.8
LLaVA [56]	50.0	0.0	50	0.0	50.0	0.1	50.1	0.3	50.5	6.2	53.8	12.1	50.7	3.1
PandaGPT [73]	50.2	0.4	51.6	3.2	50.0	0.0	50.2	0.4	50.2	0.4	50.0	0.3	50.4	0.8
mPLUG-Owl [92]	52.7	11.6	52.8	12.6	48.6	7.5	50.0	9.1	47.8	11.5	45.4	10.3	49.6	10.4
Visual-GLM6B [17]	47.4	5.1	45.7	2.8	46.8	5.7	48.5	6.5	48.2	7.0	47.3	5.0	47.3	5.4
Otter [49]	48.5	0.0	49.4	0.0	48.3	0.0	49.0	0.6	43.3	0.1	40.6	1.9	46.5	0.4
Multimodal-GPT [25]	39.4	9.1	38	9.7	32.7	8.1	34.0	9.5	35.1	9.3	39.2	15.0	36.4	10.1
MiniGPT4 [96]	22.4	7.7	23.4	7.0	24.1	7.1	23.5	8.0	20.2	6.3	22.5	8.4	22.7	7.4
Blip2 [50]	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0
Table 7: Leaderboard of multiple choice questions of Species-L multimodal benchmark.
Models	Phylum	Class	Order	Family	Genus	Species	Avg Acc
Multimodal-GPT [25]	51.8	71.6	60.6	56.6	57.9	63.2	60.3
InstructBLIP [9]	47.8	58.7	56.3	57.5	45.3	39.8	50.9
PandaGPT [73]	53.0	44.1	42.6	52.8	38.6	34.6	44.3
mPLUG-Owl [92]	34.1	32.1	43.0	39.2	31.0	24.9	34.1
MiniGPT4 [96]	28.6	32.7	32.1	28.2	29.9	32.7	30.7
LLaVA [56]	38.1	34.2	17.3	33.4	22.2	23.4	28.1
Blip2 [50]	26.7	30.3	23.3	27.9	23.9	24.5	26.1
Visual-GLM6B [17]	23.0	12.2	13.9	30.5	15.7	11.6	17.8
Otter [49]	0.0	6.8	20.8	8.3	6.7	0.3	7.15
In our experiments, we observed that large language models commonly exhibit issues such as not answering prompts accurately and generating hallucinations (See Appendix Table 8). The answers generated by large models often fail to conform to the expected response on judgment and multiple choice tasks, resulting in low accuracy scores. Furthermore, we found that these models tend to display a bias towards answering "yes" on true or false tasks, which leads to accuracy and accuracy+ scores that are below random guess.
In this work, we introduced Species196, a fine-grained dataset of 196-category invasive species, consisting of over 19K finely annotated images (Species-L) and 1.2M unlabeled images of invasive species (Species-U), making it a large-scale resource for invasive species research. Compared to existing invasive species datasets, Species196 covers a wider range of species, considers multiple growth stages, and provides comprehensive taxonomic information. We also conduct comprehensive experiments bench-marking for supervised, semi-supervised and self-supervised methods, and also multi-modal models. Our experiments show that unsupervised pre-training like masked image modeling on Species-U leads to better performance compared to ImageNet pretraining. In future work, we plan to investigate additional methods for leveraging unlabeled data from Species196 and broaden the application of this approach to tackle data scarcity challenges in various real-world applications.
We gratefully acknowledge the support of MindSpore [41], CANN (Compute Architecture for Neural Networks) and Ascend AI Processor used for this research.
[84] Sanghyun Woo, Shoubhik Debnath, Ronghang Hu, Xinlei Chen, Zhuang Liu, In So Kweon, and Saining Xie. Convnext v2: Co-designing and scaling convnets with masked autoencoders, 2023.
[85] Xiaoping Wu, Chi Zhan, Yu-Kun Lai, Ming-Ming Cheng, and Jufeng Yang. Ip102: A large-scale benchmark dataset for insect pest recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
[86] Chengjun Xie, Rujing Wang, Jie Zhang, Peng Chen, Wei Dong, Rui Li, Tianjiao Chen, and Hongbo Chen. Multi-level learning features for automatic classification of field crop pests. Computers and Electronics in Agriculture, 152:233-241, 2018.
[87] Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoc V. Le. Self-training with noisy student improves imagenet classification, 2020.
[88] Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling, 2022.
[89] Di Xu, Yang Zhao, Xiang Hao, and Xin Meng. Pink-eggs dataset v1: A step toward invasive species management using deep learning embedded solutions, 2023.
[90] Qin Xu, Jiahui Wang, Bo Jiang, and Bin Luo. Fine-grained visual classification via internal ensemble learning transformer. IEEE Transactions on Multimedia, pages 1-14, 2023.
[91] Zhuolei Yang, Zheming Fan, Chenyu Niu, Peixin Li, and Hongjie Zhong. Development of invasive plant recognition system based on deep learning. Journal of Advances in Mathematics and Computer Science, 38(6):39-53, 2023.
[92] Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304.14178, 2023.
[93] Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models, 2023.
[94] Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M. Ni, and Heung-Yeung Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection, 2022.
[95] Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, et al. H_2 o: Heavy-hitter oracle for efficient generative inference of large language models. arXiv preprint arXiv:2306.14048, 2023.
[96] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023.
[97] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection, 2021.
A Examples of annotations in Species196-L dataset
Figure 7 displays several examples of annotation boxes in our Species196-L dataset. Although each image in our dataset contains only one class of label, the objective detection experimental results show that the dataset is challenging for detecting small targets, as some images feature densely populated biological instances.
Figure 7: Samples of Species196-L bounding box annotations. The top two rows show easy cases, while the bottom three rows present hard cases, including crowdedness, complex backgrounds, and tiny objects.
B Answers generated by different multi-modal models
For multiple-choice questions, some models output the category name directly instead of the options. Thus, we have appropriately relaxed our evaluation criteria. Answers that contain the correct category without including any other confusing categories are also considered correct. For true or false questions, we strictly evaluate the model's output. Table 8 shows generated answers for example questions.
C Experiment on CLIP zero-shot classification
We conduct performance evaluations on OpenCLIP [42] models of different sizes on Species196-L. All models we tested are pretrained on LAION-2B (en) [71] with  $224 \times 224$  image size. As shown in
Table 9: Comparison of different prompts and combination. The model used for comparison is VIT-B/32 and {c} denotes classnames.
Prompt number	Prompt	TOP-1 ACC.
(1)	{c}	10.7
(2)	a photo of {c}, a type of species.	9.58
(3)	a photo of a {c}, a type invasive species.	9.58
(4)	photo of {c}, a type of species.	10.19
(5)	photo of {c}, a type of invasive species.	10.23
(6)	a photo of the {c}, a type of species.	10.06
(7)	a photo of the {c}, a type of invasive species.	9.8
(1) + (5)	-	10.87
(1) + (4) + (5)	-	10.94
(1) + (4) + (5) + (6)	-	10.91
table 9, we first compare different prompts settings and finally select  $\{\{c\} .\}$  , "photo of  $\{c\}$  , a type of species.", "photo of  $\{c\}$  , a type of invasive species." as our prompts.
Recently, there is work [15] tring to leverage knowledge extracted from large pre-trained multi-models to facilitate model learning. In this work, We directly examine the influence of using different kinds of classnames. In our analysis, we employ scientific names, common names, appearance descriptions generated by LLM, and Minigpt-4 as classnames. In some cases, the AI generated descriptions is irrelevant to the required response. Table 10 presents examples of both relevant and irrelevant descriptions generated by the model.
The results (see Table 11) indicate that in our dataset, using common names as classnames yields the best zero-shot classification performance. For the generated descriptions used as classnames, LLM outperforms the scientific names as classnames in most models, while MiniGPT-4 has lower inference accuracy due to the high proportion of irrelevant descriptions. Compared to other fine-grained datasets, the significantly lower accuracy of Species196-L suggests that it poses a new challenge in the field of zero-shot fine-grained classification.
Table 11: Experiment results of zero-shot classification on Species196-L as well as other fine-grained datasets.
Model	Species196-L	Cars	FGVC Aircraft	Flowers102
Scientific name	Common name	Description (LLM)	Description (Minigpt-4)
ViT-B/32	10.94	16.71	10.25	5.76	86.05	24.551	71.62
ViT-B/16	10.40	16.90	10.81	6.59	88.50	26.97	71.34
ViT-L/14	12.07	18.43	12.69	6.29	92.64	36.75	75.83
ViT-H/14	15.00	22.57	15.95	6.23	93.36	42.60	80.13
D Hosting and maintenance plan
Both the Species196-L and Species196-U datasets are publicly available at https:// species-dataset.github.io/. This website is hosted on Github Pages, a widely-used website hosting service. The website contains introductions, experiment results, terms of use, and links to download the datasets, as well as usage guides. We maintain the data using Google Drive and Baidu Cloud, where we store the original URLs to download the images, ensuring that the dataset will be available for an extended period. Additionally, we will provide instructions on how to download and organize the data with code. For further maintenance, we will continue refining our dataset, such as correcting incorrect labels and annotations in Species-L, and updating a larger and more comprehensive version of Species-U.
The Species196-L dataset is provided to You under the terms of the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International Public License ("CC BY-NC-SA 4.0"), with the
Table 10: Example of relevant and irrelevant descriptions generated by LLM and Minigpt-4. For LLM, the input is only scientific name. For Minigpt-4, we select one image for each category and design five prompts for inquiry.
Prompts	Sample relevant answers	Sample irrelevant answers
Descriptions generated by LLM
1. Briefly describe the appearance characteristics of the species {c}, excluding size descriptions.	A pink, oval insect with a wax-covered body and long filaments (Phenacoccus solenopsis)	The adults are variable in size, coloration, colour pattern and facies (Rhabdoscelus obscurus)
Descriptions generated by Minigpt-4
1. Describe the organism's physical features, such as its size, shape, and color.	It has two legs. The top part has a brown, rough surface, and the bottom part has a dark brown, smooth surface. The middle part has a dark brown surface. (Spodoptera littoralis)	I'll look at this image and answer your questions. (Hoplocerambyx spinicornis)
2. Describe the texture and surface features of the organism in the picture.	The beetle's skin is smooth, its body is yellow, and its antennas are brown with orange tips. (Scolytus scolytus)	Do you think that the organism's texture and surface features would cause it to survive in the habitat shown in the picture? (Avena barbata)
3. Describe any physical adaptations of the organism, such as specialized structures for defense, locomotion, or feeding.	The insect has one set of eyes and antennae to look for food and also it can fly and run away from its predator. (Gonipterus scutellatus)	An organism's life cycle. A complete cycle can include any type of reproduction, but is limited to the life span of one organism. (Aleurodicus dispersus)
4. Describe any unique patterns or markings on the organism's body.	The organism is small, but has two eyes. The eyes are very sharp. The antennae are long and can feel things very well (Hoplocampa testudinea)	Answer the questions from the student's perspective. (Anastrepha)
5. Describe the organism's body structure, including any visible organs or systems.	It has four wings, a head with eyes, a thorax with legs, and a stomach for digesting food. (Dasineura mali)	How can humans benefit from knowing about the organism? (Hylobius pales)
additional terms included herein. This dataset is used only for non-commercial purposes such as academic research, teaching, or scientific publications. We prohibits You from using the dataset or any derivative works for commercial purposes, such as selling data or using it for commercial gain. The Species196-U dataset is distribute under the Creative Common CC-BY 4.0 license, which poses no particular restriction. The images are under their copyright.