Species196: A One-Million Semi-supervised Dataset for Fine-grained Species Recognition
Authors: Wei He
Kai Han
Ying Nie
Chengcheng Wang
Yunhe Wang
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: Invasion biometrics
Fine-grained dataset
Pre-training

Abstract: The development of foundation vision models has pushed the general visual recognition to a high level, but cannot well address the fine-grained recognition in specialized domain such as invasive species classification. Identifying and managing invasive species has strong social and ecological value. Currently, most invasive species datasets are limited in scale and cover a narrow range of species, which restricts the development of deep-learning based invasion biometrics systems. To fill the gap of this area, we introduced Species196, a large-scale semi-supervised dataset of 196-category invasive species. It collects over 19K images with expert-level accurate annotations (Species196-L), and 1.2M unlabeled images of invasive species (Species196-U). The dataset provides four experimental settings for benchmarking the existing models and algorithms, namely, supervised learning, semi-supervised learning and self-supervised pretraining. To facilitate future research on these four learning paradigms, we conduct an empirical study of the representative methods on the introduced dataset. The dataset will be made publicly available at https://species-dataset.github.io/.

Species196: A One-Million Semi-supervised Dataset for Fine-grained Species Recognition
Wei He, Kai Han", Ying Nie, Chengcheng Wang, Yunhe Wang' Huawei Noah's 's Ark Lab
thewei142,ka1 han yunhe. wang) Chuawei.com
Abstract
The development of foundation vision models has pushed the general visual recog- nition a high level, but cannot well address the fine- the-graiied recognition special- ized domain such as invasive species classification. Identifying and managing inva- sive species has strong social and ecological value. Currently, most invasive species datasets are limited in scale and cover narrow range of species, which restricts the development of deep-learning based invasion biometries systems. To fill the gap of this area, we introduced Species 196, large-scale semi-supervised dataset of 196-category invasive species. It collects over 19K images with expert-level ac- curate annotations (Species796-L), and 1.2M unlabeled images of invasive species (Species|96-U) The dataset provides four experimental settings for benchmarking the existing models and algorithms, namely, supervised learning, semi-supervised learning, self-supervised pretraining and zero-shot inference ability of large multi- modal models. To facilitate future research these four learning paradigms, we conduct an empirical study of the representative methods on the introduced dataset. The dataset is publicly available at htthtps:// https:://spccies-dee dataset. .i thub.io/
1 Introduction
Invasion biometrics play critical role in the identification and management of invasive species, which are non-native organisms capable of causing detrimental impacts on the environment, economy, and human health [18]. Traditionally, invasive species recognition systems have relied trained experts who analyze an animal or plant's physical characteristics, or use DNA testing [I]] However, these conventional methods require specialized equipment, expert knowledge, and are often time- consuming and costly, while also posing potential risks to creature well-being (22). Recent methods [53 [1] utilizing computer vision techniques and deep learning algorithms, provide cost-effective and efficient alternative solution for invasive species identification.
As computer vision-based methods continue to gain traction, the need for high-quality biometrics data becomes increasingly crucial for effective invasive species recognition. In contrast to traditional visual tasks such as image classification or object detection, which benefit from large-scale datasets like ImageNet [12), Open Images Dataset (88), COCO(55), and Objects365 [72), acquiring diverse, high-quality datasets for training and evaluating invasive species recognition algorithms remains challenge.
In response to this challenge, we introduce Species 196-1, a new fine-graing hierarchical dataset focusing on regional invasive species. The "L" donates that the dataset fine labeled. Species|96 collects 19236 images for 196 invasive species listed in the Catalogue of Quarantine Pests for Import Plants to China 65) In addition, We also provide bounding box annotation and detailed
Corresponding Author
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

multi-graine taxonomy information for each image, which assists users in building more effective invasive species biometrics systems.
Self-supervised and semi-supervised learning are particularly useful when labeled data costly difficult collect. To further enhance our Species /96 dataset, utilize the publicty available LAION-5B dataset [71] to create a new, large-scale 1.2M-image subset based on Species
invasive species, named Species I I96-U. The "U" signifies that the dataset is unlabeled. In Species 196- U, each image is retrieved from the LAION-5B dataset [7]] using a pre-trained OpenCLIP VIT-/114 [57] image encoder, based on the original finely annotated Species (96LL. This dataset expands the amount of data available for invasive species research and provides a valuable domain-specific resource for unsupervised. supervupee, or modtl learning.
Our main contributions are summarized as follows:
To the best of our knowledge, we have constructed the largest-scale invasive species biometrics dataset (Species/ 96-L., covering a wide range of 196 species. This dataset not only encompasses different life cycles but also provides detailed taxonomic information for each species, ranging from Kingdom and Phylum to Genus and Species.
We present large-scale, domain-specific unlabeled dataset, Species/96-U which is highly related to images from Species containing 1.2 million image-text pairs. This dataset serves as an ideal testbed for evaluating various pretraining and multi-modal methods.
We perform extensive experiments on our proposed datasets using a variety of methods, including state-of-the-an CNN and transformer networks, specifically designed fine-grained visual recogni- tion techniques, self-supervised and semi-supervised approaches, and also CLIP and multi-modal large models' zero-shot inference performance Species-L.. These comprehensive experiments establish a benchmark for Species196.
Related work
2.1 Invasion biometrics
Invasive species cause ecological change, harm biodiversity [ and present significant threats to global agriculture (67] and economic welfare [46]. Invasive biometrics plays a vital role in identifying and managing these species. Conventional invasive species identification methods (e.g. physical analysis and DNA testing) are costly, time-consuming, and heavily relied expert knowledge and experience 22
In recent years, there has been growing interest in using computer vision (CV) for invasive species recognition, as it offers more efficient, cost-effective, and scalable alternatives traditional methods. These methods can be divided into two types handcrafted and deep feature based. Handcrafted based methods adopting feature extractors like SIFT 61 and [O] Li el al. 53 propesed HOG and SVM based methods for identifying invasive Asian Hornets 13.. which extracting image features using the HOG algorithm, and using the SVM algorithm for target detection. More recently, deep learning techniques have shown its potential in this field. Chowdhury al. 77] suggested approach for identifying aquatic invasive species by utilizing an autoencoder feature extractor, coupled with a classifier trained to distinguish between invasive and non-invasive species. Deep convolutional neural netwok also used by Ashqar et al. 22] to identifying images of invasive hydrangea, and by Huang et al. [40] to develop fast and accurate detection technology identify invasive weeds.
Although deep learning- based methods are becoming increasingly popular, they heavily rely on high- quality annotated data compared to handcrafted-basee based methods, especially in practical applications where high-quality labeled data is often the performance bottleneck for invasion biometrics or other pest recognition applications.
2.2
Related datasets
In this section, provide overview comparing our proposed Species196 dataset with other related species recognition and fine-grained datasets. The comparison overview is shown in Table !
Pink-Fggs (89 comprises images accompanied by bounding box annotations of the invasive species Pomacea canaliculata, Cional17 (00 is semantic segmentation dataset providing pixel-level annota- tions concerning invasive species in marine environments. However, these two invasive biometrics

---

Cheridtooin Ammitnaa
Openans serchari
Eufacanlut Ripentca
Cyslia
Hyppaanzia plance
packandi
Aegile'' vitosii
Cororlintan ragel
Lelium remolentum
Alcameiica diggonos
Flaveriaa Cndertitiddndii
Copponinnes brevss
Lepodiguera
Herripperaa
Ponles
Xanthiim
Coptotentee App
Consurtes soperte
Highodel
Avieniios
Isseeta
Prinus oectts
Species 196
Aurmi
Aprales
Colorpier
tpp
Caicclll luticilia
Mollases
Acanboxiis carimittts
Cucagnifaliuu
Distori
Solmaies
Myloparims pandalina
Seltrita cErsimnnee
Cyperaes
Hystenapera
Carporyyo inooplete
Styfommaathghuaa
slenilis
Sirex noctL.i
Avena ludoviciana
Avena berbera
Bendbophagrs roddi
Achetins fulicn
Englandina roses
desperts Copaca Bortand Cermella vingata
Figure 1: Taxonomy of portion of the Species196-L dataset's taxonomy hierarchical system.
datasets possess a rather limited number of samples and focus only on specific types of organisms. In comparison, Species 196-L dataset offers nearly 20,000 samples, covering 196 diverse invasive species, making it particularly valuable in various real-world applications such as customs and border crossings quarantine.
There are also some datasets that share the same super-categories, such as weeds and insects, with Species/ Species/96-L... L. Some of them [86]
24
66 have fewer sample numbers and species compared to
Species 9.-L, while other datasets 66
36 are not publicly available so far. IP102 (85 and
CWD30 43 are two recent representative datasets in the field of insect and crop-weed. IP 102 covers 102 species of common crop insect pests with over 75, 000 images, captures various growth stages of insect pest species. CWD30 comprises over 219,770 high-resolution images of 20 weed species and 10 crop species, encompassing various growth stages, multiple viewing angles, and build hierarchical taxonomic system foe these weeds and crops. Following the best practices of IP102 and CWD30, our Species196. 996- dataset covers various life stages of insects and establishes a comprehensive taxonomy system, from domain, kingdom, down to order, family, genus, and species, aiding in the creation of a precise and robust recognition system.
As challenging benchmark dataset for fine-grained image recognition, we also compare Species196. to other popular datasets in this field. Oxford Flowers (64), CUB200 35), FGVC Aircraft (62) and Stanford Cars 47 are popular fine-grained image classification datasets with different categories of flowers, birds and cars, respectively. However, these datasets not involve the challenge of distinguishing species at different life stages and do not provide multi-grained taxonomy information.

---

Table Comparison of related species recognition and fine- ine-grained datasets
Multiple life-cycle
Dataset
Year
Meta-classes
Categories
Samples
Taxonomy
Pink-Eges 89
2023
Mollusca
3
1261
N
Cional 20
2017
Mollusca, Chordata
5
N
3P102[3 85
2019
Insects
102
75,222
Y
N
Pest ID
60
2016
Insects
12
5,136
N
N
Xie et al. 86
2018
Insects
4,500
N
Alfarisy et
2018
Insects
4,511
N
Deep Weeds [66
2019
Weeds
17,509
N
Plant Seedling 24
2017
Weeds
5,539
N
CNU 36
2019
Weeds
21
208,477
N
N
CWD30 43
2023
Crops, Weeds
30
219.778
Y
Y
Oxford Flowers
64
2008
Plants
102
8,189
N
CUB200 35
2011
Birds
200
11,788
N
Stanford Cars
77
2013
Cars
196
19.184
FGVC Aircraft
62
2013
Airerafts
100
10,000
iNat2017
37
2017
Plants. Animals
5,089
859,000
Y
Y
iNar2021
38
2021
Plants, Animals
10,000
3,286,843
Y
19 19,256
Species196
2023
Mollusca, Weeds, Insects
196
1,200,000 (unlabeled)
The iNat2017 dataset 37 introduced large-scale, fine- ame-grrined image classification and detection dataset with 859K images from 5,089 species, emphasizing the importance few-shot learning. The iNat2021 dataset (38] expanded this to 2.7M images from 10K species and highlighted the advantages of self-supervised learning methods for fine-grained classification. In contrast to the iNat2017 and iNar2021 datasets, which are derived from user-contributed observations community 44, the scale and collection method of our Species 196 dataset align more closely with real-world applications. Moreover, Species 196 provides challenging testbed for the research community to explore how to improve model's performance through limited meticulously labeled data and large-scale unlabeled data.
the iNaturalist
3 Dataset
3.1 Taxonomic system establishment
Figure [ illustrates portion of the hierarchical taxonomy system for Species/96-L. All species included in our collection are sourced from the Catalogue of Quarantine Pests for Import Plants to China (65) With the aim help construct an effective and efficient computer vision-based invasion biometrics system, we selected insects, weeds, and mollusks that are visually observable and also easily collectable using mobile devices.
We have provided a taxonomy system that includes comprehensive hierarchical classification infor- mation within the dataset. This information spans from the domain, kingdom, and phylum levels down to the order, family, genus, and species of each invasive species. We hope that the inclusion of biological taxonomy data, incorporating prior knowledge in this field, will enable users of the Species 196 dataset to explore and construct more efficient, effective, and robust invasive species identification systems.
3.2 Image collection, filtering, split, and annotation of Species196-1
this work, we utilize public available information on the internet as the source for collecting images of invasive organisms. In addition to directly using search engines, we also place a strong emphasis on exploring various global biological image repositories for image collection. Examples of the image repositories accessed include the iNaturalist community [44), the Global Invasive Species Database (GISD) (23], BugGuide [3], Biocontrole 4. and more. These diverse and trustworthy sources provided comprehensive and rich of images for the Species
Our team of five members utilized taxonomy information to search for images using both common names and scientific names. We collected images of insects at various stages of their life cycle,

---

Genus: Cydia
Family: Helicidae
Cydia
Cydia pomonella
Cydia prunivara
Cydla pyrivora
Copasa hartensis
Theba pisana
Genus: Rhynchophorus
Genus: Rhynchophorus
Rhynchophorus
Rhynchophorus
Rhynchophorus
Rhynchophorus
ferrugineus
palmarum
Aegilops
Acgilops squarrosa
Figure Display images of species belonging to the same genus or family.
such as eggs, larva, pupa, and adult (see Figure 3), because each stage can cause negative impacts on ecosystems and the economy. We also removed images that contained more than one invasive species category. Our dataset is muuli--rainee dataset with class granularity based largely on both species and genus, and when processing categories primarily by genus, have balanced the number of categorics within each genus as much as possible to help ensure a relatively even distribution. Subsequently, removed images with size lower than 128x 128 and those containing private information such as human faces. We collected 19, 19,236 images for 196 invasive species. Following the approach of Stanford Cars [7 and CUB200 B5), evenly divided train and test sets at 1:1 ratio, resulting in a train size of 9,676 images and a test size of 9,580 images.
In real- world pest control, precise identification and location of pests vital. This task, often com- plicated by cluttered backgrounds and multiple pests one image, aids effective pest management. After collection and filtering, labeled 19,236 images with 24,476 bounding boxes in the COCO format [54) for Species Species196-L, 196L, which took five individuals about six months.
Eggs
Larva
Pupa
Adult
Opogona sacchari
Cacoecimorpha
pronubana
Mayetiola destructor
Rhynchophorus
palmarum
Figure 3: Invasive insect life cycles: egg, larva, pupa and adult.
3.3 Challenges of fine granularity and data imbalance
Species 196iee share common challenges like other fine-grained datasets such as high similarity between different classes and low variability within each class. Figure shows example of classes
5

---

belonging to the same genus, which are difficult to distinguish. Data imbalance is another challenge. As illustrated in Figure although our dataset contains species from 21 different orders, the majority of them are concentrated in few orders, such as Coleoptera, Lepidoptera, and Hemiptera.
3.4 Data collection of Species196-U
Prior research has demonstrated that domain domain-adaptive addatiive pretraining has the potential to improve performance in both the natural language [38] and computer vision [27] fields. In recent years, the introduction of the CLIP (Contrastive Language-Ine Image Pre-training) method [68] and the emergence of web-harvested image-text data like LAIONSB [7]] have provided us with the opportunity to build
large-scale -specfi data at minimal cost.
In this work, we utilize the clip-retrieval 3 to retrieve the large-scale Species-U dataset from LAION-SB. We first evaluated various retrieval methods and discovered that image-to-image retrieval surpassed other text-to-image retrieval approaches in terms of relevance. These approaches included retrieval based on common names, scientific names, and artificial descriptions. Specifically, randomly sampled three images for each category and retrieved 8,000 unlabeled images per class. For insects with different life cycles, we additionally sampled eggs, larvae, and pupae once for each species. After removing duplicates, we obtained final dataset consisting of approximately 1.5 million images.
For comparison with popular experiments using ImageNet-1K IK, we chose 1.2 million image-text pairs from LAION-5B, number smaller than that in ImageNet-IK.
3.5 Retrieved Example Images from the Species196-U Dataset
We use image-retrieve for creating Species U. For each category, we randomly sampled three images and retrieved 8,000 unlabeled images per class from LAIONSB. As shown in Figure 4 the 5,000,, image sorted by descending similarity scores, the retrieved image remains highly relevant the original image.
Specles196
somple IMGG
100
clip-retrieval
Figure 4: Clip-retrieval process of Species16-U from Species196-L. Displaying similarity scores in
descending order, we show items No. 100, 500, 1000, and 5000.

---

Experiment
To benchmark Species/96-1, we assessed methods with CNN, Transformer, or hybrid backbones of varying scales, including fine-grained models. We also experimented with unsupervised and semi-supervised methods on the Species196- I9- dataset. Moreover, we examined zero-shot inference capabilities by testing the performance of CLIP and several large multi-modal models.
4.1 Experiment on supervised learning methods
We first tested mainstream visual backbones and several recent fine- tine-grained classification methods. Keeping the potential deployment of invasion biometrics systems on the edge side mind, we specifically compared lightweight models. Detailed comparison results are presented in Table As for experimental details, we utilized timm as the codebase to evaluate different models. For networks with the same architecture and similar complexity (e.g., CNN-based, Transformer-basee and Hybrid structures), all models were trained for 300 epochs with the same input resolution of x 224 for fair comparison. For fine-grained classification network methods, we selected MetaFormer-2 14], TransFG [3], and IELT [90], which have demonstrated promising performance other fine-graine grained classification datasets CUB200 [35].
+ performs best when trained from scratch. Fur- thermore, on the Species /96-I. dataset, pre-training and transfer learning significantly outperformed training from scratch models of all sizes, particularly transformer-basee and hybrid networks such as TNT-S (30] and CMT-S [26]. Among the top-performing models in small, medium, and large-scale comparisons are VIT-XS [63] MaxViT-T (79), and MVīTv2-B [52]. MetaFormer-2 [14 achieved impressive accuracies of 87.69% withand 88.69% on the Species 19-- dataset, respec- tively. This performance was achieved through pre-training with ImageNet- I and Imagenet-22K.
Our experimental results indicate that ResNet50
MetaFormer-2 [4] achieved impressive performance our Species 19-L dataset, reaching state-of- the-art (SOTA) accuracy of 88.66 69% with a resolution of 384x384 input size. It also achieved top-1 accuracy of 87.69% in the 224x224 resolution.
We also conducted experiments on popular object detection networks, ranging from CNN-based methods such as Faster-RCNN 770 and YOLOX-L [1], to DETR likee Deformable DETR [97 and DINO 194). Using Imagenet- Ik pre-trained weights, experiment results show that DINO with Swin-Base backbone achieved the best accuracy, while Deformable DETR with ResNet50 backbone attained a balance between accuracy and parameters. The comparison detail is shown in Table 3
Coleoptera
Lepidoptera
Hemiptera
Diptera
Hymenoptera
Blattodea-
Thysanoptera
Stylommetophore
Zygophyllales
Order
Solanales
Poales
Oxalidales
Malpighiales
Lamlales
Fabales
Dipsacales
Cyperales
Caryophyllales
Brassicales
Asterales
Apiales
1000
2000
3000
4000
5000
6000
Instance Number
Figure 5: Imbalance distribution of Species Sociies966. 196-L based with order-level granularity.

---

Table 2: Comparison of different modern backbones and fine-grained methods. f and donates using trained from scratch.
Imagenet-1K, Imagene22K pretrained weight, separately. The
Model
Resolution
Params.
FLOPs.
Top
ACC.
Top ACC.
FlarACRO
MobileViT-XS ViT-XS 63
224
2.3 M
0.7 G
64.11 78.55*
83.51 / 991922
53.52 / 69.01
GhostNet 1.0
29
224
5.2 M
0.1
62.75 76.02
82.58 / 90.771
51.30/ / 64.93"
LfficiietNeetB00
75
224
5.3
0.4
62.88 78.26'
81.66 99.601
53.13 66.91
MobileNetV3 Large
224
5.4
0.2
62.75 77.83
81.46 90.771
49.99 66.50"
RegNetY-4GF
69
224
20.6 M
4.0G
43.01 82.25
69.02 93.771
28.99 71.244
Deit-S 78
224
22 M
4.6
36.89 / 77.21
56.79 / /91522
29.35 65.25"
TNT-S 30
224
23.8 M
5.2G
38.66 80.67
59.14/93.171
30,67 69.34
CMT-S
224
25.1 M
4G
40.86 81.12
60.10/93.321
33.25 70.40
Resnet50
33
224
25.6 M
4.1G
64.32 78.11
81.70/91.11
53.31/ 67.29
Swin-T
224
28 M
4.5G
46.88 81.66*
68.57 /93.52t
37.30 / 77.20"
Convnext-T
59
224
29 M
4.5G
46.36/ 78.94"
68.59/92.441
37.16 70.43
MaxViT-T
79
224
31 M
5.6G
52.19 83.35
72.12/94.161
42,40/ 62.56
MViTv2-B
52
224
52 M
10.2
46.22 83.79
66.21 / 194.811
35.83 72.94
Resmet200-D
224
65 M
26G
51.35
82.11
73.07 99.766
37.70 70.61"
VIT-B/32 [6
224
86 M
8.6G
32.59/ 74.68
53.76/ 89.761
25.20 63.38"
Swin-B 1X
224
88 M
15.4
48.72 82.88
69.71 / 94.30t
39.28 72.04"
Pyramid VIG-B
28
224
82.6 M
16.8
61,59 / 82.82
82.93 / 91.961
34.27 72.40
MetaFormer-2
224
81 M
87.691
MetaFormer-2
3842
81 M
88.69F
TransFG 31
224
85.8 M
84.42
IELT 90
448
93.5 M
81.92F
4.2
Experiment semi-supervised and self-supervised learning methods
4.2.1
Semi-supervised learning
The Noisy Student approach leverages teacher model to generate pseudo labels for unlabeled data, training larger student model with both labeled and unlabeled sets. We used similar semi- supervised method on our smaller-scale Species/96. dataset compared to YFCC100M [76] and incorporated data augmentation and dropout for noise injection
JFT (74] Following this approach, in training. using a model of equal or smaller size for labeling unlabeled data, then training larger model. Tablee[shooe that using Noisy Student training our 1.2 million unlabeled data enhances performance when the student model is larger. However, if the student network is identical to the teacher's, accuracy may decline.
4.2.2 Unsupervised learning
In recent years, with methods like MAE 32 SimMIM 88 and LocalMIM 81 masked image modeling unsupervised learning has received increasing attention. ConvNeXt V2 and SparK [77) further successfully bring MIM into CNN networks. Inspired by some success works in the NLP domain (e.g. (27). (5T), we constructed the Species- U dataset for the Invasion Biometrics task and conduct an empirical study on different MIM methods(see Table We found that even with less data, pretraining on our Species-U dataset can surpass Imagenet- pretrained models on both CNN-based and Transformer-basee networks with the same or fewer pretraining epochs. The results
Table 3: Comparison of average precision performance of object detection methods. IoU threshold range of 0.5 to 0.95.
Methods
Backbone
#Params.
AP
AP
AP M
AP
Faster- Faster-RCNN RCNN
Resnet50
M
44.7
5.2
24.9
46.0
YOLOX-L 21
CSPNet
80
54 M
50.3
9.9
31.0
51.3
Deformable DETR
Resnet50
33
40 M
56.9
10.8
36.9
S8.3
DINO 94
Resnet50
M
57.7
9.1
40.0
59.2
DINO 94
Swin- Base
58
109 M
67.7
19.1
46.5
69.6

---

Table 4: Comparison using student model with the same size or with a larger size. Student Acc. represents the top-1 accuracy of the student network at the end of the last iteration.
Teacher
Teacher Acc.
Student
Student Ace.
ResNet18
71.1
ResNet18
70.6
ResNet18
71.1
ResNer34
73.4
ResNet34
72.3
ResNet34
72.0
Table 5: Comparison of different model architecture with different MIM methods. PT/FT donates pre-train and fine-tune stage.
Backbone
PT method
PT data
PT epoch
FT epoch
Top ACC.
Top 5 ACC.
Vision Transformer Backbone
SimMIM
88
ImageNet-1K IK
800
100
80.9
94.9
VIT-B
SimMIM
88
Species-U
800
100
81.0
94.6
SimMIM
88
ImageNet-IK
800
100
80.5
94.5
Swin-B
SimMIM
88
Species-U
800
100
81.6
95.0
Convolutional Backhone
SparK
ImageNet-IK
1600
300
Resnet50
SparK
77
Species-U U
800
300
73.2
88.9
FCMAE
84
ImagoNet-IK
1600
300
77.1
92.6
ConvNeXt V2-T
FCMAE
Species-U
1600
300
79.3
93.1
FCMAE
ImageNet-IK
1600
300
79.9
93.1
ConvNeXt V2-B
FCMAE
Species-U
300
81.2
94.0
also indicate that models with stronger hierarchical structures, such traditional convolutional networks and Swin Transformers, tend benefit more from pretraining on the Species-U dataset.
4.3 Experiment multimodal large language models
Recently, there has been a surge of interest in the field of Multimodal Large Language Models (MLLM) 993 82 82 45 95). These models leverage the ability Large Language Models (LLMs) to perform variety of multimodal tasks effectivelly. However, we discovered that there are few existing benchmarks that can assess the MLLM's ability to handle fine-grained knowledge, let alone test its performance across various levels of granularity.
We designed question-an caassion-and-aaa and-answer tasks using images from Species/96-L, which included both multiple-choice and true or false questions, and evaluated them across 9 different Multimodal Large Language Models (MLLM) I7 73
56]. Our benchmark design is based on six different levels of taxonomy information. ranging from coarse to fine granularity, including Phylum, Class, Order, Family, Genus, and Species (scientific name). Each category consists of 1000 image-based questions. For the design and evaluation metrics of true or false questions, we followed the approach [19]. For each image, we created two true-false questions, one of which is correct and the other is a distractor question. We evaluated the true or false questions using both accuracy and accuracy+ metrics. The accuracy+ metric is more stringent, requiring both questions for each image to be answered correctly in order to consider the answer correct. Figure 6 shows our example question settings.
92 50 96
In terms of experimental design, we tested all models at the 7B scale across different tasks. The results in Table(6jindicate that InstructBLIIP 9 achieved best performance on the most true of false questions, however, its ACC+ score still lower tha random accuracy 25% for some categories. In five of six categories of multiple-choice questions, MuTt has achieved the highest accuracy, thereby leading the leaderboard (See Table 77)

---

questions
GransinyyPPyoe
Growdadyyy
coneet knowar
boes the lmage contašn any species balonging to the mullusce? Flaase respond with either "yess"
foes the Image contaon any species belonging to the ropilllir Please respond with aither "Yyea""
aoes the Leage contaši any species belumging Paggollaphty Plause rerpond "*yes"
Boes the Srage, contain any species bellnnging, Eurytomet Plaase ""yes"
choice
Granstnnty: Clese
Grevutaity Order
tnllwwngg prlegrriee
Which the fnlltnn oxlegriie doss the
spaciies image
species image belong
Caspoofylides
Bi Cocaiton
B: alidde
C. Asterulss
Figure 6: Example of our questions for multimodal large language models.
Table 6: Leaderboard of true false questions of Species-L multimodal benchmark.
Plylen
Species
Mides
ACC())
(CC
(ACO/)
(CCI(S)
(4008)
(CC)())
460/9:
ACC (13)
.6C..9
ACC
.ACC15)
ACC
.ACC19)
ACC
64.9
31.8
04.5
24.1
17.41
96.1
19.8
Pastar
51.6
3.3
500
080
90.2
10.0
50.4
0.3
67.7
11.6
12.8
17.6
08.0
7.1
90.0
911
47.8
11.5
45.4
10.3
49.6
16.4
Masal GLMMs
[+
47.4
5.1
63.7
2.5
46.8
53
48.5
6.5
48.2
3/0
47.3
5.0
47.3
5.4
0.5
32.7
10.1
MunieP44 PM
22.4
7.7
23.4
7.9
24.1
7.3
23.5
5.0
20.2
22.5
9.4
22.7
7.4
[0pp
08
060
0/0
0.0
04
00
0.
0.0
0.1
0.0
0.0
0.0
0.0
0.0
Table Leaderboard of multiple choice questions Species-L multimodal benchmark.
Models
Phylum
Class
Order
Family
Genus
Species
Avg Ace
Multimodal-GPT
25
51.8
71.6
60.6
56.6
57.9
63.2
60.3
InstructBIII JP
47.8
58.7
56.3
57.5
45.3
39.8
50.9
PandaGPT
53.0
44.1
42.6
$2.8
38.6
44.3
mPLUG-Owl
34.1
32.1
43.0
39.2
31.0
24.9
34.1
MiniGPT4 996
28.6
32.7
32.1
28.2
29.9
32.7
30.7
LLaVA 56
38.1
34.2
17.3
33.4
22.2
23.4
28.1
Blip2 [50
26.7
30.3
23.3
27.9
23.9
24.5
26.1
Visual GLM6B
23.0
12.2
13.9
30.5
15.7
17.8
49
0.0
6.8
20.8
8.3
6.7
7.15
In our experiments, we observed that large language models commonly exhibit issues such as not answering prompts accurately and generating hallucinations (See Appendix Table 8). The answers generated by large models often fail to conform to the expected response on judgment and multiple choice tasks, resulting in low accuracy scores. Furthermore, we found that these models tend to display bias towards answering "yes" on true or false tasks, which leads to accuracy and accuracy+ scores that are below random guess.
Conclusion
In this work, introduced Species196, fine- fine-grained dataset of 196-category invasive species, consisting of over 19K finely annotated images (Species-L) and 1.2M unlabeled images of invasive species (Species-U), making it large-scale resource for invasive species research. Compared to existing invasive species datasets, Species 196 covers wider range of species, considers multiple growth stages, and provides comprehensive taxonomic information. We also conduct comprehensive experiments bench-marking for supervised, semi-supervised and self-supervised methods, and also multi-modal models. Our experiment shows that unsupervised pre-training like masked image modeling on Species-U leads to better performance compared to ImageNet pretraining. In future work, we plan investigate additional methods for leveraging unlabeled data from Species 96 and broaden the application of this approach to tackle data scarcity challenges in various real-worls applications.
10

---

Acknowledgement
We gratefully acknowledge the support of MindSpore [41], CANN (Compute Architecture for Neural Networks) and Ascend Al Processor used for this research.
References
[1] Ahmad Arib Alfarisy, Quan Chen, and Minyi Guo. Deep learning based classification for paddy pests & diseases recognition, In Proceedings of 2018 international conference mathematics and artificial intelligence, pages 21-25, 2018.
[2] Belal AM Ashqar and Samy S Abu-Naser. Identifying images of invasive hydrangea using pre-trained deep convolutional neural networks. International Journal of Academic Engineering Research (IJAER), 3(3):28-36, 2019.
[3] Romain Beaumont, Clip retrieval: Easily compute clip embeddings and build a clip retrieval system with them. https://github. http:shhww..coomoomm55////////// retrivea 2022.
[4] Biocontrole, https::/bbtccontrool.
Bugguide.
[6] Nicolas Carion, Francisco Massa, Gabriel Synnaeve. Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers, 2020.
[7] Shaif Chowdhury and Greg Hamerly. Recognition of aquatic invasive species larvae using autoencoder- based feature averaging. In Advances in Visual Computing: 17th International Symposium, ISVC 2022, San Diego, CA. USA. October 3-5, 2022, Proceedings, Part I. pages 145-161. Springer, 2022.
[8] Ross N. Cuthbert and Elizabeta Briski. Temperature, not salinity, drives impact of an emerging invasive species. Science of The Total Environment, 180:146640, August 2021.
19] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang. Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purposs vision-language models with instruction tuning, 2023.
[10] Navneet Dalal and Bill Triggs. 2005 ieee computer society conference on computer vision and pattern recognition (cvpr'05) 05). Histograms of oriented gradients for human detection, 2005:886-893 2005.
[11] John Darling and Michael Blum. Dna-based methods for monitoring invasive species: a review and prospectus. Biological Invasions, 9:751-765, 2007.
[12] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. Ieee, 2009.
[13] Use Case Asian Hornet Detection. Asian hornet detection dataset. https://universe row com/use- case- www.hhotet--etetttoaassaaaaeeeee hwt.ssiiannnnoneeeeeettttiiooooa a6ael leb 2022. visited on 2023-05-31.
[14] Qishuai Diao, Yi Jiang, Bin Wen, Jia Sun, and Zehuan Yuan. Metaformer: A unified meta framework for fine-grained recognition, 2022.
[15] Ning Ding, Yehui Tang, Zhongqian Fu, Chao Xu, Kai Han, and Yunhe Wang. Gpt4image: Can large pre-trained models help vision models on perception tasks?, 2023.
[16] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby, An image is worth 16x16 words: Transformers for image recognition at scale, 2021.
[17] Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang, Glm: General language model pretraining with autoregressive blank infilling, arXiv preprint arXiv:2103.103660 2021.
[18] Joan G Ehrenfeld. Ecosystem consequences of biological invasions. Annual review of ecology, evolution, and systematics, 10, 2010.
[19] Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXhv:2306.13:: 13394.
11

---

[20] Angus Galloway, Graham Taylor, Aaron Ramsay, and Medhat Moussa. The cional7 dataset for semantic segmentation of invasive species in marine aquaculture environment, 2017.
[21] Zheng Ge, Songtao Liu, Feng Wang, Zeming Li, and Jian Sun. Yolox: Exceeding yolo series in 2021, 2021.
[22] Jhony- Hery-eeri Giraldo-Zulugg,, Augusto Salazar, and Juan M. Daza. Semi-supervised recognition of the diploglossus millepunctatus lizard species using artificial vision algorithms, 2016.
[23] Global invasive species database. http://wwe wucngisuuiii org/gisd/
[24] Thomas Mosgaard Giselsson, Rasmus Nyholm Jørgensen, Peter Kryger Jensen, Mads Dyrmann, and Henrik Skov Midtiby. A public image database for benchmark of plant scedling classification algorithms, 2017.
[25] Tao Gong, Chengqi Lyu, Shilong Zhang, Yudong Wang, Miao Zheng, Qian Zhao, Kuikun Liu, Wenwei Zhang. Ping Luo, and Kai Chen. Multimodal-gpi A vision and language model for dialogue with humans. arXiv preprint arXiv:2305. 04790, 2023.
[26] Jianyuan Guo, Kai Han, Han Wu, Yehui Tang. Xinghao Chen, Yunhe Wang, and Chang Xu. Cmt: Convolutional neural networks meet vision transformers, 2022.
[27] Suchin Gururangan, Ana Marasović. Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. Don't stop pretraining: Adapt language models to domains and tasks, 2020.
[28] Kai Han, Yunhe Wang, Jianyuan Guo, Ychui Tang, and Enhua Wu. Vision gnn: An image is worth graph of nodes, 2022.
[29] Kai Han, Yunhe Wang. Qi Tian, Jianyuan Guo, Chunjing Xu, and Chang Xu. Ghostnet: More features from cheap operations, 2020.
[30] Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. Transformer in transformer, 2021.
[31] Ju He, Jie-Neng Chen, Shuai Liu, Adam Kortylewski, Cheng Yang. Yutong Bai, and Changhu Wang. Transfg: A transformer architecture for tine-grained recognition, 2021.
[32] Kaiming Hc, Xinlei Chen, Saining Xic, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners, 2021.
[33] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, 2015.
[34] Tong He, Zhi Zhang, Hang Zhang, Zhongyue Zhang, Junyuan Xie, and Mu Li. Bag of tricks for image classification with convolutional neural networks, 2018
[35] Xiangteng He and Yuxin Peng. Fine-grained visual-textual representation learning. IEEE Transactions on Circuits and Systems for Video Technology. 30(2):520 5311, feb 2020.
[36] Hoang Trong Yu Gwang-hyun. Dang Thanh Vu, and Kim Jon-y. Late fusion of multimodal deep neural networks for weeds classification. Computers and Electronics in Agriculture, 175:105506, August 2020.
[37] Grant Van Horn, Oisin Mac Aodha, Yang Song. Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset, 2018.
[38] Grant Van Hom, Elijah Cole, Sara Beery, Kimberly Wilber, Serge Belongie, and Oisin Mac Aodha. Benchmarking representation learning for natural world image collections, 2021.
[39] Andrew Howard, Mark Sandler, Grace Chu, Liang-Chich Chen, Bo Chen, Mingxing Tan, Weijun Wang. Yukun Zhu, Ruoming Pang, Vijay Vasudevan, Quoe V. Le, and Hartwig Adam. Searching for mobilenetv3, 2019.
[40] Yiqi Huang, Jie Li, Rui Yang, Fukuan Wang, Yanzhou Li, Shuo Zhang, Fanghao Wan, Xi Qiao, and Wanqiang Qian. Hyperspectral imaging for identification of an invasive plant mikania micrantha kunth. Frontiers in Plant Science, 12:626516, 16, 2021.
[41] Huawei. Mindspore. https://ww.miindspor... cn/ 2020.
12

---

[42] Gabriel Ilharco, Mitchell Wortsman, Ross Wightman. Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannanch Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, July 2021. you use this software, please cite below.
[43] Talha Ilyas, Dewa Made Sri Arsa, Khubaib Ahmad, Yong Chae Jeong, Okjae Won, Jong Hoon Lee, and Hyongsuk Kim. Cwd30: comprehensive and holistic dataset for crop weed recognition precision agriculture, 2023.
[44] A community for naturalists. bttps::/www.inatet
org/ 2020. Accessed: 2020-11-14.
[45] Ajay Jaiswal, Shiwei Liu, Tianlong Chen, and Zhangyang Wang. The emergence of essential sparsity in large pre-trained models: The weights that matter, 2023.
[46] Sunny L Jardine and James N Sanchirico. Estimating the cost of invasive species control. Journal of Environmental Economics and Management, 87:242- 257, 2018.
[47] Jonathan Krause, Jia Deng. Michael Stark, and Li Fei-Fei. Collecting large- scale dataset of fine- fine-grained cars. 2013.
[48] Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. International Journal of Computer Vision, 128(7):1955) 18567))996-99888 1981, 2020.
[49] Bo Li. Yuanhan Zhang. Liangyu Chen, Jinghao Wang, Fanyi Pu, Jingkang Yang. Chunyuan Li. and Ziwei Liu. Mimic-it: Multi-modal in-context instruction tuning. arXiv preprint arXiv:2306.0544 05425, 2023.
[50] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language image pro-training with frozen image encoders and large language models. arXiv preprint arXiv:2301. /2597, 2023.
[51] Shiyang Li, Semih Yavuz, Wenhu Chen, and Xifeng Yan. Task-adaptive pre-training and self-training are complementary natural language understanding. 2023.
[52] Yanghao Li, Chao- Yuan Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer. Mvitv2: Improved multiscale vision transformers for classification and detection, 2022.
[53] Yiman Li, Yaxin Li, Jinao Liu, and Cong Fu. Research on invasive species recognition based on svm+ hog. In 202/ 2nd International Conference on Artificial Intelligence and Information Systems, pages 1-7, 2021.
[54] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. Microsoft coco: Common objects in context, 2015.
[55] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
[56] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023.
[57] Haotian Liu, Kilho Son, Jianwei Yang, Ce Liu, Jianfeng Gao, Yong Jac Lee, and Chumyuan Li. Learning customized visual models with aatrivaal-augmented knowledge, 2023.
[58] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows, 2021.
[59] Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xic. A convnet for the 2020s, 2022.
[60] Ziyi Liu, Junfeng Gao, Guoguo Yang, Huan Zhang, and Yong He. Localization and classification of paddy field pests using saliency map and deep convolutional neural network. Scientific reports, 6( 6(1):20410, 2016.
[61] David G Lowe. Distinctive image features from scale-invariant keypoints. International journal of computer vision, 60:91-110, 2004.
[62] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine- grained visual classification of aireraft, 2013.
13

---

[63] Sachin Mehta and Mohammad Rastegari. Mobilevit: Light-weight, general-purpose, and mobile-friendll vision transformer, 2022.
[64] Maria-Flena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Indian Conference Computer Vision, Graphics and Image Processing. Dec 2008
[65] General Administration of Customs of the People's Republic of China. Catalogue of quaramtine pests for import plants china (update2021044)) httpp://www. http://www..ppp...ttt///oooooooe wttt/hhh/counbreee/hhhhea reportingobliggation/ 2023.
[66] Alex Olsen. Dmitry A. Konovalov, Bronson Philippa, Peter Ridd, Jake C. Wood. Jamie Johns, Wesley Banks, Benjamin Girgenti, Owen Kenny, James Whinney, Brendan Calvert, Mostafa Rahimi Azghadi, and Ronald D. White. WeepWeeds: A Multiclass Weed Species Image Dataset for Deep Learning. Scientific Reporty, 9(058), 2 2019.
[67] Dean R Paini, Andy W Sheppard. David C Cook, Paul De Barro, Susan P Worner, and Matthew B Thomas. Global threat to agriculture from invasive species. Proceedings of the National Academy of Sciences, 113(27):7575-7577 2016.
[68] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021.
[69] Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces, 2020.
[70] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real- sal-time object detection with region proposal networks, 2016.
[71] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. arXiv preprint arXiv:2210.08400, 2022.
[72] Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng. Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: A large-scale, high-quality dataset for object detection. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4430-8439, 2019.
[73] Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang. and Deng Cai. Pandagpt: One model to instruction- follow them all. arXiv preprint arXiv:2305. 16355, 2023.
[74] Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data in deep learning era, 2017.
[75] Mingxing Tan and Quoe V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks, 2020.
[76] Bart Thomee, David A. Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. YFCC100m. Communications of the ACM, 59(2):64-73, jan 2016.
[77] Keyu Tian, Yi Jiang, Qishuai Diao, Chen Lin, Liwci Wang, and Zehuan Yuan. Designing bert for convolutional networks: Sparse and hierarchical masked modeling. arXiv:2301.0358) 2023.
[78] Hugo Touvron, Matthiuu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers distillation through attention, 2021.
[79] Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li. Maxvit: Multi-axis vision transformer, 2022.
[80] Chien- Yao Wang, Hong- Yuan Mark I Liao, I-Hau Ych, Yuch-Hua Wu, Ping- Yang Chen, and Jun-Wei Hsich. Cspnet: A new backbone that can enhance learning capability of enn, 2019.
[81] Haoqing Wang, Ychui Tang, Yunhe Wang, Jianyuan Guo, Zhi-Hong Deng, and Kai Han. Masked image modeling with local multi-scale reconstruction, 2023.
[82] Xiao Wang, Guangyao Chen, Guangwu Qian, Pengeheng Gao, Xiao Xiao-Yong Wei, Yaowei Wang, Yonghong Tian, and Wen Gao. Large-scale mudti pre-trained models: A comprehensive survey, 2023.
[83] Ross Wightman. Pytorch image models. 2019.
iaaaa-immmaaaaae models
14

---

[84] Sanghyun Woo, Shoubhik Debuath, Ronghang Hu. Xinlei Chen, Zhuang Liu, In So Kweon, and Saining Xie. Convnext v2: Co-designing and scaling convnets with masked autoencoders, 2023.
[85] Xiaoping Wu, Chi Zhan, Yu-Kun Lai, Ming-Ming Cheng, and Jufeng Yang. 1p102: A large-scale benchmark dataset for insect pest recognition. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2019.
[86] Chengjun Xie, Rujing Wang. Jie Zhang, Peng Chen, Wei Dong, Rui Li, Tianjiao Chen, and Hongbo Chen. Multi-level learning features for automatic classification of field crop pests. Computers and Electronics in Agriculure, 152:233 241, 2018
[87] Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoc V. Le. Self-training with noisy student improves imagenet classification, 2020.
[88] Zhenda Xie, Zheng Zhang, Yue Cao. Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. 2022.
[89] Xu, Yang Zhao, Xiang Hao, Xin Meng. Pink-eggs dataset vI: A step toward invasive species management using deep learning embedded solutions, 2023.
[90] Qin Xu, Jiahui Wang. Jiang, and Bin Luo. Fine-grained visual classification via internal ensemble learning transformer. IEEE Transactions on Multimedia, pages 1-14, 2023.
[91] Zhuolei Yang, Zheming Fan, Chenyu Niu, Peixin Li, and Hongjie Zhong. Development invasive plant recognition system based on deep learning. Journal of Advances in Mathematics and Computer Science, 38(6):39-55 53, 2023.
[92] Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality, arXiv preprint arXiv:2304. 14178, 2023.
[93] Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models, 2023.
[94] Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M. Ni, and Heung- Yeung Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection, 2022.
[95] Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng. Ruisi Cai, Zhao Song. Yuandong Tian, Christopher Ré, Clark Barrett, et al. II 2 o: Heavy-hitter oracle for efficient generative inference of large language models. arXiv preprint arXiv:2306. 14048, 2023.
[96] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:334411059.. 2023.
[97] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end to to-end object detection, 2021.
15

---

A Examples of annotations in Species196-L dataset
Figure 7 displays several examples of annotation boxes in our Species/96-L dataset. Although each image our dataset contains only one class of label, the objective detection experimental results shows that the dataset challenging for detecting small targets, as some images feature densely populated biological instances.
Figure 7: Samples of Species/96-L bounding box annotations. The top two rows show easy cases, while the bottom three rows present hard cases, including crowdedness, complex backgrounds, and tiny objects.
B Answers generated by different multi-modal models
For multiple-choice questions, some models output the category name directly instead of the options. Thus, we have appropriately relaxed our evaluation criteria. Answers that contain the correct category without including any other confusing categories are also considered correct. For true or false questions, we strictly evaluate the model's output. Table shows generated answers for example questions.
C Experiment on CLIP zero-shot classification
We conduct performance evaluations on OpenCLIP 42 models of different sizes on Species 196-I..
All models we tested are pretrained on LAION-2B (cn) with 224x224 image size. As shown in
16

---

Table Examples of answers generated by different models: For multiple-choice questions, some models output the category name directly instead of the options. Thus, we have appropriately relaxed our evaluation criteria. Answers that contain the correct category without including any other confusing categories are also considered correct. For true or false questions, we strictly evaluate the model's ss output.
Models
Trme or False Questions
Multiple Choice Questions
Poomet What the the iagge the Please chose bateer Uiienale Mossolo headlett arowen hrods.
spedde belond seto the kcbler?? seppord
specit belongings the kobaled
repord atterr "yef "wa" www.ee
bf enower
'yes"
Lillopadda"
"Yes, the image contains plant
belonging Apiales,
"Yes, the contains speciess
belonggng to Faholes, which can identified small white flower with pink anthers."
TThe belongs to categooy Magneliopoida, includes flowering plants and their insects."
which is group of (lowering plants."
Blip2
Yes, the image contains specics bekonging Fabules which is plant group communly referred the pea pea plants
"The species in the image belongs the order Insecia, more specifically onder Lepidoplera, which includes buteefliee and moths."
"Yes, contains belanging Apiales,
mPI.UG-Owl
as fcatures flower of cumot plani.
The flower is close-up up shot the plants bud which can identified belonging to the Faholes family."
image shows specimen
from the under insecta. specifically close
YYes, image contains species the Appipilles family, which is flowering plant."
"Yes, the image contains the Fabuocac family."
caterpillar on
pocce of whise paper
"Yes, The imuge contains species belonging to Apiales.
"A: Liliogeoda Monocuts Insecta Magnoliopsida eée Image: Ahd Instruction: the image species in the image'
"AAd Imnger Image: 460 Image:
sndde codge VYss/binnem python3 . coding: utf Createdd 2017-05-10 15:05:39. import at argaars. sys, json. re. requeses, urllib erree, rrlliir
Multimodal-GGPT 25
Add Image: #itt Image: the Image:
Emage: Image: Image:
"The image features close-up of brown, long haired caterpillar, Insecta species.
part
PandsGPT
"Yes."
"Yox."
The caterpillar mot
plant,
insect.
and it does not belong the Liliopsida, Monocots, Mognoliopsida groups"
Yes, there plant of theApiaccae family this image, which includes many different species honeysuckle and violets. The photo shows several flowers on the plant's stem."
"Yes, (bere plant from Fobaceuu family image. pink flowers un its stem. The plant appears which suggests it might be an cenamnttl variery of plants like daffodils or violets."
Visua--ii M6R MLR hi
"The image shows brown butterfly with wings small
be
17

---

Table 9: Comparison of different prompts and combination. The model used for comparison is
VIT-B/32 and (c) denotes classnames.
(1) T ACC.
Prompt number
Prompt
10.7
photo of [e], a type species.
9.58
photo of [c]. a invasive species.
9.58
photo {c), type of species.
10.19
(5)
photo of [c). type of invasive species.
10.23
(6)
photo of the {e], type of species.
10.06
(7)
photo of [e], a type invasive species.
9.8
(1)+
10.87
(1)+
10.94
(1) (4) (4)
10.91
table we first compare different prompts settings and finally select ('(c))" "photo of {c}, type species.", ,", "photo of {c}, type invasive species ') as our prompts.
Recently, there is work [15] tring to leverage knowledge extracted from large pre-trained multi-models facilitate model learning. In this work, We directly examine the influence of using different kinds of classnames. In our analysis, we employ scientific names, common names, appearance descriptions generated by LLM, and Minigpt-4 classnames. In some cases, the Al generated descriptions is irrelevant to the required response. Table presents examples of both relevant and irrelevant descriptions generated by the model.
The results (see Table IT) indicate that our dataset, using common names as classnames yields the best zero-shot classification performance. For the generated descriptions used classnames, L.LM outperforms the scientific names as classnames most models, while MiniGPT-4 has lower inference accuracy due the high proportion of irrelevant descriptions. Compared to other fine-grained datasets, the significantly lower accuracy of Species 196-L suggests that poses a new challenge in the field of zero-shot fine-grained classification.
16-1. Table 11: Experiment results of zero-shot classification on Species 96-le as well as other fine-grained datasets.
Species196-L
Model
Description (LLM)
Description (Minigpt-4)
Cars
FGVC Aircraft
Flowers102
Scientific name
Common name
VT-B7722
10.94
16.71
10.25
5.76
86.05
24.551
71.62
VIT-B/16
10.40
16.90
10.81
6.59
88.50
26.97
71.34
VIT-L/14
12.07
18.43
12.69
6.29
92.64
36.75
75.83
VIT-H/14
15.00
22.57
15.95
6.23
93.36
42.60
80.13
D Hosting and maintenance plan
Both the Species196-L and Species196-U datasets are publicly available at https:// species-dataset. wit.ae..igttt..... io/ This website hosted on Github Pages, widely-used web- site hosting service. The website contains introductions, experiment results, terms of use, and links to download the datasets, as well as usage guides. We maintain the data using Google Drive and Baidu Cloud, where we store the original URLs to download the images, ensuring that the dataset will be available for an extended period. Additionally, we will provide instructions on how download and organize the data with code. For further maintenance, we will continue refining our dataset, such as correcting incorrect labels and annotations Species-L., and updating a larger and more comprehensive version of Species-U.
E License
The Species / 196-1. dataset is provided You under the terms of the Creative Commons Attribution- NonCommercial--harre 4.0 International Public License ("CC BY-NC-SA 4.0"), with the
18

---

Table 10: Example of relevent and irrelevant descriptions generated by LLM and Minigpt-4.For LLM, the input is only scientific name. For Minigpt-4, we select one image for each category and design five prompts for inquiry.
Prompis
Sample relevant answers
Sample irrelevant answers
Descriptions generated by LLM
Briefly describe the appearance characteristics of the species (c). excluding size descriptions.
A pink, oval insect with wax-covered body and long filaments (Phenacoccus solenopsis)
adults are variable size. coloration. colour pattern and facies (Rhabdoscelus obscurus)
Descriptions generated by Minigpi-4
Describe the organism's physical features, such its size, shape. and color.
It has legs. The top part has a brown, rough surface, and the bottom part has dark brown, smooth surface. The middle part has a brown surface. (Spodoptera littorafis)
I'll look this image and answer your questions. (Hoplocerambyx spinicornis)
2. Describe the texture and surface features of the organism the picture.
The beetle's is skin is smooth. its body is yellow, and its antennas are brown with orange tips. (Scolynus scolytus)
Do you think that the organism's 's texture and surface features would cause to survive the habitat shown in the picture? (Avena barbata)
3. Describe any physical adaptations of the organism, such as specialized structures for defense, locomotion, feeding.
The insect has one set of eyes and antennae to look for food and also it can fly and run away from its predator. (Gonipterus scutellatus)
An organism's 's life cycle. A complete cycle can include any type of reproduction, but limited
span one organism. (Alewodicus dispersus)
4. Describe any unique patterns or markings on the organism's 's body.
The organism is small, but has two eyes. The eyes are very sharp. The antennae are long and feel things very well (Hoplocampa testudinea)
Answer the questions from the student's 's perspective. (Anastrepha)
5. Descrihe the organism's 's body structure, including any visible organs or systems.
It has four wings, head with eyes, a thorax with legs, and stomacc for digesting food. (Dasineura mali)
How can humans henefit from knowing about the organism? (Hylobius pales)
additional terms included herein. This dataset is used only for non-commercial purposes such as academic research, teaching, or scientific publications. We prohibits You from using the dataset any derivative works for commercial purposes, such as selling data or using it for commercial gain. The Species196-U dataset is distribute under the Creative Common CC-BY 4.0 license, which poses no particular restriction. The images are under their copyright.
19


Reviews:
[{"review_id": "bKZcM6gyGW", "reviewer": "Reviewer_QxhT", "summary": "", "questions": "", "limitations": "* Compared to fine-grained classification datasets (e.g., CUB-200-2011 and iNaturalist variants), the proposed dataset is large enough, but the class number is relatively small. Meanwhile, the labeled 19K dataset is extremely imbalanced, and some classes might have limited data over 196 categories, which will be too difficult for semi-supervised learning settings.\n\n* More semi-supervised learning baselines should be included in the benchmark results. Please refer to the semi-supervised learning benchmark USB [1] as the template. Popular semi-supervised methods like FixMatch [2] and semi-supervised fine-tuning experiments of pre-trained models [3] should be included.\n\nReference\n\n[1] Yidong Wang, et al. USB: A Unified Semi-supervised Learning Benchmark for Classification. NeurIPS, 2022.\n\n[2] Kihyuk Sohn, et al. FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence. NeurIPS, 2020.\n\n[3] Ting Chen, et al. Big Self-Supervised Models are Strong Semi-Supervised Learners. NeurIPS, 2020.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "* Semi-supervised learning with open-world fine-grained images is a novel problem in the era of large datasets and large pre-trained models. This dataset covers a diverse and large set of 196 invasive species, larger and more diverse than most prior work. This is useful for developing practical invasive species recognition systems.\n\n* This paper builds the labeled and unlabeled datasets with CLIP pre-trained models on LAION-5B, which is a novel strategy. The large unlabeled dataset Species196-U enables experiments on popular semi-supervised techniques, and baseline results take modern network architectures and self-supervised pre-training into consideration.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "GtfiloEBez", "reviewer": "Reviewer_3Z3U", "summary": "", "questions": "", "limitations": "- Data imbalance which is a big limitation of the dataset is discussed.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "- The proposed dataset is very useful in practice. It contains a lot of expert annotations of invasive species at a larger scale than previous work. It will be useful for both practitioners who want to specifically tackle this problem, and computer vision researchers who are looking for a fine-grained classification benchmark.\n\n- The final comparison and analysis is extensive with many approaches (supervised, semi-supervised, self-supervised), backbones (convnets and vision transformers) and tasks (detection and classification).", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "lWPy15gMTU", "reviewer": "Reviewer_foZN", "summary": "", "questions": "", "limitations": "1.\tThe study reports results of performance indices that may be affected by classes with a higher number of samples.\n2.\tAs the labeling of bounding boxes for pest locations was carried out on randomly selected subsets of images, their quantity should be reported somewhere.\n3.\tThe download links at https://species-dataset.github.io/download.html are not permanent and appear to be more like personal (Google Drive) storage. Therefore, the availability and maintenance of the benchmark require serious attention.", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1.\tThe introduction of a new large dataset for the development of machine learning approaches devoted to species of Mollusca, weeds, and insects recognition.  \n2.\tResults of a large number of representative methods for future reference.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "TQdxudUf1F", "reviewer": "Reviewer_H3At", "summary": "", "questions": "", "limitations": "There are not any immediate limitations of the presented work.", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "The paper presents a well crafted dataset, consisting of images with expert annotations of invasive species, accompanied with taxonomical information regarding the life stage of the pictured species. The combination with the large scale unlabeled Species196-U dataset, makes this a strong contribution to the fine-grained image analysis community, and very relevant. The datasets are thoroughly evaluated across several tasks.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "zqCppaaPrY", "reviewer": "Reviewer_fB1t", "summary": "", "questions": "", "limitations": "I think authors have mentioned the limitations and potential negative societal impact of their work. ", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1.\tThe paper is generally easy to read and easy to follow.\n2.\tThe paper states that bounding boxes were labeled for insect pests in accordance with the COCO format. This indicates that the dataset includes spatial information and enables sample-aware recognition, which can be beneficial for accurate pest location and categorization tasks.\n3.\tThe establishment of a hierarchical taxonomy system, spanning from domain to species level, provides detailed classification information for each invasive species. This can be valuable for researchers and practitioners working in the field of invasive species identification and management.\n4.\tThe Species196-L dataset includes images of invasive species at various life stages, such as eggs, larvae, pupae, and adults. This multi-grained approach allows for the study of different developmental stages and their impact on ecosystems and economies.\n5.\tThe article provides a wealth of baseline experiments, offering significant assistance to other researchers in furthering this research endeavor.\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: This paper received overall positive reviews. Reviewers generally regard the problem and the way of data construction novel, the dataset useful for fine-grained classification study, and the comparison and analysis extensive. They also raised some concerns and suggestions: clarification on some algorithm and dataset construction details, the data imbalance problem, evaluation with more and sophisticated algorithms, writing quality, etc. The rebuttal was persuasive and addressed most concerns. The final ratings unanimously recommend acceptance. The AC checked the paper, rebuttal, and review comments, and recommends accepting the paper.
Decision: Accept (Poster)