multimodal-vit-bert

This model is a fine-tuned version of the Vit ImageProcessor and Bert Tokenizer on an Kaggle dataset. It achieves the following results on the evaluation set:

• Loss: 1.1591
• Accuracy: 0.6514

Model description

The Model classifies snake species.

Intended uses & limitations

Classifies an image of a snake if additionally the continent the image was taken is added. Is trained with followin species: ["Agkistrodon contortrix", "Agkistrodon piscivorus", "Ahaetulla nasuta", "Ahaetulla prasina", "Arizona elegans", "Aspidites melanocephalus", "Atractus crassicaudatus", "Austrelaps superbus", "Bitis arietans", "Bitis gabonica", "Boa constrictor", "Bogertophis subocularis", "Boiga irregularis", "Boiga kraepelini", "Bothriechis schlegelii", "Bothrops asper", "Bothrops atrox", "Bungarus multicinctus", "Carphophis amoenus", "Carphophis vermis", "Causus rhombeatus", "Cemophora coccinea", "Charina bottae", "Chrysopelea ornata", "Clonophis kirtlandii", "Contia tenuis", "Corallus caninus", "Corallus hortulanus", "Coronella girondica", "Crotalus adamanteus", "Crotalus atrox", "Crotalus cerastes", "Crotalus cerberus", "Crotalus lepidus", "Crotalus molossus", "Crotalus ornatus", "Crotalus ruber", "Crotalus scutulatus", "Crotalus stephensi", "Crotalus tigris", "Crotalus triseriatus", "Crotalus viridis", "Crotaphopeltis hotamboeia", "Daboia russelii", "Dendrelaphis pictus", "Dendrelaphis punctulatus", "Dendroaspis polylepis", "Diadophis punctatus", "Drymarchon couperi", "Elaphe dione", "Epicrates cenchria", "Eunectes murinus", "Farancia abacura", "Gonyosoma oxycephalum", "Hemorrhois hippocrepis", "Heterodon nasicus", "Heterodon simus", "Hierophis viridiflavus", "Hypsiglena torquata", "Imantodes cenchoa", "Lampropeltis alterna", "Lampropeltis calligaster", "Lampropeltis getula", "Lampropeltis pyromelana", "Lampropeltis triangulum", "Lampropeltis zonata", "Laticauda colubrina", "Leptodeira annulata", "Leptophis ahaetulla", "Leptophis diplotropis", "Leptophis mexicanus", "Lycodon capucinus", "Malpolon monspessulanus", "Masticophis bilineatus", "Masticophis lateralis", "Masticophis schotti", "Masticophis taeniatus", "Micrurus fulvius", "Micrurus tener", "Morelia spilota", "Morelia viridis", "Naja atra", "Naja naja", "Naja nivea", "Natrix maura", "Nerodia cyclopion", "Nerodia floridana", "Nerodia taxispilota", "Ninia sebae", "Opheodrys aestivus", "Ophiophagus hannah", "Oxybelis aeneus", "Oxyuranus scutellatus", "Phyllorhynchus decurtatus", "Pituophis catenifer", "Pituophis deppei", "Protobothrops mucrosquamatus", "Psammodynastes pulverulentus", "Pseudaspis cana", "Pseudechis australis", "Pseudechis porphyriacus", "Pseudonaja textilis", "Python molurus", "Python regius", "Regina septemvittata", "Rhabdophis subminiatus", "Rhabdophis tigrinus", "Rhadinaea flavilata", "Rhinocheilus lecontei", "Salvadora grahamiae", "Salvadora hexalepis", "Senticolis triaspis", "Sistrurus catenatus", "Sistrurus miliarius", "Spilotes pullatus", "Tantilla coronata", "Tantilla gracilis", "Tantilla hobartsmithi", "Tantilla planiceps", "Thamnophis atratus", "Thamnophis couchii", "Thamnophis cyrtopsis", "Thamnophis marcianus", "Thamnophis ordinoides", "Thamnophis proximus", "Thamnophis radix", "Trimeresurus stejnegeri", "Tropidoclonion lineatum", "Tropidolaemus subannulatus", "Tropidolaemus wagleri", "Vipera ammodytes", "Vipera aspis", "Vipera seoanei", "Virginia valeriae", "Xenochrophis piscator"]

Training and evaluation data

DatasetDict({ train: Dataset({ features: ['image', 'label'], num_rows: 21434 }) validation: Dataset({ features: ['image', 'label'], num_rows: 2382 }) test: Dataset({ features: ['image', 'label'], num_rows: 3138 }) })

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 16
• eval_batch_size: 16
• seed: 42
• optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 7

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
1.8458	1.0	2680	1.6756	0.5243
0.9511	2.0	5360	1.1107	0.6700
0.3969	3.0	8040	1.1924	0.6826
0.1333	4.0	10720	1.3762	0.6986

Framework versions

• Transformers 4.50.0
• Pytorch 2.6.0+cu124
• Datasets 3.4.1
• Tokenizers 0.21.1