bengid/vit-flower-classifier

ViT-B/16 Flower Classifier

Fine-tuned torchvision.models.vit_b_16 (ImageNet-1K pretrained) for 102-class flower classification on the Oxford-102 Flowers dataset, with the full backbone unfrozen during fine-tuning. Achieves 0.9796 accuracy / 0.9637 macro-F1 on a held-out test split.

Note: earlier versions of this card reported 1.0 accuracy / 1.0 F1. Those numbers were produced by a data leak in the training pipeline's split logic and have been corrected — see Metrics correction below. The weights are unaffected; only the measurement was.

Recommended when accuracy is the priority and the extra size/latency budget is acceptable — e.g. offline batch labeling, research baselines, or any deployment where a ~344MB model and ~87ms mean inference isn't a constraint. For latency- or memory-constrained serving, see EfficientNetV2-S Flower Classifier, which gives up ~2.7pp of test F1 for ~4x less size and ~3x less latency.

Usage

import torch
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
from torchvision import models

weights_path = hf_hub_download(repo_id="bengid/vit-flower-classifier", filename="vit-flower-classifier.safetensors")

model = models.vit_b_16(weights=None)
model.heads[-1] = torch.nn.Linear(model.heads[-1].in_features, 102)
model.load_state_dict(load_file(weights_path, device="cpu"))
model.eval()

# preprocessing: resize(256) -> center-crop(224) -> normalize with dataset mean/std
# see src/utils.py:get_transforms() in the training repo for the exact pipeline
#(https://github.com/ben-gid/flowers/blob/main/src/utils.py)

Training Data

Oxford-102 Flowers — 8,189 images across 102 flower species, downloaded via torchvision.datasets.Flowers102. Class-weighted CrossEntropyLoss was used to correct for the dataset's uneven per-class image counts.

Training Procedure

Single-stage fine-tune with a two-phase backbone unfreeze callback (BackboneFinetuning): the ViT backbone starts frozen (only the classification head trains), then unfreezes at a fixed epoch with its own, lower learning rate and a separate parameter group — unlike this project's original (v1) EfficientNet-B0 model, which only ever unfroze its last 3 backbone blocks.

Hyperparameters

Parameter	Value
Optimizer	AdamW
LR scheduler	Cosine annealing (T_max=50, eta_min=1e-06)
Head LR (before unfreeze)	1e-3
Head LR (after unfreeze)	1e-3
Backbone LR (after unfreeze)	1e-5
Unfreeze epoch	5
Max epochs	50
Batch size	64
Effective batch size	256
Gradient accumulation	4
Precision	16-mixed
Weight decay	0.01
Early stopping patience	5

Evaluation

Oxford-102 split 70/15/15 by random_split(seed=42) → 5,733 train / 1,228 val / 1,228 test. Val and test are disjoint from train and from each other.

Metric	Validation	Test
Accuracy	0.9796	0.9796
Macro F1	0.9658	0.9637
Loss	0.1015	0.0973

Val and test accuracy are identical here by coincidence — both splits are 1,228 images and the model gets 1,203 right on each. The F1 scores differ, as expected.

Property	Value
Parameters	85,877,094
Model size	343.5 MB
Checkpoint size	1030.7 MB
Mean latency	86.5 ms
p95 latency	104.0 ms

Latency measured on ryzen 5600x cpu at batch size 1.

Macro F1 sits ~1.6pp below accuracy on both splits, which is the signature of uneven per-class performance on a dataset whose class counts range from 40 to 258 images — the rare classes are where the misses are. Per-class F1 is computed but not currently exported; see test_per_class_f1 in src/classifier.py.

Metrics correction — earlier scores were invalid

An earlier version of this card claimed 1.0 accuracy / 1.0 F1. That was a measurement bug, not a real result. FlowerDataModule.setup() in src/data.py built all three splits from the same subset:

train_subset, val_subset, test_subset = random_split(full_ds, (0.7, 0.15, 0.15), generator=generator)

self.train_set = SubsetWithTransform(train_subset, self.transform_train)
self.val_set   = SubsetWithTransform(train_subset, self.transform_test)  # bug — should be val_subset
self.test_set  = SubsetWithTransform(train_subset, self.transform_test)  # bug — should be test_subset

val_subset and test_subset were built and then discarded. Validation and test scored the model against the exact images it had trained on — total leakage, not partial. A ~1.0 score was the expected outcome of memorization and carried no information about generalization.

The weights are not contaminated. Training only ever read train_subset, so no held-out image ever reached a gradient. What the bug corrupted was measurement — and, through it, model selection: ModelCheckpoint and EarlyStopping were both driven by val_acc, which was really train accuracy. This checkpoint was therefore chosen on a signal that couldn't distinguish good epochs from overfit ones, and is unlikely to be the best epoch of its run.

The split logic is now fixed. The metrics above come from re-scoring this same published checkpoint against genuinely held-out data via notebooks/reevaluate_published_models.ipynb.

A full retrain on the corrected splits is planned. Expect these numbers to improve: this is a leak-era checkpoint re-measured honestly, not a model whose training was ever guided by a real validation signal. Early stopping never had a reason to fire at the right time, and no hyperparameter choice in this run was validated against held-out data.

Strengths & Weaknesses

Strengths:

• Best accuracy and macro-F1 of every architecture evaluated for this project (see comparison below) — attention-based global context handles flowers that differ mainly in overall shape/arrangement rather than local texture.
• Full-backbone fine-tuning lets every ViT layer adapt to the flower domain, avoiding the ceiling that partial-unfreeze approaches hit.
• Holds its lead on the corrected metrics: the ~2.7pp test-F1 margin over EfficientNetV2-S is real, where under the leaked numbers both models looked tied at ~1.0.

Weaknesses:

• Largest and slowest model in the lineup by a wide margin — ~4x the parameters and ~3x the mean latency of EfficientNetV2-S, for ~2.7pp of test F1. Whether that trade is worth it depends entirely on your latency budget; for most serving it isn't.
• ViTs are comparatively data-hungry and were historically harder to fine-tune from limited data before full-backbone unfreezing + a long enough schedule (see "Why the earlier models underperformed" below) — this model only reaches its ceiling because both were used.
• Not a good fit for edge/mobile or high-throughput serving given its size and latency.
• Selected by a checkpoint callback that was reading a leaked metric (see above), so this is likely not the best epoch this recipe can produce.

Limitations

• Closed-set, single-label: trained on exactly 102 Oxford flower species; will confidently misclassify any other flower species, non-flower image, or multi-flower image into one of the 102 known classes — there is no out-of-distribution rejection.
• Fixed input pipeline: expects a 224×224 center-cropped, normalized input (resize-then-crop). Unusual aspect ratios or off-center subjects can crop the flower out of frame.
• No adversarial robustness or calibration guarantees — confidence scores are not calibrated probabilities.
• Reported metrics are on held-out Oxford-102 val/test splits; real-world images (different lighting, backgrounds, camera quality) may perform worse.
• Macro F1 (0.964) is the number to trust, not accuracy (0.980) — Oxford-102 is class-imbalanced (40–258 images per class), and the gap between the two means errors concentrate in rare classes. If your use case cares about the long tail, budget for the F1 figure.
• This checkpoint predates the split fix and was selected on a leaked validation signal — a retrain is planned (see Metrics correction).

Intended Use

Intended uses:

• Flower species identification within the 102 Oxford-102 classes (gardening/botany apps, educational tools, dataset labeling).
• Backend model for this project's v2 /classify API endpoint when accuracy is prioritized over latency.

Out-of-scope uses:

• General-purpose plant, object, or scene classification outside the 102 trained species.
• Medical, toxicity, or safety-related plant identification.
• Any use where a wrong classification has safety or financial consequences without human review.

Model Comparison

This project trained four models in total, in this order. All figures are test-split accuracy on held-out data:

Model	Test Acc	Test F1	Params	Size (MB)	Best For
SimpleCNN (scratch)	~0.63	-	-	-	historical baseline only
EfficientNet-B0 (v1, partial unfreeze)	>0.93	-	-	-	historical baseline only
EfficientNetV2-S	0.9642	0.9364	20,308,150	81.8	efficient production serving
ViT-B/16 (this model)	0.9796	0.9637	85,877,094	343.5	maximum accuracy

The two v1 models were trained by the older, pre-Lightning pipeline (api/app/v1/flowers/train_scratch.py), which split train/val/test correctly — their numbers were never affected by the leak and are directly comparable to the corrected v2 figures above.

Why the earlier models underperformed

• SimpleCNN (scratch) was trained from randomly initialized weights with no ImageNet pretraining, on a 6-block custom CNN — too little capacity and too little prior visual knowledge to learn 102 fine-grained flower classes from ~8k images alone.
• EfficientNet-B0 (v1) started from ImageNet-pretrained weights but only ever unfroze its last 3 backbone blocks during fine-tuning (see this project's root README.md for the original two-stage recipe) — the earlier backbone layers, tuned for general ImageNet features, never adapted to flower-specific low/mid-level features.
• Both EfficientNetV2-S and ViT-B/16 (this model) unfreeze the entire backbone during fine-tuning, which drives the improvement from ~93% to ~96–98% test accuracy.

Note that this last gain is ~3–5 points, not the ~7 points the leaked metrics implied. Full-backbone unfreezing is a real improvement over partial unfreezing, but a far more modest one than a jump from 93% to "100%" suggested. The leaked numbers made a good architectural decision look like a spectacular one.

License

Apache 2.0, consistent with this project's license.

Citation

Base model (Vision Transformer):

@article{dosovitskiy2020vit,
  title={An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale},
  author={Dosovitskiy, Alexey and Beyer, Lucas and Kolesnikov, Alexander and Weissenborn, Dirk and Zhai, Xiaohua and Unterthiner, Thomas and Dehghani, Mostafa and Minderer, Matthias and Heigold, Georg and Gelly, Sylvain and Uszkoreit, Jakob and Houlsby, Neil},
  journal={arXiv preprint arXiv:2010.11929},
  year={2020}
}

Training dataset:

@inproceedings{nilsback2008automated,
  title={Automated flower classification over a large number of classes},
  author={Nilsback, Maria-Elena and Zisserman, Andrew},
  booktitle={2008 Sixth Indian Conference on Computer Vision, Graphics \& Image Processing},
  pages={722--729},
  year={2008},
  organization={IEEE}
}