CraftConnect-ViT

A fine-tuned Vision Transformer (ViT-B/16) that classifies photographs of Indian folk-art paintings into one of seven traditional art styles. It's the vision component of CraftConnect, a multimodal platform that turns craft photos and voice input from Indian artisans into marketplace-ready product listings — this model handles the "what style of art is this?" step so listings can be auto-tagged with the correct cultural category.

Quick facts

Developed by	Rohan L C (GitHub · LinkedIn)
Model type	Image classifier (Vision Transformer)
Base architecture	ViT-B/16, initialized from torchvision's ViT_B_16_Weights.IMAGENET1K_V1 (ImageNet-1K pretrained)
Task	Multi-class image classification, 7 classes
Validation accuracy	89.51%
Formats provided	PyTorch (vit_best.pt, full checkpoint) and ONNX (vit_best.onnx, opset 17, CPU-verified)
License	MIT
Part of	CraftConnect

Intended use

Classifying photographs of paintings/artworks into one of the seven Indian folk-art traditions listed below, primarily as a preprocessing step for marketplace listing generation (auto-tagging craft photos with the correct art-form category).

Out of scope:

• Art styles outside the 7 trained categories — the model will force a prediction into one of the 7 classes regardless of what's actually in the image; there is no "unknown/other" class.
• Authentication or provenance verification (this is a style classifier, not an authenticity or forgery detector).
• Non-painting inputs (photos of people, objects, text documents, etc.) — trained exclusively on painting/artwork images.
• High-stakes or commercial decisions without human review, given the small training set (see Limitations).

Classes

Index	Class
0	GOND
1	KALIGHAT
2	KANGRA
3	KERALA_MURAL
4	MADHUBANI
5	PICHWAI
6	WARLI

Training data

810 images total across the 7 classes above, split 80/20 with stratification (train_test_split(..., stratify=targets, random_state=42)) — 648 training images, 162 validation images. Training-time augmentation: RandomResizedCrop(224, scale=(0.6, 1.0)), RandomHorizontalFlip(), ColorJitter(0.2, 0.2), normalized with standard ImageNet mean/std. Validation used deterministic Resize(256) + CenterCrop(224) with the same normalization, no augmentation.

Training procedure

• Base model: torchvision.models.vit_b_16, initialized with ViT_B_16_Weights.IMAGENET1K_V1, classification head replaced with a 7-way linear layer.
• Loss: Cross-entropy with label smoothing (0.1).
• Optimizer: AdamW, learning rate 3e-4.
• Batch size: 32, epochs: 15 (best checkpoint selected by validation accuracy, achieved at epoch 15/15).
• Export: Best checkpoint exported to ONNX (opset 17) via torch.onnx.export, with numerical parity against the PyTorch model confirmed on CPU using onnxruntime.InferenceSession.

Evaluation

Metric	Value
Validation accuracy	89.51% (0.8950617...)
Validation set size	162 images (20% stratified holdout)
Best epoch	15 / 15

Note on the ONNX export: the 89.51% accuracy figure comes from the original PyTorch model evaluated on the real validation set. The ONNX export's correctness was separately verified by comparing PyTorch vs. ONNX outputs on synthetic random-noise tensors (confirming the export preserves the exact computed function) — this is a numerical-parity check, not an independent re-run of accuracy on real images. In local CPU benchmarking, ONNX inference speed was comparable to PyTorch (no measured speedup either direction), so the ONNX version should be understood as a portable, dependency-light deployment format rather than a faster one.

How to use

PyTorch

import torch
import torch.nn as nn
from torchvision import models, transforms
from huggingface_hub import hf_hub_download
from PIL import Image

CLASSES = ["GOND", "KALIGHAT", "KANGRA", "KERALA_MURAL", "MADHUBANI", "PICHWAI", "WARLI"]

# Rebuild the exact architecture used during training
model = models.vit_b_16(weights=None)
model.heads.head = nn.Linear(model.heads.head.in_features, len(CLASSES))

# Download and load the fine-tuned weights
ckpt_path = hf_hub_download(repo_id="RohanLC/craftconnect-vit", filename="vit_best.pt")
checkpoint = torch.load(ckpt_path, map_location="cpu")
model.load_state_dict(checkpoint["model_state"])
model.eval()

preprocess = transforms.Compose([
    transforms.Resize(256),
    transforms.CenterCrop(224),
    transforms.ToTensor(),
    transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]),
])

image = Image.open("your_craft_photo.jpg").convert("RGB")
x = preprocess(image).unsqueeze(0)

with torch.no_grad():
    pred_idx = model(x).argmax(1).item()

print(f"Predicted style: {CLASSES[pred_idx]}")

ONNX Runtime (CPU)

import numpy as np
import onnxruntime as ort
from torchvision import transforms
from huggingface_hub import hf_hub_download
from PIL import Image

CLASSES = ["GOND", "KALIGHAT", "KANGRA", "KERALA_MURAL", "MADHUBANI", "PICHWAI", "WARLI"]

onnx_path = hf_hub_download(repo_id="RohanLC/craftconnect-vit", filename="vit_best.onnx")
session = ort.InferenceSession(onnx_path, providers=["CPUExecutionProvider"])

preprocess = transforms.Compose([
    transforms.Resize(256),
    transforms.CenterCrop(224),
    transforms.ToTensor(),
    transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]),
])

image = Image.open("your_craft_photo.jpg").convert("RGB")
x = preprocess(image).unsqueeze(0).numpy()

outputs = session.run(None, {"input": x})[0]
pred_idx = int(np.argmax(outputs[0]))

print(f"Predicted style: {CLASSES[pred_idx]}")

Limitations and bias

• Small dataset. 810 images across 7 classes is a small fine-tuning set for a ~86M-parameter ViT. Regularization (augmentation, label smoothing) helps, but the model has not been validated on an independent test set separate from the single 80/20 split, and no cross-validation was performed — 89.51% should be read as a single-split estimate, not a robust generalization bound.
• No held-out test set. Only a train/validation split exists; there is no third, never-touched test set for a final unbiased estimate.
• Per-class performance unknown. No confusion matrix or per-class precision/recall has been published yet — overall accuracy may mask weaker performance on visually similar styles (e.g. regional mural traditions).
• Domain-narrow. Trained only on the 7 listed styles under presumably clean, single-subject photo conditions. Real-world artisan photos (poor lighting, occlusion, multiple items in frame, low-quality phone cameras) were not explicitly evaluated as a distinct test condition.
• Fixed label set. The model always outputs one of the 7 classes; it has no mechanism to express "none of these" or low-confidence abstention beyond reading the raw softmax scores yourself.

Citation

If you reference this model, please credit:

Rohan L C, "CraftConnect-ViT: Fine-tuned ViT-B/16 for Indian Folk-Art Classification," 2025.
https://huggingface.co/RohanLC/craftconnect-vit

Links: CraftConnect on GitHub · LinkedIn