Basavaraj000/pancake

Multimodal CLIP Classifier

A multimodal classification model that takes an image + text pair and predicts one of N classes. Built by fine-tuning a small classification head on top of frozen CLIP image and text embeddings.

⚠️ Placeholder values below (repo id, class names, dataset, metrics) — fill these in with your actual details before publishing.

Model Details

• Base model: openai/clip-vit-base-patch32
• Architecture: CLIP image + text encoders (frozen) → concatenated embeddings → 2-layer MLP classification head
• Task: Multimodal classification ({{NUM_CLASSES}} classes)
• Input: one image + one short text string
• Output: class label + per-class probabilities
• Repo: {{your-username}}/{{your-model-name}}

Intended Use

Describe here what the image+text pairs represent in your use case, and what the classes mean (e.g. "classifies product photos + descriptions into category X/Y/Z").

Training Data

• Dataset: {{dataset name / link}}
• Size: {{N examples}}
• Split: {{train/val/test sizes}}

Training Procedure

• Backbone frozen, only the classification head trained
• Optimizer: AdamW, lr=1e-4
• Loss: Cross-entropy
• Epochs: {{N}}
• Batch size: {{N}}

Evaluation Results

Metric	Value
Accuracy	{{TBD}}
F1 (macro)	{{TBD}}

How to Use

from model import MultimodalClassifier
from inference import load_model, predict

model = load_model("checkpoints/model.pt", num_classes=4,
                    base_model_name="openai/clip-vit-base-patch32")

result = predict(model, image_path="example.jpg", text="a description")
print(result["predicted_class"], result["confidence"])

Limitations

• Inherits any biases present in the pretrained CLIP model.
• Frozen backbone means performance is capped by CLIP's original embedding quality — unfreeze (freeze_backbone=False) and fine-tune end-to-end for more task-specific performance if you have enough data.

Files

• model.py — model definition (MultimodalClassifier)
• train.py — fine-tuning script
• inference.py — single-example inference / CLI
• requirements.txt — dependencies

---