Image Caption Generator (ResNet50 + LSTM)

A ResNet50 (frozen, transfer learning) + LSTM decoder model that generates natural-language captions for images. Trained on Flickr8k.

Full project code, training pipeline, and documentation: https://github.com/adhamashraf7788/Image-Caption-Generator
Live interactive demo (Hugging Face Space): https://huggingface.co/spaces/AdhamAshraf/image_caption_generator

Files in this repo

vocab.json                             # vocabulary (shared across both checkpoints)
base_resnet_lstm/
├── best_model.pt                      # baseline checkpoint
└── config.yaml                        # baseline training config
resnet_lstm_regularized/
├── best_model.pt                      # regularized checkpoint (recommended -- best results)
└── config.yaml                        # regularized training config

Two checkpoints are provided:

Checkpoint	BLEU-4 (beam-3)	Notes
`base_resnet_lstm/best_model.pt`	0.1364	Initial baseline
`resnet_lstm_regularized/best_model.pt`	0.1557	Added LSTM output dropout, weight decay, gradient clipping — recommended

Both checkpoints share the same vocab.json (identical vocabulary, 2,662 tokens).

Architecture

Image → ResNet50 (frozen, ImageNet-pretrained) → 2048-d feature
      → Linear(2048 → 256) projection
      → fed as first input step to a 1-layer LSTM (hidden_dim=512)
      → LSTM generates caption word-by-word (beam search recommended, width 3)

Full architecture, preprocessing, and training details: see the GitHub README.

How to use

Requires the inference code from the GitHub repo (src/inference/predict.py and its dependencies) — these checkpoints are not standalone transformers-compatible weights, they're plain PyTorch state_dicts wrapped with config metadata.

from huggingface_hub import hf_hub_download
from src.inference.predict import Predictor  # from the GitHub repo's src/

checkpoint_path = hf_hub_download(
    repo_id="AdhamAshraf/image-caption-generator",
    filename="resnet_lstm_regularized/best_model.pt",
)
vocab_path = hf_hub_download(
    repo_id="AdhamAshraf/image-caption-generator",
    filename="vocab.json",
)

predictor = Predictor(checkpoint_path=checkpoint_path, vocab_path=vocab_path, device="cpu")
caption = predictor.predict("path/to/image.jpg")
print(caption)

Training data

Flickr8k — 8,091 images, 5 human-written reference captions each. Split 80/10/10 (by image, not caption, to avoid leakage) using a fixed seed.

Evaluation results (test set, 810 images)

Metric	Baseline + greedy	Baseline + beam-3	Regularized + greedy	Regularized + beam-3
BLEU-1	0.5127	0.5240	0.5444	0.5517
BLEU-4	0.1221	0.1364	0.1435	0.1557
ROUGE-L	0.4177	0.4265	0.4434	0.4527
METEOR	0.3266	0.3267	0.3480	0.3528

Full evaluation methodology, qualitative examples, and failure-case analysis: see the GitHub README.

Limitations

Trained on a small (8k image) dataset; struggles with image content/styles underrepresented in Flickr8k (predominantly people, dogs, and outdoor scenes).
Even the regularized model still shows some overfitting past its best epoch.
Generated captions are sometimes fluent but not fully grounded in image-specific detail.

See the GitHub README's Limitations section for a full discussion, including a documented failure case and how regularization + beam search improved it.

Downloads last month: -; Downloads are not tracked for this model. How to track

AdhamAshraf
/

image-caption-generator