AhmedSSabir/ModernBERT-CNN-Caption-Reranker

ModernBERT-CNN Caption Reranker

ModernBERT-base with a CNN head that scores the relatedness between image-context words and a candidate caption. The output is a probability from 0 to 1. Use it to rank candidate captions or assign soft pseudo-labels to context/caption pairs. The model does not read images or generate captions.

Usage

Install the dependencies from requirements.txt, then run:

from huggingface_hub import snapshot_download
import sys

path = snapshot_download("AhmedSSabir/ModernBERT-CNN-Caption-Reranker")
sys.path.insert(0, path)
from inference import RelatednessScorer

scorer = RelatednessScorer(path, device="cpu")  # use "cuda" for a GPU
scores = scorer.score("dog, ball, grass", [
    "A dog plays with a ball on the grass.",
    "A person cooks in a kitchen.",
])
print(scores)

This is a custom CNN architecture. Use the included inference helper; the standard AutoModel loader is not configured for this head. Context words should come from the image's detected objects or scene labels.

Model and training

• Base: answerdotai/ModernBERT-base; all 22 encoder layers fine-tuned, embedding parameters frozen.
• Sequence length: 64 tokens for the combined context/caption input.
• Head: 32 convolution filters, kernel width 3, ReLU, local max pooling, time-major flattening, sigmoid score.
• Training: 330,790 pairs; validation: 41,275 pairs. Adam, learning rate 1e-5, batch size 16, seed 42; BF16/TF32 and compilation used during training.
• Early stopping after three epochs; best validation checkpoint: epoch 1, validation loss 0.256281.

Evaluation

Caption reranking: 45,000 candidates, nine per image, over 5,000 images. Select the highest model score per image, with the earliest candidate winning ties. No reference-based tuning was performed.

Metric	Score
BLEU-4	0.329180
METEOR	0.271592
ROUGE-L	0.552392
CIDEr	1.096723
SPICE	0.205827
BERTScore F1	0.939820

Additional tests:

• 41,849 held-out pseudo-labeled pairs: accuracy 0.890893, F1 0.904035, ROC-AUC 0.961563.
• 5,822 Flickr8K expert-rated pairs / 1,000 images: Spearman 0.293677, Kendall tau-b 0.212962 using fixed combined image-derived textual context.

Limitations

Training labels derive from semantic similarity scores thresholded at 0.3; they are not human judgments. Output probabilities are not calibrated human quality ratings, and human-rating agreement is modest. Object extraction errors, truncated inputs and domain shift affect results. Evaluation is from one training seed, and the derived training split and visual extraction pipeline differ from the historical experiment in the original paper.

A small CLIP-teacher pilot did not improve agreement with teacher preferences on the validation set (51/64 pairs before and after training). Caption-reranking metrics and human-rated quality were not evaluated for this pilot, so its effect on those outcomes remains unknown.

Citation

@InProceedings{Sabir_2023_CVPR,
  author    = {Sabir, Ahmed and Moreno-Noguer, Francesc and Padr{\'o}, Llu{\'i}s},
  title     = {Visual Semantic Relatedness Dataset for Image Captioning},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops},
  month     = {June},
  year      = {2023},
  pages     = {5598--5606},
  doi       = {10.1109/CVPRW59228.2023.00592},
  url       = {https://openaccess.thecvf.com/content/CVPR2023W/O-DRUM/html/Sabir_Visual_Semantic_Relatedness_Dataset_for_Image_Captioning_CVPRW_2023_paper.html}
}