tiny-vision v1.0

Let your AI assistant watch videos. Many assistants, Claude included, can look at images but can't open a video file. tiny-vision turns a local video into a scene-by-scene description the assistant can read, through MCP (Claude Desktop, Claude Code, Cursor, VS Code, LM Studio and other MCP apps) or as plain text for any LLM. It runs locally, so nothing is uploaded.

It's a 36M-parameter image and video captioner, fast on a laptop CPU. It uses Meta's DINOv2-small as a frozen image encoder (its weights are unchanged, so this is not a fine-tune of DINOv2); the adapter and caption decoder on top (14M parameters) were trained from scratch on COCO.

  • 89% of Florence-2-base's COCO caption score at 1/6 the size
  • 8Γ— faster on GPU, 75Γ— faster on CPU (54 ms per image on an Apple M5 CPU)
  • 1.2 GB peak GPU memory, 144 MB weights

Benchmarks

COCO Karpathy test (5,000 images)

Decoding BLEU-4 CIDEr
greedy 0.342 1.145
beam 3 (default) 0.351 1.163

Against other small open models. Same 1,000 COCO test images, same scorer, Apple M5 (MPS GPU batch 16; CPU = single image). Reproduce with bench.py.

Model Params CIDEr BLEU-4 GPU img/s GPU peak memory CPU ms/img
tiny-vision v1.0 (greedy) 36M 1.196 0.339 52 1.2 GB 54
tiny-vision v1.0 (beam 3) 36M 1.213 0.348 24 1.2 GB 101
Florence-2-base 231M 1.348 0.364 6.7 7.0 GB 4,063
Florence-2-base (beam 3) 231M 1.364 0.368 5.3 7.0 GB 4,228
SmolVLM-256M-Instruct ΒΉ 256M 0.126 0.051 0.5 23 GB 55,800

ΒΉ SmolVLM is a chat model. Even when asked for one short sentence, it writes long descriptions, which caption metrics punish, so its quality score isn't a fair comparison. Its speed and memory numbers are. Florence-2 is a multi-task model (detection, OCR, grounding and more) and does more than captioning.

Progress across versions (greedy decoding)

Version Encoder Train data COCO CIDEr Flickr30k CIDEr
v0.1 from scratch Flickr8k (6k images) 0.061 0.131
v0.2 (unreleased) from scratch Flickr30k (29k) 0.121 0.204
v1.0 frozen DINOv2-small COCO (113k) 1.145 0.423

Metrics are corpus BLEU-4 and CIDEr-D, computed with a pure-Python reimplementation of the coco-caption scorers. It matches the reference code exactly on the same tokens, but tokenization is simple lowercase plus punctuation stripping rather than the PTB tokenizer, so scores are close to, but not identical with, numbers in papers.

Architecture

image / video frames β†’ DINOv2-small (frozen) β†’ 2Γ—2 pool β†’ adapter β†’ 6-layer decoder β†’ text
                        22M, Meta, Apache-2.0    64 tokens   MLP      prefix-LM, 384-dim
Part Details Trained here
Vision encoder DINOv2-small ViT-S/14, 224px β†’ 16Γ—16 patches, frozen no (pretrained by Meta)
Pooling + time 2Γ—2 average pool β†’ 64 tokens per frame, sin-cos time embedding per video frame β€”
Adapter 2-layer MLP (384 β†’ 384) yes
Caption decoder 6-layer prefix-LM transformer, 384-dim, RMSNorm + SwiGLU, 8,000-word vocabulary, tied embeddings yes

14M of the 36M parameters are trained, all on COCO. The encoder weights are bundled in model.safetensors, so the model runs fully offline after download.

Training

  • Data: COCO Karpathy train split (113k images, 566,747 captions)
  • 40,000 steps, batch 64 (~4.5 caption-epochs), AdamW (lr 3e-4, warmup 1,500, cosine to 3e-5), weight decay 0.05, bfloat16 autocast, random-resized-crop augmentation
  • One Apple M5 MacBook Pro GPU, ~3 hours. Final validation loss 2.117

Video: scene-by-scene summaries

summarize_video (MCP) and caption.py clip.mp4 --summary (CLI) do three things:

  • split the video into shots (hard cuts plus slow camera drift)
  • caption the start, middle and end of each shot
  • group consecutive shots that show the same thing into scenes

The output is compact text that an LLM turns into a real summary:

video: clip.mp4 Β· 00:09 long Β· 3 scenes (3 shots) Β· avg scene 3s
recurring: dogs (2 scenes)

scene 1 [00:00–00:03] (3s)
  seen: a group of dogs running in a field. | three dogs running in a field with a bench in the background.
  keywords: dogs, running, field, bench
...

Over MCP, the connected assistant writes the summary. From the CLI, --summarize uses a local Qwen2.5-1.5B-Instruct. Frames are captioned independently and there's no audio yet, so it describes what's seen, not what's said or how things move.

Example outputs (COCO test, beam 3)

tiny-vision Human reference
a man riding a motorcycle down a dirt road. A man with a red helmet on a small moped on a dirt road.
a man riding a bike next to a train on a track. A man on a bicycle riding next to a train
a group of people riding bikes down a street. people on bicycles ride down a busy street
a bathroom with a shower, toilet, and sink. A bathroom with an enclosed shower next to a sink and a toilet.
a woman sitting at a table with a cup of coffee. A young girl inhales with the intent of blowing out a candle. (wrong)

Usage

git clone https://github.com/amar683/tiny-vision && cd tiny-vision
uv venv && uv pip install -r requirements.txt
python caption.py --run amartomar683/tiny-vision photo.jpg
python caption.py --run amartomar683/tiny-vision clip.mp4 --scenes
python eval.py --run amartomar683/tiny-vision --dataset jxie/coco_captions --beams 3

As an MCP server (mcp_server.py), it provides the summarize_video, caption_image and describe_video tools. Add it to any app that runs local MCP servers, and that app's LLM can work with local images and videos. Then ask things like "what happens in ~/Downloads/talk.mp4?".

Limitations

  • COCO-style captions only: one short sentence about everyday scenes. No OCR, no precise counting, no names of people, places or brands, and no question answering.
  • Domain shift: it scores lower on Flickr30k (CIDEr 0.42) than on COCO. Unusual content (screenshots, documents, art, medical images) gives confident but wrong captions.
  • English only, word-level vocabulary (unknown words become <unk>).
  • Video frames are captioned independently. It doesn't understand actions across frames.
  • Like any model trained on COCO, it inherits COCO's biases (for example, guessing gender from appearance).

License and data

  • Code and the trained adapter/decoder weights: Apache-2.0.
  • Encoder: DINOv2-small by Meta AI, Apache-2.0.
  • Training data: COCO captions (annotations CC BY 4.0). The images come from Flickr under their own licenses; see the COCO terms of use.

Previous version: amartomar683/tiny-vision-flickr8k (v0.1, from scratch).

Downloads last month
-
Safetensors
Model size
36.1M params
Tensor type
F32
Β·
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Dataset used to train amartomar683/tiny-vision