adityarajsahu/LaTeX-OCR

A compact image-to-LaTeX model built by merging:

• Vision encoder: openai/clip-vit-base-patch32
• Language decoder: EleutherAI/pythia-70m
• Projector: a small MLP mapping CLIP patch embeddings into Pythia's embedding space

Fine-tuned on lukbl/LaTeX-OCR-dataset to convert images of rendered math formulas into their LaTeX source.

Usage

This model uses custom (trust_remote_code-free) classes registered in this repo; load it via the project's LaTeXOCRModel, or copy src/model.py into your own code:

from transformers import AutoTokenizer, CLIPImageProcessor
from src.model import LaTeXOCRModel
from PIL import Image

repo_id = "adityarajsahu/LaTeX-OCR"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
image_processor = CLIPImageProcessor.from_pretrained(repo_id)
model = LaTeXOCRModel.from_pretrained(repo_id)

image = Image.open("formula.png").convert("RGB")
pixel_values = image_processor(images=image, return_tensors="pt")["pixel_values"]

generated_ids = model.generate(pixel_values=pixel_values, tokenizer=tokenizer, max_new_tokens=192)
print(tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0])

Evaluation

See eval_report_test.json in this repo for BLEU-4, exact-match rate, and normalized edit-distance on a held-out split, plus sample predictions.

Training details

Trained with src/train.py in the accompanying project repository — see configs/config.yaml there for the exact hyperparameters used.