diffusionpen-matan-ft

DiffusionPen (Nikolaidou et al., ECCV 2024) fine-tuned on Hebrew handwritten words. It produces one word per image (64×256 px) in the style of a given writer. The style is set by 5 reference word crops.

• Base: warm-started from cyttic/diffusionpen-matan, the same architecture fine-tuned further on the same data.
• Training data: cyttic/matan-hebrew-handwriting-words, 22,148 word crops by 491 writers.
• Architecture: UNet with 491 style classes and a 29-symbol vocabulary (character_classes.json). Text is encoded with CANINE-c. It works in the latent space of the SD-1.5 VAE. Style comes from the DiffusionPen MobileNetV2 style encoder.
• Characters: the 27 Hebrew letters, including final forms. No punctuation, digits or Latin.

Files

file	what
ema_ckpt.pt	UNet EMA weights (use this one for sampling)
ckpt.pt, optim.pt	raw weights and optimizer state (only needed to resume training)
sample.py	minimal runnable sampler
unet.py, feature_extractor.py	model code from the DiffusionPen repo (unchanged, MIT, see LICENSE-DiffusionPen)
style_models/iam_style_diffusionpen.pth	DiffusionPen style encoder (original release)
character_classes.json, writers_dict_train.json	vocabulary and writer-id → style-class maps

Quick start

pip install -r requirements.txt     # torch, diffusers, transformers, timm, einops, ...
python sample.py --text "שלום עולם" --style w100 --n 2
python sample.py --text "שלום" --refs r1.png r2.png r3.png r4.png r5.png   # your own style references

On the first run, sample.py downloads everything it needs from the Hub:

• this repo;
• the reference crops (from the dataset above);
• the SD-1.5 VAE and DDIM scheduler (stable-diffusion-v1-5/stable-diffusion-v1-5);
• google/canine-c.

It runs on GPU, or on CPU (slowly). Output goes to ./samples/. Each word is generated separately; to build a line, concatenate the words right to left.

Examples (examples/, writer w100, 30 DDIM steps):