donut-irs990

This is a fine-tuned version of naver-clova-ix/donut-base trained to read IRS Form 990 documents and extract structured field data from them directly from images, no traditional OCR pipeline needed.

It was trained on a synthetic dataset of IRS 990 forms, covering 1,075 named fields across the form. Training ran for 3 epochs, roughly 36,000 steps, over about 17 hours on an NVIDIA DGX Spark (GB10 GPU).

This is the first model I have ever fine-tuned, built as part of a larger document AI pipeline for agnostic usecase. The goal is to eventually take a scanned Form 990 image and get back clean, structured field values automatically.

How it works

Donut (Document Understanding Transformer) reads the document image with a Swin Transformer encoder, then decodes structured text with an MBart decoder. There is no OCR step. The model outputs tagged field values like <f1_1>some value</f1_1> which map back to specific lines on the form.

Current state

The model extracts values from images and the vision encoder is clearly reading the right content. There is a known tokenizer vocabulary issue where some field tokens appear as <unk> in the output, which affects the structured parsing. This is a work in progress and will be addressed in a follow-up training run with corrected tokenizer setup.

Training setup

Base model: naver-clova-ix/donut-base. Dataset: synthetic IRS Form 990 images with ground truth field annotations. Training hardware: NVIDIA DGX Spark with GB10 GPU. Epochs: 3. Batch size: 2. Learning rate: 0.0002. Image size: 1280x960.

Notes

This is an early checkpoint of an ongoing project. The architecture works and the model is learning to read form fields correctly. Future versions will fix the tokenizer issue and train on a larger, more diverse dataset.