UrbanAI-EH/mdco-chartqa-gemma4

MD-CO ChartQA — gemma4 Reproducibility Bundle

Frozen reproduction of MD-CO (Multi-task Distillation for Chart QA + OCR) on google/gemma-4-E4B-it over ChartQA (Masry et al. 2023).

Results (paper Table 1 convention, test split, relaxed accuracy %)

Convention: human = trained on train_human → eval test_human; augmented = trained on train_aug → eval test_aug; avg = (human + aug) / 2.

Strategy	human	augmented	avg
ZS	44.72	30.08	37.40
FT	65.28	93.04	79.16
MFT	78.32	94.08	86.20
DT	71.60	93.60	82.60
MDCO	75.84	91.84	83.84

MD-CO knowledge-distillation tuning (human split)

Config	human	augmented*
baseline (T=1)	75.84	91.36
ocr0 (OCR-KD off)	77.36	90.96
T3 (temp=3)	78.24	91.60
ocrT3 (OCR off + T3)	79.04	90.72

*augmented column here = same human-trained checkpoint evaluated on test_aug (diagnostic).

Key finding: raising the sequence-level distillation temperature (T=1 → 3) lifts MD-CO on the human split above the strong MFT baseline (see tuning table). The cross-checkpoint augmented MD-CO is still being tuned, so the headline MD-CO row above uses the baseline config.

Repo layout

• code/ — full source (train.py, evaluate.py, configs/, models/, trainers/, data/, scripts/)
• checkpoints/ — trained LoRA adapters: gemma4_dt_augmented, gemma4_dt_human, gemma4_ft_augmented, gemma4_ft_human, gemma4_mdco_augmented, gemma4_mdco_human, gemma4_mdco_human_T3, gemma4_mdco_human_ocr0, gemma4_mdco_human_ocrT3, gemma4_mft_augmented, gemma4_mft_human
• results/ — per-run eval summary + predictions JSON
• REPRODUCIBILITY.md — env + exact commands

Quick eval

pip install -r code/requirements.txt
python code/evaluate.py --model gemma4 --strategy mdco --data both --split test \
  --checkpoint checkpoints/gemma4_mdco_human --run-name gemma4_mdco_human \
  --override code/configs/vessl.yaml

Citation

@article{go2026mdco,
  title={MD-CO: A Knowledge Distillation Framework for Sophisticated Understanding
          and Reasoning in Chart Question Answering},
  author={Go, Young-Min and Jung, Hae Sun and Uprety, Sudan Prasad and Park, Keon Chul},
  journal={International Journal on Document Analysis and Recognition},
  year={2026}
}