UrbanAI-EH/mdco-chartqa-llava

MD-CO ChartQA — llava Reproducibility Bundle

Frozen reproduction of MD-CO (Multi-task Distillation for Chart QA + OCR) on llava-hf/llava-1.5-13b-hf over ChartQA (Masry et al. 2023).

Results (paper Table 1 convention, test split, relaxed accuracy %)

Convention: human = trained on train_human → eval test_human; augmented = trained on train_aug → eval test_aug; avg = (human + aug) / 2.

Strategy	human	augmented	avg
ZS	21.36	17.68	19.52
FT	16.32	46.80	31.56
MFT	33.92	53.84	43.88
DT	25.44	46.56	36.00
MDCO	29.60	53.28	41.44

MD-CO knowledge-distillation tuning (human split)

Config	human	augmented*
baseline (T=1)	29.60	40.72
T3 (temp=3)	29.60	42.16
ocrT3 (OCR off + T3)	29.76	41.44

*augmented column here = same human-trained checkpoint evaluated on test_aug (diagnostic).

Limitation: LLaVA-1.5 is near-chance on charts zero-shot, so it cannot absorb the teacher distribution — KD does not help and MFT remains the best strategy. This is a documented capacity-floor boundary of MD-CO.

Repo layout

• code/ — full source (train.py, evaluate.py, configs/, models/, trainers/, data/, scripts/)
• checkpoints/ — trained LoRA adapters: llava_dt_augmented, llava_dt_human, llava_ft_augmented, llava_ft_human, llava_mdco_augmented, llava_mdco_human, llava_mdco_human_T3, llava_mdco_human_ocrT3, llava_mft_augmented, llava_mft_human
• results/ — per-run eval summary + predictions JSON
• REPRODUCIBILITY.md — env + exact commands

Quick eval

pip install -r code/requirements.txt
python code/evaluate.py --model llava --strategy mdco --data both --split test \
  --checkpoint checkpoints/llava_mdco_human --run-name llava_mdco_human \
  --override code/configs/vessl.yaml

Citation

@article{go2026mdco,
  title={MD-CO: A Knowledge Distillation Framework for Sophisticated Understanding
          and Reasoning in Chart Question Answering},
  author={Go, Young-Min and Jung, Hae Sun and Uprety, Sudan Prasad and Park, Keon Chul},
  journal={International Journal on Document Analysis and Recognition},
  year={2026}
}