ACDiR-LLaDA-Math500

This repository reproduces the reported ACDiR result on the legacy HuggingFaceH4 MATH-500 evaluation protocol. It combines:

• the frozen GSAI-ML/LLaDA-8B-Instruct actor;
• weights/critic-ckpt-000040.pt;
• count-set critic-guided remasking; and
• the vendored exact LLaDA runner derived from LMDeploy.

This is not a standalone Transformers checkpoint and cannot be loaded with AutoModel.from_pretrained() as a complete actor-plus-critic model.

Reported result

Dataset	Examples	Guided	Actor-only	Delta
HuggingFaceH4/MATH-500 test	500	44.00% (220/500)	39.60% (198/500)	+4.40 pp

The included test set is from HuggingFaceH4/MATH-500 revision 6e4ed1a2a79af7d8630a6b768ec859cb5af4d3be. It is not the reordered/ reprocessed ankner/math-500 split used by some JustGRPO evaluations, so the scores must not be compared as though they used identical examples and grading.

The 44.00% result intentionally retains the original ACDiR legacy answer extractor and grader. Replacing the grader changes the evaluation protocol and is outside exact reproduction of this result.

Important decoding settings are stored in configs/math500_44.json:

steps=256                 gen_length=512
block_length=32           block_steps=16
batch_size=1              temperature=0
remask_method=count_set   remask_temperature=0.4
lookback_blocks=1         remask_min_age_current=2
remask_max_age_lookback=6
reforward_after_remask=True
deterministic_joint_argmax=False
force_remask_window=0

Hardware and software

The release path was validated with:

• Linux and Python 3.11;
• PyTorch 2.5.1 with CUDA 12.1 wheels;
• a CUDA 12.6 toolkit for building FlashAttention 2.8.3;
• BF16 inference; and
• one NVIDIA H200 GPU for the strict reference run.

The full actor is approximately 16 GB in BF16 before runtime buffers. GPUs with less memory have not been validated for exact reproduction. The reported path uses LMDeploy full-window inference and varlen FlashAttention. Disabling varlen flash or changing the backend is useful for portability testing but is not the strict 44.00% configuration.

Install

Clone this Hugging Face model repository, enter its root directory, and create a clean Python 3.11 environment:

cd acdir-llada-math500

python3.11 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip setuptools wheel ninja packaging
python -m pip install torch==2.5.1 --index-url https://download.pytorch.org/whl/cu121
python -m pip install -r requirements.txt
python -m pip install flash-attn==2.8.3 --no-build-isolation

If a different PyTorch CUDA wheel is required by the host driver, install that wheel first and treat the run as a portability reproduction unless the final outputs match the reference counts.

The evaluator automatically downloads the pinned base-model revision 08b83a6feb34df1a6011b80c3c00c7563e963b07 into .cache/huggingface/hub. Pass a local model directory with --base_model to avoid the download.

Reproduce 44.00%

Run on exactly one visible GPU; batch size 1 is part of the reported protocol:

export LLADA_EXACT_BACKEND=lmdeploy
export LLADA_LMDEPLOY_FAST_MODE=full_window
export LLADA_LMDEPLOY_CUDAGRAPH=0
export LLADA_LMDEPLOY_VARLEN_FLASH=1

python eval_math500.py \
  --batch_size 1 \
  --nproc_per_node 1 \
  --result_dir outputs/math500_44

To use an existing actor copy:

python eval_math500.py \
  --base_model /absolute/path/to/LLaDA-8B-Instruct \
  --batch_size 1 \
  --nproc_per_node 1 \
  --result_dir outputs/math500_44

The expected final summary is:

guided   44.00%  (220/500)
base     39.60%  (198/500)  delta=+4.40%

Every run writes:

outputs/math500_44/eval_command.txt
outputs/math500_44/predictions/predictions_rank000.json

The prediction JSON contains the prompt, full guided sequence, full baseline sequence, gold solution, extracted answers, correctness flags, level, and per-sample ACDiR remask statistics. With multiple ranks, one file is produced per rank; the reference 44.00% run uses one rank.

Slurm

Portable Slurm helpers are included. Account, partition, environment-module, and CUDA-module names are cluster-specific and should be supplied at submit time. For example:

sbatch -A YOUR_ACCOUNT -p YOUR_GPU_PARTITION \
  --export=ALL,CONDA_MODULE=miniconda3,CUDA_MODULE=cuda/12.6,RUN_FLASH_ATTN_BUILD=1 \
  scripts/batch/setup_env.batch

sbatch -A YOUR_ACCOUNT -p YOUR_GPU_PARTITION \
  --export=ALL,CONDA_MODULE=miniconda3,CUDA_MODULE=cuda/12.6,LLADA_LMDEPLOY_VARLEN_FLASH=1 \
  scripts/batch/eval_math500.batch

If the cluster does not use environment modules, activate or expose conda, nvcc, and the CUDA libraries before submission and omit the module variables.

Regression checks

The CPU suite checks the release policy and rollout invariants without loading the 8B actor:

CUDA_VISIBLE_DEVICES="" python -m unittest discover -s tests -v

These tests do not replace the full 500-example GPU reproduction.

Serve locally

The same pinned actor download and critic can be exposed through a small OpenAI-compatible endpoint:

python serve_openai.py \
  --host 127.0.0.1 \
  --port 23333

The endpoint is /v1/chat/completions.

Artifact integrity

Artifact	SHA256
weights/critic-ckpt-000040.pt	b8f86493bfd629968e18ed362f47614affc869af3ab01827255e4025ba26e68c
datasets/MATH500/test/data-00000-of-00001.arrow	ff2663846092b986df3026f53904030cbaf8e061c9f978d319a7bd86b3a04ea4
configs/math500_44.json	b430248e1199d8d0d27dbf73f277fe3d436a4a2dd4866e41615bea410c1828c9

scripts/upload_to_hf.py excludes local caches, generated outputs, temporary files, and Slurm logs while retaining the portable batch scripts.

License and upstream components

The ACDiR release code and critic are provided under the Apache License 2.0; see LICENSE. Vendored LMDeploy code retains its Apache-2.0 license in lmdeploy/LICENSE. The LLaDA base model is not redistributed here and remains under its upstream MIT license. The included evaluation records originate from HuggingFaceH4/MATH-500 and remain subject to the upstream dataset/source terms.

Citation

Please cite the ACDiR release and LLaDA when using this checkpoint:

@misc{acdir2026,
  title        = {ACDiR-LLaDA-Math500},
  author       = {{ACDiR Project}},
  year         = {2026},
  howpublished = {Hugging Face model release}
}

@article{nie2025large,
  title   = {Large Language Diffusion Models},
  author  = {Nie, Shen and Zhu, Fengqi and You, Zebin and Zhang, Xiaolu and
             Ou, Jingyang and Hu, Jun and Zhou, Jun and Lin, Yankai and
             Wen, Ji-Rong and Li, Chongxuan},
  journal = {arXiv preprint arXiv:2502.09992},
  year    = {2025}
}