MUWU/MAGIC-IA3-experts

MAGIC: IA³ expert checkpoints

The 11 task expert checkpoints used in MAGIC's revised T0 IA³ merging experiment: RTE, CB, Winogrande, WiC, WSC, COPA, HellaSwag, Story Cloze and ANLI rounds 1–3. They share the pretrained bigscience/T0_3B backbone. MAGIC code is at lyymuwu/MAGIC.

These are the exact stronger-expert pool supplied to TA, TIES and DARE in the revised experiment. The checkpoints contain absolute IA³ gates, with pretrained value 1. Each has 192 vectors and 540,672 parameters. All 11 files together occupy about 24.4 MB. Backbone weights and dataset text are not included.

Files

• checkpoints/{task}/ia3.pt: task expert gates, unchanged from the experimental files.
• load_expert.py: loader for the native checkpoint format.
• configs/experts.json: expert hashes and preprocessing scales already included in the files.
• configs/base_model_config.json: shared backbone architecture.
• configs/evaluation.json, evaluation_indices.json: evaluation settings and example indices per prompt.
• configs/fusion_settings.json, sensitivity_ranking.json: merging and WSC settings.
• configs/training_archive/: archived continuation-training and expert-selection settings. Paths beginning with archive/ identify historical experiment records, rather than files bundled in this release.
• configs/results.json: mean accuracies for the revised experiment.
• reference/ia3.py: the original gate implementation used in the experiments.
• release_manifest.json, SHA256SUMS: file sizes and checksums.

Loading

Install PyTorch, Transformers, SentencePiece and huggingface_hub. Download the repository and import the supplied loader:

from huggingface_hub import snapshot_download
from pathlib import Path
import sys
import torch

root = Path(snapshot_download('MUWU/MAGIC-IA3-experts'))
sys.path.insert(0, str(root))
from load_expert import load_expert
model, tokenizer = load_expert(root / 'checkpoints/rte/ia3.pt', torch_dtype=torch.float32)
model = model.cuda().eval()

The loader wraps attention k/v and gated feed-forward wi_1 linear outputs, including decoder cross-attention, with the stored gates. This matches the experiment's implementation. These files are not Hugging Face PEFT adapter_model checkpoints. Do not pass them directly to PeftModel.from_pretrained.

For merging, the task increment is gate - 1. Expert-specific scaling is already included in the stored gates and must not be applied a second time. The common backbone's archived safetensors SHA256 is 6adbed46b2d65da3a43da32d3a2ab91eb643e5f359bd50a552f7d13bfe099a71.

Evaluation

The revised table uses the original 32 validation examples per task (352 unique examples), with 3,680 prompt-example pairs. Accuracy is rounded per prompt, the median is taken across prompts for each task, then tasks are averaged equally. These validation results are distinct from the separately archived full-benchmark experiment.

Model	Mean accuracy (%)
Pretrained T0	53.82
Task experts	75.56
TA	63.93
TA + WSC	68.48
TIES	68.76
TIES + WSC	69.33
DARE	63.93
DARE + WSC	67.61

WSC uses the shared top-20 sensitivity set. Evaluation used FP32 parameters, BF16 forward passes, disabled TF32, batch size 16 and maximum input length 512. The original environment used PyTorch 2.7.0+cu126 and Transformers 4.46.3. For exact multiple-choice evaluation, use the MAGIC scoring protocol and recorded prompt/example indices rather than free-form generation.