Attention Research Checkpoints
Two experimental attention checkpoints, with their original weight bytes and separately documented loading requirements.
| Artifact | File | Bytes | SHA-256 |
|---|---|---|---|
| Depth-Attention, step 50,000 | model.safetensors | 1,855,036,720 | 1b5b9be7e3744417be24517d7095538a17baa62e0bac59c114ed5fc0f4d2aa44 |
| SparDA indexer, step 2,000 | indexer.pt | 203,587,595 | d46752606e29a3c2b838cdd988ab7878c2ef2bbaef8805f5a130ea44cbb88e95 |
Depth-Attention
The historical experiment name is “410M.” The stored checkpoint actually contains 463,749,120 parameters in 356 FP32 tensors, with a model vocabulary of 50,277. The supplied configuration retains the architecture and token-ID settings and removes local source-location metadata.
Use the matching custom Depth-Attention implementation and the experiment's pretokenized input IDs. The original 40-entry placeholder tokenizer is omitted because it does not provide the model's 50,277-token text mapping. The LlamaForCausalLM configuration label alone does not establish compatibility with stock Transformers: the weights include custom query/key normalization and output-layer normalization parameters. Text inference and standard automatic loading have not been validated for this release.
This is the completed reference run at step 50,000. The checkpoint was exported from a LLaMA-Factory-style training workspace. Optimizer, scheduler, RNG and training_args.bin payloads were removed before publication, so it is not a full resume checkpoint; the training and evaluation record is retained as trainer_state.json (50,000 steps, 5,012 logged entries). A historical revision label research-weights-20260930 referred to separate research exports (none_steps1000, v_mlp_steps1000, none_steps2000, qcv_steps2000, and the removed SrcQV1000 / Resformer2000 payloads). Those exports are not present here; only the native root model is stored.
SparDA indexer
This checkpoint contains 33 indexer projection tensors for use with MiniCPM4.1-8B and compatible SparDA code. It does not include the 8B base model weights and cannot run as a standalone language model.
| Record | Source run | Step | Bytes | SHA-256 |
|---|---|---|---|---|
| Baseline indexer (weight published) | lr5e-4_st2000_seq64k_ga4_tks2_tkst1_prolong64k_minicpm |
2,000 | 203,587,595 | d46752606e29a3c2b838cdd988ab7878c2ef2bbaef8805f5a130ea44cbb88e95 |
| Newmethod fixed-resume 2026-07-05 | lr5e-4_st2000_seq64k_ga4_tks2_tkst1_minicpm_newmethod_sparda_settings_fixed_resume_20260705_prolong64k_minicpm |
1,440 | 203,587,723 | 21a5aa3e019c0af7f6631a9d0fcce3250cbad0d1ac13387b7c288fa542122a66 |
Only the baseline indexer weight is published. The newmethod fixed-resume run is retained as a manifest only — weight not uploaded; manifest retained only: manifests/minicpm-newmethod-sparda-settings-fixed-resume-20260705.json.
The original .pt file includes optimizer, scheduler, RNG and experiment state. It is not a tensor-only export; use the matching research loader. Compatibility with torch.load(weights_only=True) and end-to-end inference have not been validated for this release.
Training settings recorded alongside the checkpoint (sparse_config): kernel 32 / stride 16, training kernel 2 / stride 1, block 64, window 2,048, top-k 64, dense length 8,192, selection_method: current, use_q_future_for_topk: true, selection_bound_type: partial_hybrid with selection_partial_dims: 64. sparse_config, the checkpoint key list and the per-run metadata are in sparda/baseline-step2000/; the base model is openbmb/MiniCPM4.1-8B. Evaluation consumed the checkpoint as --minicpm-sparda-indexer-path (accuracy) and --minicpm-indexer-path (efficiency).
SparDA experiment weights
Five indexer/router weights from the SparDA experiment family, kept under sparda/experiments/. They are the final/best/serve exports of their runs (intermediate phase-tiny-step-* files are not included).
| Artifact | Source run | Step | Bytes | SHA-256 |
|---|---|---|---|---|
| qams_car_v12_scored_query | qams_car_v12_scored_query_20260902 |
100 | 1,751,157 | 25aad4a3ea39c4105da56bec4d933fadd033d6b9c1982d3faff0498876150aa1 |
| qams_car_formal | qams_car formal (best accuracy) |
60 | 405,099 | d3499032444bf41bcafffeea8ae15637351c15e750860c329f4f50347cf925f8 |
| child_hierarchy | child_hierarchy_formal_R8_C83_K96_lr5e4_gbs64_v3_20260902 |
800 | 547,863 | a54ab5642b4852faed2a44728ef18c46e177ecf4df208910aa7f21c41c37a25d |
| hcr_minicpm | hcr_minicpm/hcr_R4_L32_seq16384_s2000 |
2,000 | 114,999 | feb37c927caaa6f15d1cb7f7c9cf13cba4a049ae7822f60d685fdd180717519b |
| mci_8b_dsa | mci_8b_dsa/mci8b_fullp6_Lall_seq16k_autobsz_50k_s2000 |
2,000 | 3,496,241 | 33a4700378152e521d9fe908ca8aed975a46d13c688ff11ffe2b943337dabc7e |
These are sanitised re-exports: absolute source-workspace paths embedded in the saved training arguments were replaced with <source-workspace>/ prefixes, and every tensor payload was verified equal to the source original (torch.equal, plus scalar/key equality) before publication. Per-artifact source size, source SHA-256 and verification counts are in sparda/experiments/<name>/manifest.json; the collection summary is sparda/experiments/MANIFEST.json.
A complete public mirror of the SparDA experiment workspace, including all 87 checkpoint payloads, is published separately as Xnhyacinth/sparda-checkpoints; the five weights above are the curated subset in this collection.
Licensing and scope
No blanket redistribution license is asserted for these research artifacts; the card uses license: other. The separately required MiniCPM4.1-8B base model is published by OpenBMB under Apache-2.0. Follow its terms and notices when obtaining or redistributing that dependency. Its base weights are not bundled here.
This release contains the two selected weight payloads, the Depth-Attention configuration and training-record files, and the SparDA indexer provenance metadata. Training logs, source-workspace absolute paths, account metadata and placeholder tokenizer files are excluded.