Xnhyacinth/Depth-Attention-Checkpoints

Depth-Attention step-50000 research checkpoint

This repository contains the completed step-50,000 Depth-Attention reference weights as safetensors, plus the recorded model and generation configurations. The historical experiment calls this a 410M model; the checkpoint contains 463,749,120 parameters in 356 FP32 tensors and has a vocabulary of 50,277. Recorded training reached 50,000/50,000 steps; this is the completed endpoint, not a separately selected best checkpoint. No optimizer, scheduler, RNG, trainer state or resume checkpoint is included.

The weights require the matching custom Depth-Attention implementation, including its query/key and output normalization behavior. Although the config identifies LlamaForCausalLM, stock Transformers compatibility has not been established. The matching 50,277-token text tokenizer and custom loader are not provided here. The original small placeholder tokenizer is intentionally excluded; do not use it as a substitute. Neither text generation nor GPU inference was tested during this publication.

The safetensors file is the byte-identical historical weight export; its SHA-256 is 1b5b9be7e3744417be24517d7095538a17baa62e0bac59c114ed5fc0f4d2aa44. Local file SHA-256, safetensors structure and tensor count were checked before publication; the remote LFS identity and bounded byte-range readback were checked afterwards. No blanket license is asserted; review applicable terms before redistribution.

Download the files for research with hf download Xnhyacinth/Depth-Attention-Checkpoints --local-dir /volume/pt-dev/qjiu/depth-attention-checkpoint. The exported files alone are not a complete ready-to-run text inference package.