Delta Attention Residuals
Thank you very much for sharing this paper. It was excellent timing, since I had not started the full training run yet.
KaiNomos was originally using Block AttnRes, but after reviewing the paper, I decided to replace it with Delta Block Attention Residuals before training. I am currently updating the implementation and running the necessary validation tests.
The Hugging Face repository is still titled KaiNomos-110M, but the architecture has since been expanded, and the current model is now around the 500M-parameter range. I will update the repository information once the configuration is finalized.
I am developing KaiNomos as part of a broader effort to encourage more people to design, train, and openly share their own small language models. I would like to help grow a community around independently developed compact LLMs, where people can exchange architectures, experimental results, failures, and relevant research rather than only publishing finished models.
Your message directly affected the model design before the main training run, so I genuinely appreciate you taking the time to share it. I will also make sure to cite the Delta Attention Residuals paper clearly in the repository and documentation.
but you need to know Delta Attention Residuals is more expensive in both computation cost and memory consumption compared to AttnRes.
@khtsly You were right about the computational and memory cost.
I implemented and tested the full per-sublayer Delta Attention Residuals variant on KaiNomos. With a sequence length of 1024, it ran out of memory at micro-batch 2 on my RTX 3090, reaching about 23.2 GB before failing.
It fits at micro-batch 1, but throughput drops from about 1,790 tokens/s with Delta Block to only 902 tokens/s. That would increase the estimated training time for 16B tokens from roughly 103 days to 205 days, so the full per-sublayer version is not practical for my current hardware.
I will therefore keep the block-level Delta Attention Residuals implementation. It fits comfortably at micro-batch 2, remains stable, and provides a much better performance-to-cost tradeoff.
Thank you again for warning me before I committed to the full training run. Your comment was accurate and helped me avoid making the wrong implementation choice.