LoserCheems/Doge-22M

Doge 22M

Doge is an ongoing research project where we aim to train a series of small language models to further explore whether the Transformer framework allows for more complex feedforward network structures, enabling the model to have fewer cache states and larger knowledge capacity.

In addition, Doge uses Inner Function Attention with Dynamic Mask as sequence transformation and Cross Domain Mixture of Experts as state transformation. This model is trained by Jingze Shi, it only allows text input and text generation, for detailed algorithm and model architecture, please refer to Wonderful Matrices, the ongoing research repository is Doge.

Uses

>>> from transformers import AutoTokenizer, AutoModelForCausalLM

>>> tokenizer = AutoTokenizer.from_pretrained("LoserCheems/Doge-22M")
>>> model = AutoModelForCausalLM.from_pretrained("LoserCheems/Doge-22M", trust_remote_code=True)
>>> inputs = tokenizer("Hey how are you doing?", return_tensors="pt")

>>> out = model.generate(**inputs, max_new_tokens=10)
>>> print(tokenizer.batch_decode(out))

Training

Model

• Architecture: Architecture details refer to Wonderful Matrices
• Pretraining data: HuggingFaceTB/smollm-corpus
• Pretraining steps: 5k
• Pretraining tokens: 1B
• Precision: bfloat16

NOTE: This model has not been fine-tuned for instruction

Environment

• Image: nvcr.io/nvidia/pytorch:24.10-py3
• Hardware: 1x NVIDIA RTX 4090
• Software: Transformers

Citation

@misc{shi2024wonderfulmatrices,
      title={Wonderful Matrices: More Efficient and Effective Architecture for Language Modeling Tasks}, 
      author={Jingze Shi and Bingheng Wu and Lu He and Luchang Jiang},
      year={2024},
      eprint={2407.16958},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2407.16958}, 
}