Direct Preference Heads (Preprint)
Collection
This collection contains the pre-trained, fine-tuned and aligned models for the Direct Preference Heads paper.
•
1 item
•
Updated
•
1
550M parameter Transformer-XL style model trained on 100B tokens of The Pile!
This model was originally trained for the "Direct Prefrence Heads" paper, but will also be used as the basis for much of my future research. All code used to train and run these models is available here: https://github.com/Avelina9X/memory-transformer-pt4
Name | Value |
---|---|
Total Parameters | 551M |
Non-Embedding Parameters | 512M |
Vocab Size | 50272 |
768 | |
1536 | |
18 | |
FFN Activation | SwiGLU |
4096 | |
Attention Type | Full |
Positon Embedding | Reversed RoPE with ABF |
24 | |
64 | |
Trained Context | 2048 |
Trained Memory | 2048 |
Max Inference Context | 4096 |
Model | Link |
---|---|
Pre-Trained Model | lovelace-medium-alpha1 |
Fine-Tuned Model | lovelace-medium-alpha1-instruct |
DPH Aligned Model | lovelace-medium-alpha1-instruct-hf |
DPH Aligned Model (Multiple Heads) | lovelace-medium-alpha1-instruct-hf-multihead |