This model is intended to be used with this repository: https://github.com/Coperek/energy-based-language-modeling
This model is an energy-based transformer - it assigns each token in a sequence an energy value. Low energy values mean that the token is a correct completion. Trained using noise contrastive learning on a single consumer GPU for around 80 hours. A mix of fineweb-edu and openbookcorpus was used.
Parameters: 200M Block count: 24 Hidden dim: 768 Attention heads: 12 Vocab size: 32001
Example output (generate_exact(), p=0.002): Prompt: "Once upon a time, there was a" Completion: "Once upon a time, there was a great deal of confusion among the people about what they thought constituted slavery. Somewhere else in the world today, however, there were still lots of people who believed that slavery was indeed happening."