QED-Base-v1-300M
Model Description
QED-Base-v1-300M is a decoder-only Transformer language model trained entirely from scratch.
Unlike fine-tuned or continually pretrained models, QED-Base-v1 was not initialized from or adapted from any existing pretrained model. All model weights were learned through original autoregressive pretraining using an independently created dataset.
The model serves as the foundation for future instruction-tuned and task-specific QED models while remaining suitable for language modeling research and experimentation.
Model Details
| Property | Value |
|---|---|
| Model Name | QED-Base-v1-300M |
| Model Type | Decoder-only Transformer |
| Parameters | 299.82M |
| Hidden Dimension | 1024 |
| Layers | 12 |
| Attention Heads | 16 |
| Activation Function | SwiGLU |
| Normalization | RMSNorm |
| Position Encoding | RoPE |
| Vocabulary Size | 96,000 |
| Tokenizer | SentencePiece |
| Training Method | Pretraining from Scratch |
| Base Model | None |
Architecture
QED-Base-v1 retains a modern decoder-only Transformer architecture featuring:
- Rotary Positional Embeddings (RoPE)
- RMSNorm
- SwiGLU feed-forward networks
- Multi-head causal self-attention
- Weight-tied input and output embeddings
The architecture is designed as a general-purpose language foundation model suitable for pretraining, research, and downstream fine-tuning.
Training Data
QED-Base-v1 was pretrained using a fully original dataset created specifically for this project.
The training corpus was independently authored and prepared for large-scale autoregressive language modeling.
The data preparation process included:
- Original content creation
- Custom formatting
- Custom preprocessing
- SentencePiece tokenization
- Dataset preparation for autoregressive pretraining
No existing pretrained model weights were used during training.
Intended Use
QED-Base-v1 is intended for:
- Language model research
- Transformer architecture experimentation
- Fine-tuning into instruction-following models
- Domain adaptation
- Educational projects
- Creating derivative language models
This is a foundation model and has not been instruction-tuned.
Limitations
As a pretrained foundation model:
- It is not optimized for instruction following.
- It may generate incomplete or unexpected responses.
- It has not been aligned with human preferences.
- It may require instruction tuning for assistant-style applications.
- Outputs should be reviewed before use in production or high-risk scenarios.
License
MIT License.
- Downloads last month
- 77