Instella✨: Fully Open Language Models with Stellar Performance

AMD is excited to announce Instella, a family of fully open state-of-the-art 3-billion-parameter language models (LMs) trained from scratch on AMD Instinct™ MI300X GPUs. Instella models outperform existing fully open models of similar sizes and achieve competitive performance compared to state-of-the-art open-weight models such as Llama-3.2-3B, Gemma-2-2B, and Qwen-2.5-3B, including their instruction-tuned counterparts.

Figure 1: Pareto frontier of pre-training tokens vs average performance for pre-trained and instruction-tuned models.

By training Instella from scratch on Instinct MI300X GPUs, we highlight our hardware’s capability and scalability in handling demanding large-scale AI training workloads, offering a viable alternative in the AI hardware landscape. In line with the AMD commitment to open source, we are releasing all artifacts related to Instella models here, including the model weights, detailed training configurations, datasets, and code, enabling the AI community to collaborate, replicate, and innovate, thereby accelerating progress.

Takeaways

Announcing Instella, a series of 3 billion parameter language models developed by AMD, trained from scratch on 128 Instinct MI300X GPUs.
Instella models significantly outperform existing fully open LMs (Figure 1) of comparable size, as well as bridge the gap between fully open and open weight models by achieving competitive performance compared state-of-the-art open weight models and their instruction-tuned counterparts.
Fully open and accessible: Fully open-source release of model weights, training hyperparameters, datasets, and code, fostering innovation and collaboration within the AI community.
Supported by the AMD ROCm software stack, Instella employs efficient training techniques such as FlashAttention-2, Torch Compile, and Fully Sharded Data Parallelism (FSDP) with hybrid sharding to scale model training over a large cluster.

Instella Models

In this release, we introduce the following Instella models:

Model	Stage	Training Data (Tokens)	Description
Instella-3B-Stage1	Pre-training (Stage 1)	4.065 Trillion	First stage pre-training to develop proficiency in natural language.
Instella-3B	Pre-training (Stage 2)	57.575 Billion	Second stage pre-training to further enhance problem solving capabilities.
Instella-3B-SFT	SFT	8.902 Billion (x3 epochs)	Supervised Fine-tuning (SFT) to enable instruction-following capabilities.
Instella-3B-Instruct	DPO	760 Million	Alignment to human preferences and strengthen chat capabilities with direct preference optimization (DPO).
	Total:	4.15 Trillion

Table 1: Instella models and training stages.

The Instella models are text-only, autoregressive transformer-based LMs having 3 billion parameters. Architecture-wise, Instella is packed with 36 decoder layers, each having 32 attention heads. These models support a sequence length of up to 4,096 tokens and have a vocabulary size of ~50,000 tokens using the OLMo tokenizer. During both pre-training and fine-tuning, we utilized FlashAttention-2, Torch Compile, and bfloat16 mixed-precision training to reduce memory usage, leading to computational speedups and optimal resource utilization. To balance inter-node memory efficiency and intra-node communication overhead within our cluster, we employed fully sharded data parallelism (FSDP) with hybrid sharding, with model parameters, gradients, and optimizer states sharded within a node and replicated across the nodes.

Our training pipeline is based on the open-sourced OLMo codebase, adapted, and optimized for our hardware and model architecture. For pre-training we used a total of 128 Instinct MI300X GPUs distributed across 16 nodes with each node having 8x Instinct MI300X GPUs. We evaluated our models and baselines using standard tasks from OLMES, FastChat MT-Bench, and Alpaca. For more details about the architecture, training pipeline/hyperparameters and evaluation results, please refer to our Blog, Hugging Face model card and Github repository.

Training Pipeline

The training of the Instella models comprised of four stages, where each stage incrementally enhanced the model’s capabilities from fundamental natural language understanding to instruction following and alignment towards human preferences.

Model Summary

Stage	Model	Training Tokens	Layers	Attention Heads	Model Hidden Size	MLP Hidden Size	Context Length	RoPE Theta
Pre-training	Instella-3B-stage1	4.065T	36	32	2560	13824	4096	10,000
Pre-training	Instella-3B	57.575B	36	32	2560	13824	4096	10,000
SFT	Instella-3B-SFT	8.902B (x3)	36	32	2560	13824	4096	10,000
SFT+DPO	Instella-3B-instruct	760M	36	32	2560	13824	4096	10,000

Hyparparameter

Stage	Optimizer	Peak LR	LR Scheduler	Alpha F	Warmup (steps)	Weight Decay	Decay Norm & Bias	Decay Embedding	Batch Size (Tokens)	Epochs
Pretraining Stage 1	AdamW(0.9,0.95)	4.0e-4	cosine_with_warmup	0.1	2000	0.1	True	True	4M	1
Pretraining Stage 2	AdamW(0.9,0.95)	4.0e-5	cosine_with_warmup	0.0	0	0.1	True	True	4M	1
SFT	AdamW(0.9,0.95)	1.0e-5	linear_with_warmup	0.001	500	0.1	True	True	0.5M	3
DPO	AdamW(0.9,0.95)	5.0e-7	linear	--	10%	0.1	--	--	0.25M	1

Getting Started

Installation

First, install PyTorch according to the instructions specific to your operating system. For AMD GPUs, you can also start with a rocm/pytorch docker.

To install from source (recommended for training/fine-tuning) run:

git clone https://github.com/AMD-AIG-AIMA/Instella.git
cd Instella
# install Flash-Attention on MI300X
GPU_ARCH=gfx942 MAX_JOBS=$(nproc) pip install git+https://github.com/Dao-AILab/flash-attention.git -v
# install other dependencies
pip install -e .[all]

Example Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
checkpoint = "amd/Instella-3B-Instruct"

tokenizer = AutoTokenizer.from_pretrained(checkpoint, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(checkpoint, device_map="auto", trust_remote_code=True)

prompt = [{"role": "user", "content": "What are the benefits of open-source AI research?"}]
inputs = tokenizer.apply_chat_template(
    prompt,
    add_generation_prompt=True,
    return_tensors='pt'
)

tokens = model.generate(
    inputs.to(model.device),
    max_new_tokens=1024,
    temperature=0.8,
    do_sample=True
)

print(tokenizer.decode(tokens[0], skip_special_tokens=False))

Chat in TRL

You can also use the TRL CLI to chat with the model from the terminal:

pip install trl
trl chat --model_name_or_path amd/Instella-3B-Instruct --trust_remote_code --max_new_tokens 1024

# <root>:
# which is bigger 9.8 or 9.11?

# <amd/Instella-3B-Instruct>:
# 9.8 is bigger than 9.11. The difference between the two numbers is 0.69 (9.8 - 9.11 = 0.69), which indicates that 9.8 is 0.69 units larger than 9.11.

Results

Pre-training

Models	Size	Training Tokens	Avg	ARC Challenge	ARC Easy	BoolQ	Hellaswag	PiQA	SciQ	Winnograde	OpenBookQA	MMLU	BBH (3-shot)	GSM8k (8-shot)
Open Weight Models
Gemma-2-2B	2.61B	~2T	59.34	39.46	59.30	74.50	70.50	76.40	96.60	69.80	44.80	53.28	40.75	27.37
Llama-3.2-3B	3.21B	~9T	62.51	47.16	64.91	74.80	73.10	75.90	95.30	70.30	51.20	57.81	47.00	30.10
Qwen2.5-3B	3.09B	~18T	68.30	51.51	67.19	79.10	72.10	77.40	95.50	69.30	51.40	67.22	56.69	63.84
Fully Open Models
Pythia-2.8b	2.91B	300B	49.83	40.47	60.70	64.80	60.10	72.50	89.70	60.80	42.60	26.09	27.69	2.73
GPTNeo-2.7B	2.72B	~420B	47.96	38.46	54.56	62.70	55.20	70.80	88.00	58.30	40.80	27.83	27.25	3.71
OpenELM-3B	3.04B	~1.5T	52.28	37.46	58.42	68.60	71.70	75.60	92.50	65.40	46.40	26.69	29.40	2.96
StableLM-3B-4E1T	2.8B	~4T	58.51	44.82	67.02	75.40	74.20	78.40	93.40	68.40	48.60	45.19	37.33	10.84
Instella-3B-Stage1	3.11B	~4T	61.33	53.85	73.16	78.70	74.20	77.50	94.90	71.20	51.40	54.69	34.30	10.77
Instella-3B	3.11B	~4T+60B	66.59	52.84	70.53	76.50	75.00	77.80	96.40	73.10	52.40	58.31	39.74	59.82

Table 2: Pre-trained model performance on standard benchmarks. Here Bold represents the best performance, and Underscore represents the second best performance.

Both Instella-3B-Stage1 & Instella-3B models outperform all the other fully open models over all the benchmarks individually (except PIQA). Our final pre-trained checkpoint Instella-3B outperforms the existing top performant fully open pre-trained models by a lead of ⬆️8.08% on average, with significant improvements in ARC Challenge [+8.02%], ARC Easy [+3.51%], Winnograde [+4.7%], OpenBookQA [+3.88%], MMLU [+13.12%] and ️GSM8K [+48.98%].
Second stage pre-training elevated the overall average performance relative to stage-1 by ⬆️5.26%, substantially narrowing the performance gap between Instella-3B model vs the closed-source models, and outperforming Llama-3.2-3B by ⬆️4.08% on average (+5.69% [ARC Challenge], +5.61% [ARC Easy], and +29.72% [GSM8k]), Gemma-2-2B by ⬆️7.25% on average (+13.38% [ARC Challenge], +11.23% [ARC Easy], +4.5% [Hellaswag], +7.6% [OpenBookQA], +5.03% [MMLU], and +32.45% [GSM8k]), and is competitive with Qwen-2.5-3B on the majority of the benchmarks.
The multi-stage pre-training with diverse and high-quality data mix significantly enhanced Instella-3B’s capabilities, establishing it as a competitive and open alternative in the landscape of comparable size language models.

Instruction-tuning Results

Models	Size	Training Tokens	Avg	MMLU	TruthfulQA	BBH	GPQA	GSM8K	Minerva MATH	IFEval	AlpacaEval 2	MT-Bench
Open Weight Models
Gemma-2-2B-Instruct	2.61B	~2T	39.04	58.35	55.76	42.96	25.22	53.45	22.48	55.64	29.41	8.07
Llama-3.2-3B-Instruct	3.21B	~9T	47.53	61.50	50.23	61.50	29.69	77.03	46.00	75.42	19.31	7.13
Qwen2.5-3B-Instruct	3.09B	~18T	48.72	66.90	57.16	57.29	28.13	75.97	60.42	62.48	22.12	8.00
Fully Open Models
StableLM-zephyr-3B	2.8B	4T	30.50	45.10	47.90	39.32	25.67	58.38	10.38	34.20	7.51	6.04
OpenELM-3B-Instruct	3.04B	~1.5T	14.11	27.36	38.08	24.24	18.08	1.59	0.38	16.08	0.21	1.00
Instella-3B-SFT	3.11B	~4T	42.05	58.76	52.49	46.00	28.13	71.72	40.50	66.17	7.58	7.07
Instella-3B-Instruct	3.11B	~4T	44.87	58.90	55.47	46.75	30.13	73.92	42.46	71.35	17.59	7.23

Table 2: Instruct model performance on standard benchmarks. Here Bold represents the best performance, and Underscore represents the second best performance.

Instella-3B-Instruct model consistently outperforms other fully open models across all evaluated benchmarks with a significant average score lead of ⬆️ 14.37% w.r.t the next top performing fully open instruction-tuned models. With substantial margins across all the chat benchmarks (+13% [MMLU], 7.57% [TruthfulQA], 7.43% [BBH], +4.46% [GPQA], +37.15 [IFEval], 10.08% [Alpaca 2], and 1.2% [MT-Bench]).
Instella-3B-Instruct narrows the performance gap with leading open-weight models. Instella-3B-Instruct performs on par with or slightly surpasses existing state-of-the-art open weight instruction-tuned models such as Llama-3.2-3B-Instruct (+5.24% [TruthfulQA], 0.45% [GPQA], and +0.1% [MT-Bench]), and Qwen2.5-3B-Instruct (+2.01% [GPQA] and +8.87% [IFEval]), while significantly outperforming Gemma-2-2B-Instruct with an average score lead of ⬆️5.83% (+0.55% [MMLU], +3.79 [BBH], +4.91 [GPQA], +20.47 [GSM8k], +19.98 [Minerva MATH], and +15.17% [IFEval]).
Overall, Instella-3B-Instruct excels in instruction following tasks and multi-turn QA tasks like TruthfulQA, GPQA, IFEval and MT-Bench, while being highly competitive compared to existing state-of-the-art open weight models on other knowledge recall and math benchmarks, while being trained on significantly fewer training tokens.

Training Data

Stage	Model	Dataset	License
Pre-training Stage 1	Instella-3B-stage1	https://huggingface.co/datasets/allenai/OLMoE-mix-0924	ODC-BY-1.0
Pre-training Stage 2	Instella-3B	https://huggingface.co/datasets/allenai/tulu-3-sft-mixture	ODC-BY-1.0
Pre-training Stage 2	Instella-3B	https://huggingface.co/datasets/allenai/dolmino-mix-1124	ODC-BY-1.0
Pre-training Stage 2	Instella-3B	https://huggingface.co/datasets/teknium/OpenHermes-2.5	Refer source materials
Pre-training Stage 2	Instella-3B	https://huggingface.co/datasets/TIGER-Lab/WebinstructSub	Apache-2.0
Pre-training Stage 2	Instella-3B	https://huggingface.co/datasets/m-a-p/Code-Feedback	Apache-2.0
Pre-training Stage 2	Instella-3B	https://huggingface.co/datasets/HuggingFaceH4/ultrachat_200k	MIT
Pre-training Stage 2	Instella-3B	https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus/viewer/python-edu	ODC-BY-1.0
Pre-training Stage 2	Instella-3B	https://github.com/google-deepmind/mathematics_dataset	Apache-2.0
Pre-training Stage 2	Instella-3B	https://huggingface.co/datasets/amd/Instella-GSM8K-synthetic	LICENSE
SFT	Instella-3B-SFT	https://huggingface.co/datasets/nvidia/OpenMathinstruct-2	CC-BY-4.0
SFT	Instella-3B-SFT	https://huggingface.co/datasets/cais/mmlu	MIT
SFT	Instella-3B-SFT	https://huggingface.co/datasets/HuggingFaceTB/smoltalk	Apache-2.0
SFT	Instella-3B-SFT	https://huggingface.co/datasets/GAIR/o1-journey	Refer source materials
SFT	Instella-3B-SFT	https://huggingface.co/datasets/allenai/tulu-3-sft-personas-instruction-following (subset of Tulu3)	ODC-BY-1.0
DPO	Instella-3B-instruct	https://huggingface.co/datasets/allenai/olmo-2-1124-7b-preference-mix	ODC-BY-1.0

Further information concerning the training datasets, including applicable licensing terms and use restrictions, may be located at the linked source location.

Conclusion

The release of the Instella family of models represents a significant stride in advancing open-source AI and demonstrating the capabilities of AMD hardware in large-scale language model training. The 3 billion parameter models from Instella family significantly outperform present fully open comparable size models in key benchmarks while also being competitive to comparable open-weight models, which we attribute to the high-quality data-mix selection, multi-stage training pipeline, and the use of high-performance Instinct MI300X GPUs for large scale training.

By fully open sourcing the Instella models, including weights, training configurations, datasets, and code, we aim to foster innovation and collaboration within the AI community. We believe that transparency, reproducibility and accessibility are key drivers of progress in AI research and development. We invite developers, researchers, and AI enthusiasts to explore Instella, contribute to its ongoing improvement, and join us in pushing the boundaries of what is possible with language models.

We will continue enhancing the models across multiple dimensions, including context length, reasoning ability, and multimodal capabilities. Additionally, we will scale up both the model and dataset while exploring diverse architectural approaches. Keep your eyes peeled for more exciting blogs on the Instella LMs family, its features and capabilities!

Additional Resources

Hugging Face Model Cards

Pre-trained models:
- Instella-3B-Stage1: amd/Instella-3B-Stage1, First stage pre-training checkpoint.
- Instella-3B: amd/Instella-3B, Final pre-training checkpoint.
Instruction-tuned models:
- Instella-3B-SFT: amd/Instella-3B-SFT, Supervised fine-tuned checkpoint.
- Instella-3B-Instruct: amd/Instella-3B-Instruct, Final Instruction-tuned checkpoint.

Datasets

Second stage pre-training GSM8k synthetic dataset: amd/Instella-GSM8K-synthetic

The dataset consists of two splits: train and train_119K.
For Instella-3B model second stage pre-training we used the train_119K split, which is a subset of the larger train split.

Code

Github: https://github.com/AMD-AIG-AIMA/Instella

Please refer to the following blogs to get started with using these techniques on AMD GPUs:

Bias, Risks, and Limitations

The models are being released for research purposes only and are not intended for use cases that require high levels of factuality, safety-critical situations, health, or medical applications, generating false information, facilitating toxic conversations.
Model checkpoints are made accessible without any safety promises. It is crucial for users to conduct comprehensive evaluations and implement safety filtering mechanisms as per their respective use cases.
It may be possible to prompt the model to generate content that may be factually inaccurate, harmful, violent, toxic, biased, or otherwise objectionable. Such content may also get generated by prompts that did not intend to produce output as such. Users are thus requested to be aware of this and exercise caution and responsible thinking when using the model.
Multi-lingual abilities of the models have not been tested and thus may misunderstand and generate erroneous responses across different languages.

License

The Instella-3B models are licensed for academic and research purposes under a ReasearchRAIL license.
The amd/Instella-GSM8K-synthetic dataset used in second stage pre-training is built with Qwen2.5-72B-Instruct, and is licensed for academic and research purposes under a ReasearchRAIL license. Refer to the LICENSE and NOTICES in the amd/Instella-GSM8K-synthetic dataset card files for more information.
Refer to the LICENSE and NOTICES files for more information.

Citations

Feel free to cite our Instella-3B models:

@misc{Instella,
    title = {Instella: Fully Open Language Models with Stellar Performance},
    url = {https://huggingface.co/amd/Instella-3B},
    author = {Jiang Liu, Jialian Wu, Xiaodong Yu, Prakamya Mishra, Sudhanshu Ranjan, Zicheng Liu, Chaitanya Manem, Yusheng Su, Pratik Prabhanjan Brahma, Gowtham Ramesh, Ximeng Sun, Ze Wang, Emad Barsoum},
    month = {March},
    year = {2025}
}

amd
/

Instella-3B