Instructions to use LumiOpen/Llama-Poro-2-8B-Long-Base with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use LumiOpen/Llama-Poro-2-8B-Long-Base with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="LumiOpen/Llama-Poro-2-8B-Long-Base")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("LumiOpen/Llama-Poro-2-8B-Long-Base")
model = AutoModelForCausalLM.from_pretrained("LumiOpen/Llama-Poro-2-8B-Long-Base", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use LumiOpen/Llama-Poro-2-8B-Long-Base with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "LumiOpen/Llama-Poro-2-8B-Long-Base"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "LumiOpen/Llama-Poro-2-8B-Long-Base",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/LumiOpen/Llama-Poro-2-8B-Long-Base

SGLang

How to use LumiOpen/Llama-Poro-2-8B-Long-Base with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "LumiOpen/Llama-Poro-2-8B-Long-Base" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "LumiOpen/Llama-Poro-2-8B-Long-Base",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "LumiOpen/Llama-Poro-2-8B-Long-Base" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "LumiOpen/Llama-Poro-2-8B-Long-Base",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use LumiOpen/Llama-Poro-2-8B-Long-Base with Docker Model Runner:
```
docker model run hf.co/LumiOpen/Llama-Poro-2-8B-Long-Base
```

Poro 2 Long Base Model Card

Poro 2 Long Base is an 8B parameter decoder-only transformer created by extending Poro 2 8B Base from an 8K to 128K token context window using LongRoPE. The model supports both English and Finnish with an extended context window of 128K tokens. Poro 2 Long Base is released as a fully open source model under the Llama 3.1 Community License.

Poro 2 Long Base was created in a collaboration between AMD Silo AI, the TurkuNLP group of the University of Turku. Training was conducted on the LUMI supercomputer, using compute resources generously provided by CSC - IT Center for Science, Finland.

This model demonstrates how continued pretraining with LongRoPE-based context extension can efficiently add long-context capabilities to existing models while maintaining strong performance in both English and Finnish.

For more details on our training and data curation process, check out our .

Poro 2 Long Model Family

The Poro 2 Long model family extends the Poro 2 models with longer context support and checkpoints trained especially on math reasoning. There are four checkpoints released: a base model, an instruction-tuned model, a math reasoning SFT checkpoint, and the final math reasoning RL checkpoint.

Base Model	Instruct	Math Reasoning SFT	Math Reasoning RL
Poro 2 Long Base	Poro 2 Long Instruct	Poro 2 Long Math Reasoning SFT	Poro 2 Long Math Reasoning RL

What does Poro mean? Poro is the Finnish word for Reindeer! 🦌 These animals are native to Finland and hold a significant and historical role in Finnish culture.

Model Overview

NOTE: This is a base model which needs further fine-tuning for most use cases.

Poro 2 Long Base extends Poro 2 8B Base with a 16× context window increase from 8K to 128K tokens. The RoPE scaling factors were found via an evolutionary search using LongRoPE-MN, a multinode fork of Microsoft's LongRoPE adapted for HPC clusters.

Hyperparameter	Value
n_parameters	8.03B
n_layers	32
n_heads	32
n_kv_heads	8
d_model	4096
ffn_hidden_size	14336
vocab_size	128256
original_max_sequence_length	8192
max_sequence_length	131072
rope_theta	500,000
context_extension_method	LongRoPE
base_model	Llama-Poro-2-8B-base

Training

LongRoPE Factor Search

The per-dimension RoPE rescaling factors were found using poro2-longrope-search, a multinode-capable fork of Microsoft's LongRoPE. The search uses a genetic algorithm (dim_mono, which enforces monotonically increasing factors per dimension) initialized from YaRN and run across 4 LUMI nodes (32 GPUs total, TP=2). The search evaluated perplexity on long documents (≥128K tokens) using flash_attention_2.

The resulting 64 non-uniform rescale factors range from ~1.07 (low-frequency RoPE dimensions) to ~17.01 (high-frequency dimensions), consistent with the LongRoPE paper's finding that high-frequency dimensions require more aggressive rescaling for large context extensions. The same factors are used for both short and long sequences (short_factor == long_factor), and a magnitude scaling factor of sqrt(1 + log(16) / log(8192)) ≈ 1.14 is applied to attention scores following the "su" policy.

Factor search hyperparameters

Hyperparameter	Value
Algorithm	`dim_mono` (monotone per-dimension genetic algorithm)
Population size	64
Max time budget	40 (× evo_scale)
Mutation numbers	16
Crossover size	16
Parents size	32
Initialization	YaRN (llama settings: beta_fast=32, beta_slow=1)
Target length	131,072
Samples	30
Dataset minimum tokens	131,072

Context Extension Training

The context extension training was conducted using a custom fork of the NVIDIA Megatron-LM framework at OpenEuroLLM/NVIDIA-Megatron-LM (feature/context-extension branch), which adds native LongRoPE support via a new LongRoPERotaryEmbedding class and the --position-embedding-type longrope argument.

Training was conducted on 64 LUMI nodes (512 AMD MI250X GPUs) with 3D parallelism (TP=2, PP=4, CP=4).

Training hyperparameters

Hyperparameter	Value
Precision	bfloat16
Optimizer	AdamW
adam-beta1 / beta2 / eps	0.9 / 0.95 / 1e-8
Learning rate	1e-5
Min learning rate	1e-7
LR decay style	linear
LR warmup steps	100
Gradient clipping	1.0
Micro batch size	1
Global batch size	32 sequences
Tokens per step	~4M (32 × 131,072)
Train iterations	1,000
Total training tokens	~4B
Max sequence length	131,072
Attention	Flash attention
Position embedding	LongRoPE
Original max position embeddings	8,192
Magnitude scaling policy	`su`

Training Dataset

The context extension training uses a length-stratified mix of long documents from Finnish and English educational PDFs, mathematics, and code. Documents are binned by tokenized length, with heavy weighting toward the longest documents to maximize exposure to long-range dependencies.

Weight	Dataset	Language	Length bin
0.05	FinePDFs-edu	Finnish	< 4K tokens
0.05	FinePDFs-edu	Finnish	4K – 16K tokens
0.10	FinePDFs-edu	Finnish	16K – 64K tokens
0.20	FinePDFs-edu	Finnish	64K – 256K tokens
0.05	FinePDFs-edu	English	< 4K tokens
0.05	FinePDFs-edu	English	4K – 16K tokens
0.10	FinePDFs-edu	English	16K – 64K tokens
0.20	FinePDFs-edu	English	64K – 256K tokens
0.10	FineMath	—	—
0.10	StarCoder	—	—

Full Collection

Stage	Model	Dataset	License
CPT	Llama-Poro-2-Long-Base	https://huggingface.co/datasets/HuggingFaceFW/finepdfs-edu	ODC-BY-1.0
CPT	Llama-Poro-2-Long-Base	https://huggingface.co/datasets/HuggingFaceTB/finemath	ODC-BY-1.0
CPT	Llama-Poro-2-Long-Base	https://huggingface.co/datasets/bigcode/starcoderdata	Refer source materials
Instruct SFT	Llama-Poro-2-Long-Instruct	https://huggingface.co/datasets/LumiOpen/poro2-instruction-collection	Llama 3 Community License
Math SFT	Llama-Poro-2-Long-Math-Reasoning-SFT-Preview	https://huggingface.co/datasets/nvidia/Llama-Nemotron-Post-Training-Dataset	CC-BY-4.0
Math SFT	Llama-Poro-2-Long-Math-Reasoning-SFT-Preview	https://huggingface.co/datasets/LumiOpen/Llama-Nemotron-Post-Training-Dataset-SFT-math-FI	CC-BY-4.0
Math RL	Llama-Poro-2-8B-Long-Math-Reasoning-RL-Preview	https://huggingface.co/datasets/Kwai-Klear/KlearReasoner-MathSub-30K	Apache-2.0

Evaluation Results

Long Context Evaluation

We evaluate long-context performance using HELMET (English) and OneRULER (English and Finnish), comparing Poro 2 8B Long Base against Llama 3.1 8B base.

HELMET (English)

	8k	16k	32k	64k	128k
Poro 2 8B Long Base	58.67	57.63	58.24	53.62	48.97
Llama 3.1 8B	60.45	57.54	56.75	52.18	49.65

OneRULER

English

	8k	32k	64k	128k
Poro 2 8B Long Base	48.57	52.27	49.43	56.57
Llama 3.1 8B	40.86	56	48.29	63.14

Finnish

	8k	32k	64k	128k
Poro 2 8B Long Base	60.86	55.14	52.86	51.43
Llama 3.1 8B	55.57	54.43	45.43	40.86

Short Context Evaluation

Finnish

Finnish benchmarks are machine-translated versions of the English benchmarks.

	Llama 3.1 8B	Poro 2 8B Base	Poro 2 Long Base
ARC Challenge (FI)	38.48	48.98	48.63
GoldenSwag (FI)	41.84	56.72	57.05
MMLU (FI)	50.21	56.20	55.93
TruthfulQA (FI)	44.69	48.76	48.47
GSM8K (FI)	31.31	42.68	44.81

English

	Llama 3.1 8B	Poro 2 8B Base	Poro 2 Long Base
ARC Challenge	58.19	61.01	61.60
GoldenSwag	76.72	75.80	76.50
MMLU	65.41	63.91	62.58
TruthfulQA	44.26	46.47	46.82
GSM8K	50.27	54.59	53.90

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_name = "LumiOpen/Llama-Poro-2-Long-Base"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

# Example usage with extended context
prompt = "Kerro minulle Suomesta."  # "Tell me about Finland" in Finnish
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=200, temperature=0.7, do_sample=True)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)

Intended Use

Poro 2 Long Base is designed for:

Further fine-tuning for downstream tasks requiring long-context support in Finnish and English
Research on long-context language models
Serving as a base for instruction-following and chat models with up to 128K context

Ethical Considerations and Limitations

Poro 2 Long Base is an advanced language model optimized for English and Finnish, with additional capabilities in code and mathematics. As with most AI-driven systems, this model may reflect imperfections, biases, and idiosyncrasies present in its training data.

Key limitations:

This is a base model and requires fine-tuning for most practical applications
Limited proficiency in languages other than English and Finnish
Potential for generating biased or inappropriate content
May produce factually incorrect information
Long-context performance may degrade for very long sequences approaching the context limit

License

Built with Llama.

Poro 2 Long Base is released under the Llama 3.1 Community License. Please review the license terms before use.

Citation

@misc{poro2_2025,
    title={Poro 2: Continued Pretraining for Language Acquisition},
    author={Elaine Zosa and Jouni Luoma and Kai Hakala and Antti Virtanen and Mika Koistinen and Risto Luukkonen and Akseli Reunamo and Sampo Pyysalo and Jonathan Burdge},
    year={2025},
    howpublished={LumiOpen}
}

@misc{ding2024longrope,
    title={LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens},
    author={Yiran Ding and Li Lyna Zhang and Chengruidong Zhang and Yuanyuan Xu and Ning Shang and Jiahang Xu and Fan Yang and Mao Yang},
    year={2024},
    eprint={2402.13753},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}