Instructions to use AETHORIA-AI/TR-HASH-MoE-50M-4B-Pretraining with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use AETHORIA-AI/TR-HASH-MoE-50M-4B-Pretraining with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="AETHORIA-AI/TR-HASH-MoE-50M-4B-Pretraining", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("AETHORIA-AI/TR-HASH-MoE-50M-4B-Pretraining", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use AETHORIA-AI/TR-HASH-MoE-50M-4B-Pretraining with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "AETHORIA-AI/TR-HASH-MoE-50M-4B-Pretraining"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AETHORIA-AI/TR-HASH-MoE-50M-4B-Pretraining",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/AETHORIA-AI/TR-HASH-MoE-50M-4B-Pretraining

SGLang

How to use AETHORIA-AI/TR-HASH-MoE-50M-4B-Pretraining with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "AETHORIA-AI/TR-HASH-MoE-50M-4B-Pretraining" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AETHORIA-AI/TR-HASH-MoE-50M-4B-Pretraining",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "AETHORIA-AI/TR-HASH-MoE-50M-4B-Pretraining" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AETHORIA-AI/TR-HASH-MoE-50M-4B-Pretraining",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use AETHORIA-AI/TR-HASH-MoE-50M-4B-Pretraining with Docker Model Runner:
```
docker model run hf.co/AETHORIA-AI/TR-HASH-MoE-50M-4B-Pretraining
```

TR-HASH-MoE-50M-4B-Pretraining

An AETHORIA-AI causal language model with 51,522,048 parameters, exported after 122,070 pretraining optimizer updates over 3,999,989,760 training tokens. The repository name rounds the parameter and token counts to 50M and 4B.

This checkpoint contains the pretraining stage only, before refinement/refit or supervised fine-tuning.

Architecture

Property	Value
Transformer layers	8
Hidden size	512
Vocabulary size	32,000
Attention	Grouped-query attention, 4 query heads / 1 key-value head
Maximum configured sequence length	2,048 tokens
Feed-forward implementation	`tr_hash_engine`
Routing strategy	`token_id_composed_modulo`
Routed experts	4
Selected experts per token	2
Shared expert	Enabled; intermediate size 1,024
Configured intermediate size	1,408
Intermediate size per routed expert	352
Position encoding	RoPE; theta 10,000
Normalization	RMSNorm
Input/output embeddings	Tied

The checkpoint uses composed-modulo routing and the corresponding inference code provided in this repository.

Training

The training source was the tokenized TR-HASH Pretraining 125B Agentic 32K dataset, revision fc738b3a10c5c093e3b34b48bcf1cb7066184706. This run used a selected approximately 4B-token prefix through its pretraining replay plan. The dataset name describes the larger source collection, not the number of tokens consumed by this model.

Setting	Value
Training stage	Pretraining
Optimizer updates	122,070
Sequence length	2,048
Data-parallel workers	8
Sequences per worker per update	2
Gradient accumulation	1
Global tokens per update	32,768
Training tokens	122,070 × 32,768 = 3,999,989,760
Optimizer	AdamW
Peak learning rate	0.0003
Learning-rate schedule	Cosine; 1,000 warmup updates
Weight decay	0.1
Training precision	bfloat16

The training-token count matches the final metrics.jsonl record and the completed update count multiplied by the configured global batch. The last recorded training loss is 3.652127 at update 122,070. This is a training-log value, not a held-out evaluation score. No benchmark scores are reported for this checkpoint.

Loading

Install torch, transformers, and safetensors. The inference implementation is included in the repository; a Complexity Framework installation is not required.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "AETHORIA-AI/TR-HASH-MoE-50M-4B-Pretraining"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    torch_dtype=torch.float32,
).eval()

Downloads last month: 207

Safetensors

Model size

52M params

Tensor type

I64

F32

I32

AETHORIA-AI
/

TR-HASH-MoE-50M-4B-Pretraining

TR-HASH-MoE-50M-4B-Pretraining

Architecture

Training

Loading

Dataset used to train AETHORIA-AI/TR-HASH-MoE-50M-4B-Pretraining