Aegis-Core

Building an open language model, one experiment at a time.

Aegis-Core is an open-source language model development project focused on building, training, and evaluating experimental language models.

The project aims to make the development process transparent and accessible to the community, including architecture decisions, training experiments, evaluation results, and known limitations.

Project Goals

• Develop an experimental language model architecture.
• Make the development process publicly visible.
• Document training and evaluation methodology.
• Encourage community review and contributions.
• Improve model capabilities through measurable experiments.

Model: Aegis-Nano

Aegis-Nano is the initial experimental model developed under Aegis-Core.

Configuration	Value
Architecture	Decoder-only Transformer
Layers	6
Attention heads	6
Embedding dimension	384
Context length	256 tokens
Vocabulary size	16,384
Parameters	17,011,584

Training

The v2 training experiment used:

• Training steps: 10,000
• Optimizer: AdamW
• Learning-rate schedule: Cosine
• Random seed: 1337

The training corpus contains English and Indonesian text.

Evaluation Results

Evaluation was performed on two evaluation sets:

• Old: Existing evaluation dataset.
• Independent: Separately prepared evaluation dataset.

The independent evaluation used stride 4. The old evaluation used stride 1.

Checkpoint	Evaluation Set	Loss	Perplexity
v1 — step 2,000	Old	6.3760	587.6
v1 — step 2,000	Independent	6.3612	578.9
v2 — step 10,000	Old	5.3670	214.2
v2 — step 10,000	Independent	5.3799	217.0

Evaluation Limitations

These results indicate lower evaluation loss for v2 compared with v1.

However:

• The independent evaluation uses stride 4 rather than evaluating every possible window.
• Dataset independence does not guarantee the absence of semantic overlap or source-level contamination.
• These measurements do not establish broad language understanding or generalization.
• Perplexity should not be interpreted as a standalone measure of model quality.

Known Limitations

Aegis-Nano remains experimental.

Observed limitations include:

• Inconsistent text coherence.
• Repetition during generation.
• Limited context length.
• No verified broad generalization benchmark yet.

Development Transparency

Aegis-Core aims to publish its development process, including:

• Model architecture and source code.
• Training methodology.
• Dataset documentation.
• Evaluation methodology and results.
• Known limitations and future improvements.

Source code and development history:

GitHub: Aegis-Core

License

This repository uses the Apache License 2.0.

Dataset and third-party component terms remain subject to their respective licenses and conditions.

Project Status

Experimental — under active development.

Model weights will be published after release checks and verification.