AEGIS-Mini-V1-LoRA

Aegis-mini-1.0 is a small language model for electronics, drones, robotics, math, and physics. This repository holds the LoRA adapter of the model. The adapter works on top of the base model Qwen3-4B. A GGUF file for phones and laptops is in a different repository: Xerxes-28/AEGIS-Mini-V1-GGUF.

NOTE: This card follows the rules of ASD-STE100 (Simplified Technical English) as closely as possible. Technical names, such as LoRA, GGUF, and quantization, are in the text on purpose.

---

Table of contents

1. Summary
2. Model details
3. Intended use
4. Safety information
5. How to use the adapter
6. Prompt format
7. Generation settings
8. Training data
9. Training procedure
10. Results
11. Limitations
12. Compute used
13. Files in this repository
14. License
15. Citations
16. Contact

---

1. Summary

Aegis-mini-1.0 is a fine-tuned version of Qwen3-4B. The author trained it on questions and answers about electronics, drones, robotics, math, and physics. Most of the training data is theory. The model is small. It can run on a phone after conversion to GGUF. The author did not run public benchmarks. Read Section 10 and Section 11 before you use the model.

2. Model details

Item	Value
Model name	Aegis-mini-1.0
Repository	Xerxes-28/AEGIS-Mini-V1-LoRA
Developer	Xerxes-28
Model type	LoRA adapter (PEFT) for a causal language model
Base model	unsloth/Qwen3-4B-unsloth-bnb-4bit (4-bit copy of Qwen3-4B)
Language	English
Task	Text generation (chat)
Training method	Supervised fine-tuning (SFT) with QLoRA
Trainable parameters	33,030,144 (0.81% of 4,055,498,240)
Training context length	2,048 tokens
Training date	2026-10-31
License	Apache-2.0 (same as the base model)
GGUF version	Xerxes-28/AEGIS-Mini-V1-GGUF

3. Intended use

3.1 Use cases

You can use the model for these tasks:

• Explain a concept in electronics, for example Ohm's law, filters, or transistor bias.
• Explain a concept in drones and robotics, for example PID control or sensor fusion.
• Do a step-by-step calculation in math or physics.
• Give a first draft of an answer that you then check.
• Study and learn. The model is a good partner for practice.

3.2 Uses that are not in scope

Do not use the model for these tasks:

• Safety-critical decisions of any kind.
• Final design values for a real circuit, battery pack, or aircraft.
• Medical, legal, or financial advice.
• Exact data from a datasheet, for example register addresses or pin numbers.

4. Safety information

WARNING DO NOT USE THE MODEL OUTPUT AS THE ONLY SOURCE FOR A SAFETY CALCULATION. The model can give a wrong number with a confident tone. A wrong number can cause injury or fire.

WARNING DO NOT TOUCH MAINS VOLTAGE OR HIGH-VOLTAGE CIRCUITS BECAUSE THE MODEL TOLD YOU IT IS SAFE. Electrical shock can cause injury or death. Ask a qualified person.

CAUTION Make sure that you check each value in the datasheet of the part. A wrong value can cause damage to the part, the battery, or the drone.

CAUTION Do not fly a drone with a setting that the model gave you before you test it in a safe place. A wrong control setting can cause a crash.

5. How to use the adapter

You need the adapter and the base model. There are three ways to load them.

5.1 Method A: Unsloth (fastest)

Unsloth loads the base model and the adapter together.

from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="Xerxes-28/AEGIS-Mini-V1-LoRA",
    max_seq_length=2048,
    load_in_4bit=True,
)
FastLanguageModel.for_inference(model)

Do not wrap this model again with PeftModel. Unsloth reads adapter_config.json and loads the adapter by itself.

5.2 Method B: Transformers and PEFT

Use this method when you do not want Unsloth. You need transformers, peft, accelerate, and bitsandbytes.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_id = "unsloth/Qwen3-4B-unsloth-bnb-4bit"   # same base that the author used for training
adapter_id = "Xerxes-28/AEGIS-Mini-V1-LoRA"

tokenizer = AutoTokenizer.from_pretrained(adapter_id)
model = AutoModelForCausalLM.from_pretrained(base_id, device_map="auto")
model = PeftModel.from_pretrained(model, adapter_id)
model.eval()

5.3 Run one answer

SYSTEM_PROMPT = (
    "You are Aegis-mini-1.0, a compact AI assistant for electronics, drones, robotics, "
    "math and physics. You are strongest at theory and calculations. Be concise: state the "
    "key formula, show the steps with units, then give the result. If you are unsure of an "
    "exact value (register address, pin number, part spec), say so and tell the user to "
    "check the datasheet instead of guessing. For topics outside your focus, say so briefly."
)

messages = [
    {"role": "system", "content": SYSTEM_PROMPT},
    {"role": "user", "content": "A 5 V supply drives a red LED (2.0 V, 10 mA) through a resistor. Find the resistor value."},
]

inputs = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt",
    return_dict=True,
).to(model.device)

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=512,
        do_sample=True,
        temperature=0.3,
        top_p=0.8,
        top_k=20,
        min_p=0.0,
        repetition_penalty=1.05,
    )

new_tokens = output[0][inputs["input_ids"].shape[1]:]
print(tokenizer.decode(new_tokens, skip_special_tokens=True))

5.4 Merge the adapter into the base model (optional)

Merge the adapter when you want one set of weights, for example to make a GGUF file.

from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="Xerxes-28/AEGIS-Mini-V1-LoRA",
    max_seq_length=2048,
    load_in_4bit=True,
)
model.save_pretrained_merged("aegis-mini-merged", tokenizer, save_method="merged_16bit")

The base weights are in 4-bit form during training. The merge step changes them back to 16-bit numbers. This step can add small rounding errors.

6. Prompt format

The model uses the chat template of Qwen3. The template uses the ChatML structure.

<|im_start|>system
{system prompt}<|im_end|>
<|im_start|>user
{question}<|im_end|>
<|im_start|>assistant

Call tokenizer.apply_chat_template to build this text. Do not type the tags by hand.

6.1 Recommended system prompt

You are Aegis-mini-1.0, a compact AI assistant for electronics, drones, robotics, math and physics. You are strongest at theory and calculations. Be concise: state the key formula, show the steps with units, then give the result. If you are unsure of an exact value (register address, pin number, part spec), say so and tell the user to check the datasheet instead of guessing. For topics outside your focus, say so briefly.

Keep the system prompt short. A long system prompt uses context and makes a phone slow.

7. Generation settings

Setting	Recommended value	Remark
Temperature	0.3 to 0.7	Use 0.3 for calculations and code. Use 0.7 for explanations.
Top-k	20
Top-p	0.8
Min-p	0
Repetition penalty	1.0 to 1.05	Do not go above 1.1.
Maximum new tokens	512 to 1024	Use a higher value for long derivations.
Context length	2,048 to 4,096	The author trained with 2,048 tokens.

CAUTION Do not set a no-repeat n-gram size. Code and equations repeat groups of tokens on purpose. This setting breaks them.

8. Training data

8.1 Sources

Source	Lines used	License	Remark
Dataset of the author	Not listed in this card	Not published	Mostly theory in electronics, drones, robotics, math, and physics.
HuggingFaceH4/MATH-500	500	MIT	500 problems and worked solutions from the MATH benchmark.
mrohith29/high-school-physics	400	MIT	High-school physics questions and answers.

The author did not publish the dataset of the author. The author can publish it later.

8.2 Data preparation

The author did these steps in order:

1. Convert each row into chat messages with a user role and an assistant role.
2. Add a system prompt to each example when a system prompt was set.
3. Apply the chat template of Qwen3 to make one text for each example.
4. Count the tokens of each example.
5. Remove each example that has more than 2,048 tokens.
6. Shuffle the data with seed 42.
7. Hold out 100 examples for validation.

The result is 2,765 training examples and 100 validation examples.

8.3 Loss mask

The training loss counts only the tokens of the answer. The tokens of the question do not add to the loss. The author used train_on_responses_only from Unsloth for this. The markers are <|im_start|>user and <|im_start|>assistant.

9. Training procedure

9.1 Hardware and software

Item	Value
GPU	NVIDIA Tesla T4 (14.6 GB), Google Colab free tier
Number of GPUs	1
Python	3.13
Main libraries	Unsloth, TRL, Transformers, PEFT, bitsandbytes, PyTorch
Experiment tracking	Weights & Biases 0.28.1
Library versions	The author did not record the exact versions.

9.2 LoRA settings

Setting	Value
Rank (r)	16
Alpha	32
Dropout	0
Bias	none
Target modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Gradient checkpointing	Unsloth method
Random state	3407

9.3 Training settings

Setting	Value
Epochs	3
Total optimizer steps	519
Batch size on each device	2
Gradient accumulation steps	8
Effective batch size	16
Peak learning rate	2e-4
Learning rate schedule	Cosine, to near zero at the end
Warmup	20 steps
Weight decay	0.01
Optimizer	AdamW 8-bit
Precision	Mixed precision, fp16 (the T4 does not support bf16)
Maximum sequence length	2,048 tokens
Seed	3407
Validation	Every 50 steps
Checkpoint	Every 100 steps (the last 2 are kept)

10. Results

10.1 Final numbers

Metric	Value
Final validation loss	1.1404
Perplexity from the final validation loss	about 3.13
Training loss (average of the whole run)	0.8454
Training loss (last logged step)	0.5838
Final gradient norm	0.848
Final learning rate	1.98e-7
Epochs finished	3
Steps finished	519
Training time	4,435 seconds (about 1 hour 14 minutes)
Training speed	1.87 samples each second, 0.117 steps each second
Total floating-point operations	4.37 x 10^16
Validation speed	6.07 samples each second

10.2 Validation loss during training

Step	Training loss	Validation loss
50	1.1866	1.2763
100	1.0245	1.1578
150	0.9912	1.1150
200	0.7534	1.0917
End (step 519)	0.5838 (last step)	1.1404

10.3 How to read these numbers

The validation loss went down from step 50 to step 200. The lowest value that the author recorded is 1.0917 at step 200. The final value is 1.1404, which is a small rise. The gap between the training loss and the validation loss also grew in epoch 3. This can be a sign of a small amount of overfitting. The model can fit the training examples a little too much.

This card lists only the values that the author recorded. The author did not run public benchmarks, such as MATH, GSM8K, or MMLU. Loss is not the same as accuracy. Do not use these numbers as proof of correct answers.

11. Limitations

• Size. The model has 4 billion parameters. It makes more errors than a large model.
• Wrong numbers. The model can make mistakes in arithmetic, in units, and in long derivations.
• Made-up facts. The model can invent exact values, such as register addresses, pin numbers, and part limits. Check each value in the datasheet.
• Theory focus. The training data is mostly theory. The model is weaker for long programs and for real hardware settings.
• No code tests. The author did not compile or run the code that the model writes.
• Language. The model is trained on English text. Other languages can give poor results.
• Reasoning. The training data does not teach long reasoning in thinking mode. Expect short and direct answers.
• Knowledge. The knowledge of the model comes from the base model and the training data. It can be old.
• Safety tuning. The author did not add new safety training. The model keeps the behavior of the base model.
• Data bias. The data is a choice of the author. The topics are not complete. The physics data is at high-school level. The MATH-500 data is at competition level.
• Overfitting. See Section 10.3.

12. Compute used

Item	Value
Hardware	1 NVIDIA Tesla T4
Training time	About 1 hour 14 minutes
Platform	Google Colab free tier

13. Files in this repository

File	Use
adapter_model.safetensors	Weights of the LoRA adapter
adapter_config.json	Settings of the adapter and the name of the base model
tokenizer.json, tokenizer_config.json, and related files	Tokenizer and chat template
README.md	This model card

14. License

The adapter has the license Apache-2.0. The base model Qwen3-4B has the license Apache-2.0. The datasets MATH-500 and high-school-physics have the license MIT. You must follow the terms of each license. Keep the license notices when you share the model.

15. Citations

15.1 How to cite this model

@misc{aegis_mini_1_0_lora_2026,
  title        = {Aegis-mini-1.0 (AEGIS-Mini-V1-LoRA): A LoRA Adapter of Qwen3-4B for Electronics, Drones, Robotics, Math, and Physics},
  author       = {Xerxes-28},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/Xerxes-28/AEGIS-Mini-V1-LoRA}},
  note         = {Hugging Face model repository}
}

15.2 Base model

@article{yang2025qwen3,
  title   = {Qwen3 Technical Report},
  author  = {Yang, An and others},
  journal = {arXiv preprint arXiv:2505.09388},
  year    = {2025},
  url     = {https://arxiv.org/abs/2505.09388}
}

15.3 Datasets

@inproceedings{hendrycks2021math,
  title     = {Measuring Mathematical Problem Solving With the {MATH} Dataset},
  author    = {Hendrycks, Dan and Burns, Collin and Kadavath, Saurav and Arora, Akul and Basart, Steven and Tang, Eric and Song, Dawn and Steinhardt, Jacob},
  booktitle = {Thirty-fifth Conference on Neural Information Processing Systems, Datasets and Benchmarks Track},
  year      = {2021},
  url       = {https://arxiv.org/abs/2103.03874}
}

@inproceedings{lightman2024lets,
  title     = {Let's Verify Step by Step},
  author    = {Lightman, Hunter and Kosaraju, Vineet and Burda, Yura and Edwards, Harri and Baker, Bowen and Lee, Teddy and Leike, Jan and Schulman, John and Sutskever, Ilya and Cobbe, Karl},
  booktitle = {The Twelfth International Conference on Learning Representations},
  year      = {2024},
  url       = {https://arxiv.org/abs/2305.20050}
}

@misc{h4_math500,
  title        = {MATH-500},
  author       = {{Hugging Face H4}},
  howpublished = {\url{https://huggingface.co/datasets/HuggingFaceH4/MATH-500}},
  note         = {Hugging Face dataset. The 500-problem test subset that OpenAI selected in Lightman et al. (2024).}
}

@misc{mrohith29_highschoolphysics,
  title        = {high-school-physics},
  author       = {mrohith29},
  howpublished = {\url{https://huggingface.co/datasets/mrohith29/high-school-physics}},
  note         = {Hugging Face dataset, MIT license}
}

15.4 Methods

@inproceedings{hu2022lora,
  title     = {{LoRA}: Low-Rank Adaptation of Large Language Models},
  author    = {Hu, Edward J. and Shen, Yelong and Wallis, Phillip and Allen-Zhu, Zeyuan and Li, Yuanzhi and Wang, Shean and Wang, Lu and Chen, Weizhu},
  booktitle = {International Conference on Learning Representations},
  year      = {2022},
  url       = {https://arxiv.org/abs/2106.09685}
}

@inproceedings{dettmers2023qlora,
  title     = {{QLoRA}: Efficient Finetuning of Quantized {LLMs}},
  author    = {Dettmers, Tim and Pagnoni, Artidoro and Holtzman, Ari and Zettlemoyer, Luke},
  booktitle = {Advances in Neural Information Processing Systems},
  year      = {2023},
  url       = {https://arxiv.org/abs/2305.14314}
}

@inproceedings{dettmers2022optimizers,
  title     = {8-bit Optimizers via Block-wise Quantization},
  author    = {Dettmers, Tim and Lewis, Mike and Shleifer, Sam and Zettlemoyer, Luke},
  booktitle = {International Conference on Learning Representations},
  year      = {2022},
  url       = {https://arxiv.org/abs/2110.02861}
}

@inproceedings{loshchilov2019adamw,
  title     = {Decoupled Weight Decay Regularization},
  author    = {Loshchilov, Ilya and Hutter, Frank},
  booktitle = {International Conference on Learning Representations},
  year      = {2019},
  url       = {https://arxiv.org/abs/1711.05101}
}

@inproceedings{loshchilov2017sgdr,
  title     = {{SGDR}: Stochastic Gradient Descent with Warm Restarts},
  author    = {Loshchilov, Ilya and Hutter, Frank},
  booktitle = {International Conference on Learning Representations},
  year      = {2017},
  url       = {https://arxiv.org/abs/1608.03983}
}

15.5 Software

@misc{unsloth,
  title        = {Unsloth},
  author       = {Han, Daniel and Han, Michael and {Unsloth team}},
  year         = {2023},
  howpublished = {\url{https://github.com/unslothai/unsloth}}
}

@misc{vonwerra2020trl,
  title        = {{TRL}: Transformer Reinforcement Learning},
  author       = {von Werra, Leandro and Belkada, Younes and Tunstall, Lewis and Beeching, Edward and Thrush, Tristan and Lambert, Nathan and Huang, Shengyi and Rasul, Kashif and Gallou{\'e}dec, Quentin},
  year         = {2020},
  howpublished = {\url{https://github.com/huggingface/trl}}
}

@inproceedings{wolf2020transformers,
  title     = {Transformers: State-of-the-Art Natural Language Processing},
  author    = {Wolf, Thomas and others},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations},
  pages     = {38--45},
  year      = {2020},
  url       = {https://aclanthology.org/2020.emnlp-demos.6}
}

@misc{wandb,
  title        = {Experiment Tracking with Weights and Biases},
  author       = {Biewald, Lukas},
  year         = {2020},
  howpublished = {\url{https://www.wandb.com/}}
}

16. Contact

Open a discussion in the Community tab of this repository. Hugging Face name of the author: Xerxes-28.