VertexAGI/amethyst-1-small

Amethyst 1 Small

Amethyst 1 Small is a general-purpose chat and instruction-following model, fine-tuned from Llama 3.1 8B Instruct using LoRA on the same distilled instruction dataset used for Amethyst 1 Mini (which is based on Gemma 3 4B). It's the second, larger model in the Amethyst family — same data, bigger base model.

Model Details

Developed by	Independent research project
Base model	meta-llama/Llama-3.1-8B-Instruct
Fine-tuning base checkpoint	mlx-community/Meta-Llama-3.1-8B-Instruct-4bit
Architecture	Llama 3.1, 8B parameters (dense, decoder-only transformer)
Fine-tuning method	LoRA (rank 8, scale 20.0), fused into the base weights — released 4-bit quantized (same quantization as the base checkpoint), not dequantized
Fine-tuning framework	MLX / mlx-lm, on Apple Silicon
Trained modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj across 16 layers
Language	English
License	Llama 3.1 Community License

Training Data

Same 1,122 instruction/response pairs (1,082 train / 40 validation) used for Amethyst 1 Mini — synthetically generated via knowledge distillation from nvidia/nemotron-3-super-120b-a12b (Nemotron-3-Super, a 120B-parameter MoE model, ~12B active) through the OpenRouter API, spanning explanation, reasoning, code, extraction, planning, roleplay, creative writing, translation, sentiment classification, and brainstorming.

Training Procedure

• Method: Supervised fine-tuning via LoRA (rank 8, dropout 0.0, scale 20.0)
• Optimizer: Adam, learning rate 1e-5 (constant schedule)
• Sequence length: 4096 tokens
• Gradient checkpointing: enabled
• Training: trained in two passes (interrupted partway by a local machine restart and resumed from an intermediate checkpoint), together substantially exceeding the original 3,246-iteration budget
• Checkpoint selection: the released weights use the checkpoint with the lowest observed validation loss (1.487) — later checkpoints in this small, repetitive dataset oscillate around a noise floor rather than improving further, matching the pattern seen with Amethyst 1 Mini

Intended Use

Amethyst 1 Small is intended as a general-purpose conversational assistant for experimentation and research into small-scale distillation pipelines. It is not intended for high-stakes, safety-critical, or production use.

Limitations

• Trained on the same small (1,122-example) synthetic dataset as Amethyst 1 Mini — behavior can be inconsistent outside the categories represented in training.
• Distilled from a single teacher model without human review of every example.
• Inherits the general limitations and knowledge cutoff of its base model, Llama 3.1 8B Instruct.
• An early, second-generation checkpoint in the Amethyst series.

Usage

from mlx_lm import load, generate
from mlx_lm.sample_utils import make_sampler

model, tokenizer = load("VertexAIco/amethyst-1-small")

messages = [{"role": "user", "content": "Explain how vaccines train the immune system, in simple terms."}]
prompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True)
out = generate(model, tokenizer, prompt=prompt, max_tokens=512, sampler=make_sampler(temp=0.0))
print(out)

Citation

@misc{amethyst1small,
  title  = {Amethyst 1 Small},
  author = {Independent research project},
  year   = {2026},
  note   = {LoRA fine-tune of Llama 3.1 8B Instruct, distilled from Nemotron-3-Super-120B-A12B}
}

This model is built on Llama 3.1 and subject to the Llama 3.1 Community License.