Spexcon S1

Spexcon S1 is the first language model project in the Spexcon family. This repository contains the v0.3 open-source development scaffold for building, fine-tuning, smoke testing, and evaluating Spexcon S1. It does not contain final Spexcon S1 weights yet.

The project is designed to support future Spexcon variants such as Spexcon S1 Mini, Spexcon S1 Code, Spexcon S1 Reason, and later Spexcon S2 models by keeping model choice, dataset formatting, LoRA settings, training arguments, and Hub upload settings in configuration.

Development Status

Status: v0.3 GPU QLoRA smoke test completed successfully on Kaggle with a Tesla T4. The resulting adapter is an experimental 300-example smoke-test adapter, not a finished Spexcon S1 release.

This release includes:

A professional Hugging Face model card.
A configurable supervised fine-tuning pipeline.
A streaming dataset preparation script for a small instruction/chat subset.
LoRA/QLoRA settings suitable for limited/free GPU environments.
A fixed-prompt qualitative evaluation script.
An example chat/inference script.
A v0.3 smoke-test runner that records measured hardware, loss, adapter reload, and fixed-prompt outputs when run on CUDA.
A verified experimental v0.3 LoRA adapter published separately at YONKWd/Spexcon-S1-v0.3-Experimental.
No full training run and no final trained model weights.

Base Model

Spexcon S1 v0.2 is configured to start from Qwen/Qwen3-1.7B-Base.

Why this base model:

It is a Qwen-family open-weight base model.
It has approximately 1.72B parameters, which is realistic for LoRA/QLoRA fine-tuning on free or limited GPUs.
Its Hugging Face metadata lists the license as Apache-2.0, which is compatible with open and commercial use.
It provides a cleaner foundation for Spexcon-specific instruction tuning than starting from a much larger checkpoint.

Important distinction:

Base model: Qwen/Qwen3-1.7B-Base.
Experimental adapter: YONKWd/Spexcon-S1-v0.3-Experimental, a clearly labeled 300-example smoke-test adapter.
Full Spexcon S1 release: does not exist yet.

v0.3 Smoke Test Result

The first real GPU QLoRA smoke test completed on Kaggle using one Tesla T4 from a T4 x2 session. The run used 300 UltraChat training examples and 50 evaluation examples, sequence length 512, LoRA rank 16, LoRA alpha 32, and QLoRA 4-bit loading.

Measured result summary:

Training completed: yes, 10 optimizer steps.
Peak GPU memory allocated: 3.32 GB.
Recorded training loss: 1.796299934387207.
Adapter reload: succeeded.
Fixed-prompt qualitative evaluation: completed for the base model and the experimental adapter.

The qualitative outputs show that the pipeline works, but the adapter is not a quality release. Some generations still repeat role text or switch language, which should be addressed with better formatting checks, cleaner datasets, and a larger v0.4 training run.

Intended Use

Spexcon S1 is intended for research, experimentation, and future instruction tuning of compact assistant-style language models. Potential target behaviors include:

General instruction following.
Writing and editing assistance.
Reasoning-style responses after suitable supervised fine-tuning.
Domain-specialized Spexcon variants trained on carefully licensed datasets.

This v0.2 repository is intended for development and reproducibility. It is not a finished assistant model.

Limitations

No full Spexcon S1 training has been run yet.
The repository does not currently include final model weights.
The v0.3 adapter is only a smoke-test artifact and should be treated as experimental.
The configured base model is not a safety-certified system.
Outputs may be inaccurate, biased, incomplete, or unsafe unless future training and evaluation address those risks.
The v0.3 qualitative evaluation found rough outputs, including repetition and occasional language switching.
Downstream users must validate dataset licenses, data quality, privacy constraints, and evaluation results before releasing trained checkpoints.

Training Method

The included training pipeline uses:

Python
Hugging Face Transformers
Hugging Face Datasets
PEFT
TRL SFTTrainer
LoRA or QLoRA
4-bit bitsandbytes loading for memory-efficient training

The default method is supervised fine-tuning with QLoRA. The script saves PEFT adapter checkpoints, which can later be merged into the base model if desired.

Dataset Pipeline

The v0.2 dataset pipeline uses HuggingFaceH4/ultrachat_200k as the first source dataset.

Dataset license: MIT, as listed on the Hugging Face dataset card.

Why this dataset:

It is English instruction/chat data.
It includes SFT-ready messages records.
It has documented train_sft and test_sft splits.
It is widely used for assistant-style supervised fine-tuning experiments.
The preparation script streams a small subset, so the full dataset is not committed or downloaded unnecessarily.

The preparation pipeline adds:

Empty/broken example filtering.
Role normalization.
Basic duplicate removal.
Deterministic shuffling and train/eval split.
Configurable sample limit and seed.
Dataset statistics in stats.json.

Dataset Format

The trainer accepts several common supervised fine-tuning formats:

messages: a chat-style list of role/content dictionaries.
text: a fully formatted training string.
prompt plus response, completion, or answer.
instruction plus output or response.

For production Spexcon S1 training, prefer a curated dataset with:

Clear system/user/assistant message turns.
Permissive licensing.
Deduplicated examples.
No private or sensitive data.
Separate train/validation/test splits.
Evaluation prompts that are held out from training.

Quick Start

Install dependencies in a fresh GPU notebook or virtual environment:

pip install -r training/requirements.txt

Prepare the v0.2 small dataset subset:

python training/prepare_dataset.py --config training/config.yaml

Validate configuration and prepared dataset loading without training:

python training/train.py --config training/config.yaml --dry-run

Run a tiny QLoRA smoke test with the training entry point:

python training/train.py --config training/config.yaml --smoke-test

Or prepare a tiny 500-example smoke subset and run the smoke test in one command:

python training/train.py --config training/config.yaml --prepare-dataset --smoke-test

Run the full v0.3 smoke-test verification script on a CUDA GPU:

python training/run_v03_smoke.py --config training/config.yaml --push-adapter

Current v0.3 status: the Kaggle T4 QLoRA smoke test completed, the adapter reloaded successfully, fixed-prompt evaluation ran, and the experimental adapter was uploaded to YONKWd/Spexcon-S1-v0.3-Experimental. See evaluation/reports/v0.3-smoke-test.md.

Run an interactive chat against the base model:

python examples/chat.py --model Qwen/Qwen3-1.7B-Base

After an experimental LoRA adapter has been trained and clearly labeled:

python examples/chat.py --model Qwen/Qwen3-1.7B-Base --adapter YONKWd/Spexcon-S1

To try the v0.3 experimental adapter explicitly:

python examples/chat.py --model Qwen/Qwen3-1.7B-Base --adapter YONKWd/Spexcon-S1-v0.3-Experimental

Run the fixed-prompt comparison evaluator:

python evaluation/evaluate.py --config training/config.yaml --adapter path-or-repo-to-adapter

Using Spexcon S1 in an App

The server/ module exposes the v0.3 experimental adapter through a small FastAPI backend with an OpenAI-compatible /v1/chat/completions endpoint. This keeps inference separate from frontends, so web, desktop, and mobile apps can all call the same Spexcon API.

Install the server dependencies:

pip install -r server/requirements.txt

Start the local API:

uvicorn server.app:app --host 0.0.0.0 --port 8000

By default, the server serves:

Base model: Qwen/Qwen3-1.7B-Base
Adapter: YONKWd/Spexcon-S1-v0.3-Experimental
Model alias: spexcon-s1

The model loads lazily on the first chat request. On CUDA, SPEXCON_QUANTIZATION=auto uses 4-bit bitsandbytes loading to reduce VRAM use. Hugging Face tokens are read only from the server environment, such as HF_TOKEN, and are never sent to client apps.

Health and model list:

curl http://localhost:8000/health
curl http://localhost:8000/v1/models

JavaScript app example using fetch():

const response = await fetch("http://localhost:8000/v1/chat/completions", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({
    model: "spexcon-s1",
    messages: [{ role: "user", content: "Hello Spexcon" }],
    temperature: 0.7,
    max_tokens: 128,
  }),
});

const data = await response.json();
console.log(data.choices[0].message.content);

OpenAI JavaScript SDK example:

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.SPEXCON_API_KEY ?? "local-dev-key",
  baseURL: "http://localhost:8000/v1",
});

const completion = await client.chat.completions.create({
  model: "spexcon-s1",
  messages: [{ role: "user", content: "Hello Spexcon" }],
});

console.log(completion.choices[0].message.content);

For hosted deployment, run this backend on a GPU-capable container or VM, set SPEXCON_API_KEY for client authentication, keep HF_TOKEN server-side only if needed, and put any web/mobile frontend in a separate app that calls the API. This repository does not configure paid hosting automatically.

Recommended Hardware

For initial Spexcon S1 QLoRA experiments:

Minimum practical: NVIDIA T4 16 GB with short sequence length and batch size 1.
Recommended free/low-cost: L4 24 GB or A10G 24 GB.
Expected VRAM: roughly 8-12 GB for conservative QLoRA settings at 1024-token sequences; 12-18 GB is more comfortable for longer contexts, evaluation, or larger batches.

License

This project scaffold is released under Apache License 2.0. The selected base model, Qwen/Qwen3-1.7B-Base, is listed on Hugging Face with the Apache-2.0 license.

Future trained Spexcon checkpoints should only be released after confirming that the training datasets and any generated or synthetic data are compatible with Apache-2.0 distribution.

Citation and Attribution

Base model:

Qwen/Qwen3-1.7B-Base
https://huggingface.co/Qwen/Qwen3-1.7B-Base
License: Apache-2.0

Downloads last month: -; Downloads are not tracked for this model. How to track