Focus-v1-Base

A lightweight pretrained language model by BRSX Labs.

Focus-v1-Base is a compact causal language model pretrained on 6 billion tokens. Designed with a lightweight architecture, it provides an accessible foundation for experimentation, continued pretraining, and supervised fine-tuning (SFT).

Its relatively small parameter count makes it suitable for researchers and developers who want to experiment with language models without requiring the resources typically associated with larger models.

Focus-v1-Base is a pretrained base model, not an instruction-tuned chatbot.

Model Specifications

Property	Value
Architecture	Decoder-only Transformer
Parameters	Approximately 100M
Pretraining	6B tokens
Hidden size	768
Transformer layers	12
Query attention heads	12
Key/value attention heads	4
Intermediate size	2048
Context length	2048 tokens
Vocabulary size	32,000
Positional encoding	RoPE
Normalization	RMSNorm
Activation	SwiGLU
Embeddings	Tied input and output embeddings

Why Focus-v1-Base?

• Lightweight: A compact architecture designed for accessible experimentation.
• 6B-token pretraining: Trained with an autoregressive next-token prediction objective over 6 billion tokens.
• Fine-tuning friendly: Its relatively small size makes supervised fine-tuning and domain adaptation more accessible than with many larger models.
• Research-oriented: Suitable for experimenting with training strategies, model adaptation, and text generation.

Intended Use

Focus-v1-Base is intended for:

• Supervised fine-tuning (SFT).
• Domain-specific adaptation.
• Autoregressive text generation and continuation.
• Small language model research and experimentation.
• Educational exploration of Transformer architectures.

Usage

Focus-v1-Base uses a custom PyTorch inference implementation.

Text Generation

python chat.py -p "Once upon a time" -n 100

Interactive Mode

python chat.py

Custom Generation Settings

python chat.py --temp 0.8 --topk 50 -n 200

Fine-Tuning

Focus-v1-Base is intended to serve as a starting checkpoint for supervised fine-tuning and specialized language model development.

Its compact architecture can reduce the computational and memory requirements of experimentation compared with larger models. Actual requirements depend on the training setup, sequence length, batch size, precision, and optimizer.

Fine-tuning with carefully curated, high-quality examples may help adapt the model to specific tasks and improve instruction-following behavior.

Limitations

Focus-v1-Base has not been instruction-tuned. It is designed to predict and continue text rather than reliably answer questions or follow instructions.

Generated output may contain repetition, factual errors, grammatical mistakes, or incoherent passages. A large pretraining token count does not by itself guarantee strong benchmark performance or reasoning ability.

Training Details

• Pretraining tokens: 6 billion
• Training objective: Autoregressive next-token prediction
• Architecture: Decoder-only Transformer
• Attention: Grouped-query attention (GQA)
• Positional encoding: Rotary positional embeddings (RoPE)
• Normalization: RMSNorm
• Feed-forward network: SwiGLU

Additional dataset details, hardware information, and reproducible evaluation results can be added as they are verified.

Repository Contents

• focus_mind.pt — Model checkpoint.
• config.json — Model configuration.
• tokenizer.json — Tokenizer.
• chat.py — Inference script.

Developer

BRSX Labs

Website: https://brsxlabs.com

---

Focus-v1-Base is a foundation for further experimentation and fine-tuning. Its compact size makes it an accessible starting point for developers interested in building specialized language models.