Velqora-160M-Instruct

Velqora-160M-Instruct is a 158,622,720-parameter, English-first, decoder-only language model trained from random initialization on a single NVIDIA GeForce RTX 4060 (8 GB) under Windows 11. It uses a Llama-style Transformer design, but it does not reuse Llama weights, a Llama tokenizer, or any other pretrained model.

This is an experimental small model intended for research, learning, local inference, and reproducible evaluation—not a production-grade or reliably factual assistant.

GGUF companion: Maxilicious20/Velqora-160M-Instruct-GGUF (only use a GGUF variant whose conversion and inference have been independently validated).

Model overview

Property	Value
Developer	Maxilicious20
Parameters	158,622,720
Model type	Decoder-only causal language model
Training origin	From scratch, no pretrained foundation weights
Primary language	English
Transformer layers	12
Hidden size	1,024
Attention	16 query heads / 4 key-value heads (GQA)
Feed-forward dimension	2,560, SwiGLU
Position embeddings	RoPE, base θ = 10,000
Normalization	RMSNorm
Token embeddings	Tied input/output embeddings
Tokenizer	Custom 32,000-token byte-level BPE
Maximum context window	2,048 tokens
Implementation	Custom PyTorch, using scaled_dot_product_attention
Training hardware	Windows 11, NVIDIA RTX 4060 8 GB

Implementation note: "Llama-style" describes the broad architecture, not compatibility with Meta Llama checkpoints, Hugging Face LlamaForCausalLM, or native transformers AutoModel loading. This release uses its own tensor names, model class, tokenizer and chat formatting.

Training

Velqora was trained in three successive stages, reusing the same model weights:

Stage	Tokens processed	Content / objective
Language-model pretraining	3,000,000,000	FineWeb-Edu and WikiText-103; next-token prediction
Conversation training	200,000,000	UltraChat 200K conversations
Supervised instruction tuning	15,000,000	Assistant-targeted UltraChat instruction data
Total	3,215,000,000	Processed tokens, not unique training tokens

The original data-preparation targets were approximately 700M packed FineWeb-Edu tokens, 50M WikiText tokens, 80M conversation tokens and 15M instruction tokens before repeat sampling; these are pipeline targets, not a claim that every token was unique after deduplication. Document and conversation samples were reused to reach the processing budgets.

Training and source data references:

• FineWeb-Edu, sample-10BT
• WikiText-103, wikitext-103-raw-v1
• UltraChat 200K, train_gen and train_sft

Evaluation: local results (November 5, 2026)

These numbers come from a local, custom zero-shot evaluation of the original PyTorch checkpoint. For the external benchmarks, the answer with the highest raw continuation likelihood was chosen. These are not official leaderboard results, and the protocol is not claimed to reproduce the standard EleutherAI LM Evaluation Harness settings.

Benchmark	Score	Evaluated / planned	Important caveat
VelqoraBench-400, standard decoding (T=0.40)	0.00%	400 / 400	Very strict generated-answer matching; can reject semantically correct longer responses
ARC-Easy	46.55%	2,376 / 2,376	Local raw-likelihood zero-shot
ARC-Challenge	22.44%	1,172 / 1,172	Local raw-likelihood zero-shot
HellaSwag	28.10%	9,609 / 10,042	Incomplete: 433 tasks missing
BoolQ	58.26%	3,270 / 3,270	Below the ~62.17% always-Yes baseline for this evaluated set
MMLU	23.13%	13,957 / 14,042	Incomplete: 85 tasks missing; strong answer-choice bias observed

Interpretation: The model can produce understandable English and occasionally answer simple factual questions, but its performance is inconsistent. Observed failures include arithmetic errors, factual hallucinations, repetition, incorrect word-sense selection and failure to obey constrained output instructions. The 0% VelqoraBench result is reported as measured, but its strict exact-match scorer is not a robust measure of semantic answer correctness. Do not compare these custom scores directly against published MMLU/ARC/HellaSwag leaderboard figures or scores from a different evaluation protocol.

If you publish a revised, fully completed benchmark, please version the evaluation results rather than silently overwriting the protocol and counts.

Download and run (custom PyTorch implementation)

The normal export is not a native transformers checkpoint. In particular, do not assume that the following works:

# NOT SUPPORTED BY THIS EXPORT
# AutoModelForCausalLM.from_pretrained("Maxilicious20/Velqora-160M-Instruct")

The repository should contain:

README.md
model.pt                      # Final instructed PyTorch state_dict
config.json                   # Velqora ModelConfig, not HF PretrainedConfig
tokenizer.json                # Custom byte-level BPE tokenizer
generation_config.json       # Custom generation defaults
training_metadata.json       # Export metadata
run_inference.py             # Independent CLI example
requirements-inference.txt
velqora/
  __init__.py
  model.py                    # Custom decoder class
  tokenization.py             # Exact prompt formatting and decoding
assets/
  velqora-logo.png
  velqorabench-160m-scores.jpg

Install the dependencies in a Python environment with a compatible PyTorch build:

pip install torch tokenizers huggingface_hub

Download the complete model repository and run the bundled CLI:

hf download Maxilicious20/Velqora-160M-Instruct --local-dir ./Velqora-160M-Instruct
python ./Velqora-160M-Instruct/run_inference.py --model-dir ./Velqora-160M-Instruct --prompt "What is the capital of France?" --temperature 0.4 --max-new-tokens 128

Or start interactive, independent (stateless) prompts:

python ./Velqora-160M-Instruct/run_inference.py --model-dir ./Velqora-160M-Instruct

The example launcher above is provided as source code. It has not been end-to-end tested here against the actual user-trained model weights, because those weights were not provided in this release-preparation session. Verify the uploaded export files and run a smoke test before calling the repository operational.

Chat serialization

The model's training uses its own role format:

<|im_start|>user
Hello!<|im_end|><|im_start|>assistant

Assistant text is normally terminated with <|eot_id|>. The special tokens also include <|unk|>, <|pad|>, <|bos|> and <|eos|>. The tokenizer must be the matching Velqora tokenizer: do not replace it with a Meta Llama tokenizer or use an unrelated chat template.

For generation, a reasonable experimental starting point is temperature 0.30–0.55 (for example, 0.40), top-p 0.90, and up to 128 new tokens. These are practical test settings, not an established optimal configuration.

Intended use and limitations

Appropriate use: research on small, from-scratch language models; learning about PyTorch model training; local English text generation; demonstrations and comparisons with appropriately matched evaluation methods.

Not recommended: high-stakes decisions, medical/legal/financial guidance, production factual QA, precise arithmetic, autonomous use, or applications requiring dependable safety or instruction following. Outputs may be wrong, biased, repetitive or inappropriate. Independent verification is essential.

The training sources may contain inaccuracies, biases, copyrighted text and personal information. The underlying data were not comprehensively audited for all such issues. Do not infer that the model is safe for all deployment settings.

License and redistribution

A model-weights license has not been confirmed in the materials available for this card. A license is therefore intentionally not declared in the YAML metadata. The publisher should select and add an accurate weights license only after reviewing their distribution rights and obligations from the training data and any included source code. Do not assume Apache-2.0 or other open licensing merely because the model was trained from scratch.

Citation and reproducibility

Model: Maxilicious20/Velqora-160M-Instruct
GGUF companion: Maxilicious20/Velqora-160M-Instruct-GGUF

For a reproducible release, preserve the final exported weights, exact tokenizer, configuration, the source-code revision, evaluation script and question-set version. The measured scores above should be associated with the original PyTorch weights, not automatically transferred to any quantized variant.