StarpowerTechnology/Liquid-WVY-TinyLM

WVY-Experimental

WVY-Experimental is a complete architecture and training kit for creating a new LFM2-style causal language model from scratch. It contains no pretrained model weights. You bring a text dataset, select or train a tokenizer, scan the available tokens, generate a model configuration, and start training with randomly initialized parameters.

The model follows the public LFM2 implementation in Hugging Face Transformers: gated depthwise short-convolution layers are mixed with grouped-query causal-attention layers, and every decoder layer includes RMS normalization, residual connections, and a SwiGLU feed-forward network. The exact upstream architecture files are included under reference/transformers-lfm2/ for inspection and attribution. Runtime construction uses the maintained transformers.Lfm2ForCausalLM implementation.

What is included

Path	Purpose
src/wvy_experimental/	Data loading, tokenizer creation, scanning, config generation, and training
configs/	Ready configurations covering roughly 10M, 25M, and 50M targets
assets/tokenizer/	Public LFM2.5 tokenizer assets for compatibility experiments
notebooks/WVY_Colab_Training_Kit.ipynb	Guided Google Colab workflow
notebooks/WVY_Kaggle_Training_Kit.ipynb	Guided Kaggle workflow
reference/transformers-lfm2/	Byte-for-byte upstream LFM2 architecture source
scripts/	Local setup and one-command preparation helpers
huggingface/model-card-template.md	Model card template for a trained checkpoint

Before training

The bundled Liquid tokenizer contains 65,536 tokens. Its embedding table can consume much of a 1M–50M parameter budget. For compact models, train a smaller tokenizer from your own corpus. An 8,192-token vocabulary is the default starting point.

The scanner's default 20 tokens per parameter is a configurable planning baseline. Liquid AI has not published a validated optimal scaling ratio for LFM2 models in the 1M–50M range. Treat the suggested target as a starting point and compare several sizes if compute permits.

Supported data

The kit accepts a file or directory and searches recursively for:

• .txt and .md files
• .json and .jsonl records
• .csv tables
• .parquet tables

Structured files must contain a text field by default. Pass --text-field content when your column uses another name.

Local setup

Python 3.10 or newer and a CUDA-capable GPU are recommended. CPU training works for smoke tests but will be slow.

cd WVY-Experimental
python -m venv .venv
source .venv/bin/activate          # Windows: .venv\Scripts\activate
python -m pip install --upgrade pip
pip install -e .

Place the dataset anywhere you want. The examples below assume data/.

1. Train a compact tokenizer

wvy-tokenizer \
  --data data/ \
  --vocab-size 8192 \
  --output artifacts/tokenizer

You may skip this and use assets/tokenizer, though its 65,536-token vocabulary makes very small parameter targets impractical.

2. Scan and count the data

wvy-scan \
  --data data/ \
  --tokenizer artifacts/tokenizer \
  --tokens-per-parameter 20 \
  --output artifacts/data_report.json

The report includes file count, document count, exact tokenizer token count, document-length statistics, tokenizer vocabulary size, and a suggested parameter target clamped to 1M–50M.

3. Generate a model configuration

wvy-build \
  --report artifacts/data_report.json \
  --tokenizer artifacts/tokenizer \
  --output artifacts/model_config.json

The builder searches full hybrid LFM2 configurations and instantiates each candidate to obtain the real parameter count. It writes the closest configuration it finds. Attention layers remain approximately 37.5% of the stack, following the 6-of-16 pattern in the public LFM2-350M configuration.

4. Train fresh weights

wvy-train \
  --data data/ \
  --tokenizer artifacts/tokenizer \
  --config artifacts/model_config.json \
  --output outputs/wvy-model \
  --sequence-length 1024 \
  --batch-size 4 \
  --gradient-accumulation 8 \
  --learning-rate 3e-4 \
  --epochs 1 \
  --bf16 \
  --gradient-checkpointing

Use --fp16 instead of --bf16 on GPUs without BF16 support. Remove both flags for FP32. Resume an interrupted run with:

wvy-train [same options] --resume-from-checkpoint outputs/wvy-model/checkpoint-500

The final Hugging Face checkpoint is written to outputs/wvy-model/final/. It contains the trained weights, model configuration, and tokenizer required by AutoModelForCausalLM.from_pretrained().

For multi-GPU training, create an Accelerate configuration once and launch the same module through Accelerate:

accelerate config
accelerate launch -m wvy_experimental.train \
  --data data/ \
  --tokenizer artifacts/tokenizer \
  --config artifacts/model_config.json \
  --output outputs/wvy-model \
  --bf16 --gradient-checkpointing

Google Colab

Open notebooks/WVY_Colab_Training_Kit.ipynb in Colab and select a GPU runtime. Run the cells from top to bottom. The notebook uploads this kit and your dataset, installs the package, trains a tokenizer, scans the corpus, generates the configuration, launches training, and downloads the final checkpoint.

For longer runs, mount Google Drive and change OUTPUT_DIR to a Drive path so checkpoints survive runtime resets.

Kaggle

Create a Kaggle notebook with a GPU accelerator. Add the zipped WVY-Experimental kit and your corpus as Kaggle datasets, then open notebooks/WVY_Kaggle_Training_Kit.ipynb. Update the two input paths in its setup cell and run all cells. Outputs and checkpoints are written under /kaggle/working/ and can be saved as a notebook version.

Kaggle internet access is only needed to install missing dependencies. The tokenizer and architecture reference are bundled.

Use a prepared configuration

The files under configs/ are starting configurations. Always check their actual parameter count after installation because vocabulary size and configuration fields determine the final number:

from wvy_experimental import create_fresh_model, load_wvy_config

config = load_wvy_config("configs/wvy_25m.json")
model = create_fresh_model(config)
print(sum(p.numel() for p in model.parameters()))

This constructs random weights. It does not download a pretrained checkpoint.

Publish a trained model to Hugging Face

After training, copy huggingface/model-card-template.md into outputs/wvy-model/final/README.md, fill in the dataset and training details, then upload the contents of final/ to a new Hugging Face model repository. Upload the trained output rather than presenting this architecture kit as a pretrained model.

Architecture and training limits

The architecture is complete and configurable; model scale comes from configuration values rather than a reduced demonstration implementation. Training quality still depends on the corpus, tokenizer, optimization settings, hardware, and number of tokens processed. Liquid AI's published models also used private data and teacher-logit knowledge distillation. Those materials are not included in the public release.

Sources and licenses

The architecture implementation under reference/transformers-lfm2/ comes from Hugging Face Transformers and is licensed under Apache License 2.0. The bundled tokenizer and related model-repository assets come from LiquidAI/LFM2.5-350M under the included LFM Open License. See THIRD_PARTY_NOTICES.md and the license files before redistribution or commercial use.