Instructions to use anicka/nla-qwen3-4b-universal-av-grpo with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use anicka/nla-qwen3-4b-universal-av-grpo with PEFT:

from peft import PeftModel
from transformers import AutoModelForCausalLM

base_model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-4B")
model = PeftModel.from_pretrained(base_model, "anicka/nla-qwen3-4b-universal-av-grpo")

Transformers

How to use anicka/nla-qwen3-4b-universal-av-grpo with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="anicka/nla-qwen3-4b-universal-av-grpo")

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("anicka/nla-qwen3-4b-universal-av-grpo", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use anicka/nla-qwen3-4b-universal-av-grpo with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "anicka/nla-qwen3-4b-universal-av-grpo"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "anicka/nla-qwen3-4b-universal-av-grpo",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/anicka/nla-qwen3-4b-universal-av-grpo

SGLang

How to use anicka/nla-qwen3-4b-universal-av-grpo with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "anicka/nla-qwen3-4b-universal-av-grpo" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "anicka/nla-qwen3-4b-universal-av-grpo",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "anicka/nla-qwen3-4b-universal-av-grpo" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "anicka/nla-qwen3-4b-universal-av-grpo",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use anicka/nla-qwen3-4b-universal-av-grpo with Docker Model Runner:
```
docker model run hf.co/anicka/nla-qwen3-4b-universal-av-grpo
```

NLA Activation Verbalizer — Qwen3 4B, universal, GRPO-refined

The universal SFT verbalizer sharpened with AR-native GRPO: the reward is whether an independent Activation Reconstructor can rebuild the original activation from the description alone. No semantic similarity model anywhere in the reward — faithfulness is scored in the model's own 2560-dim activation space. Smallest model in the series to date (4B), and the one that gains the most from GRPO (+31% relative round-trip vs +24% for the 7B sibling).

Part of the nla-at-home project.

What GRPO changed (eyeball version)

SFT describes the genre of the activation; GRPO quotes its content. Same activation (math text, L18):

SFT: "False" as a negation marker active, with "True" as a competing attractor · "logarithmic" and "exponential" as a contrastive pair (round-trip cos 0.69)

GRPO: False statement: "The statement is false" (direct negation of the claim) · Logical structure: the quadratic equation x² + 4x + 4 = 0 has a repeated root at x = −2, which is a real number (round-trip cos 0.90)

Reward design (the recipe that worked)

reward = centered_cos(AR(desc), act) × specificity(‖AR(desc) − μ_L‖)

centered cosine: per-layer mean removed from both sides — raw cosine is ~90% shared layer offset and rewards nothing.
specificity: a generic description reconstructs to a small-norm centered vector; multiplying by the reconstruction norm makes genericness unprofitable in activation space, with no contrastive hacks.
no hard negatives, no MiniLM in the gradient, KL anchor (β 0.01) to the base model without the adapter, repetition penalty 0.2 (the SFT stage's repetition loops were an explicit target).
curriculum, not reward: an oracle compass (per-layer ridge map activation→text embedding, fit fresh on this model's AR, val top-1 0.59–0.75) selects which (layer, text) pairs are decodable enough to train on, tau 0.40→0.10. The compass never touches the reward or the eval.

Trained on 9 of 36 layers (0, 4, 9, 13, 18, 22, 26, 31, 35), group size 6, 400 samples/epoch, lr 5e-6.

Two-stage history (full provenance ships with the repo)

The original 8-epoch run was interrupted mid-epoch-4 (best: epoch 3, reward 0.5302, tau at 0.27). Training resumed from that checkpoint as a fresh GRPO run with the tau schedule continuing 0.27→0.10 over 5 epochs; best checkpoint is continuation epoch 3 (reward 0.5330). The KL anchor is the adapter-free base model in both stages, so the two stages are regularization-compatible. Curriculum consumption logs for both stages ship as trained_samples_stage1.jsonl + trained_samples.jsonl, and the eval holdout below excludes texts touched by either stage.

Evaluation — round-trip on a clean holdout

284 texts that none of AV training, AR training, or either GRPO stage ever touched (eval_holdout_ids.json = the SFT validation split minus every text either curriculum consumed). Descriptions generated greedy, reconstructed by the frozen AR, centered cosine:

layer (depth)	GRPO	SFT	Δ
L0 (4%)	0.088	0.014	+0.074
L4 (10%)	0.292	0.150	+0.142
L9 (25%)	0.489	0.373	+0.116
L13 (40%)	0.535	0.419	+0.116
L18 (47%)	0.709	0.575	+0.134
L22 (63%)	0.684	0.546	+0.138
L26 (71%)	0.706	0.581	+0.125
L31 (90%)	0.694	0.545	+0.149
L35 (96%)	0.370	0.271	+0.099
mean	0.508	0.386	+0.122

GRPO wins at every depth. The extreme layers (L0, L35) stay weak for both — a property of the 4B substrate, not the training stage.

The confabulation tail

Fraction of confidently-wrong descriptions (low round-trip cosine), all nine layers / content band only (L4–L31):

	cos < 0.3	cos < 0.2	cos < 0
GRPO (all)	21.8%	14.0%	4.7%
SFT (all)	39.4%	28.1%	10.9%
GRPO (L4–L31)	12.7%	6.9%	0.6%
SFT (L4–L31)	29.0%	18.3%	6.4%

In the content band the sign-flipped tail (cos < 0) drops 10×. Raw per-item data: roundtrip_eval.records.jsonl.

Injection protocol

Identical to the SFT card: ㈎ (id 149705), normalize TO L2 norm 150.0, depth-conditioned prompt (nearest_depth_pct(L, 36)), chat template with enable_thinking=False. GRPO-sharpened adapters are numerically touchy: use CUDA (bf16/fp16) or CPU fp32 — not MPS bf16.

Limitations

Same corpus scope as the SFT stage (safety-filtered public split). GRPO improves content specificity; it does not add knowledge of activation regions the corpus never covered. Near-input (L0) and final-layer (L35) readouts remain unreliable at this model scale.

Downloads last month: 13

Model tree for anicka/nla-qwen3-4b-universal-av-grpo

Base model

Qwen/Qwen3-4B-Base

Finetuned

Qwen/Qwen3-4B

Adapter

(1115)

this model

Collection including anicka/nla-qwen3-4b-universal-av-grpo

NLA — Qwen3 4B universal

Collection

Smallest universal NLA trio (36 layers, one adapter per role). Biggest relative GRPO gain in the series (+31%). • 3 items • Updated about 1 month ago