brivangl/qwenar-0.6b-montevideo

💙 This checkpoint was trained as part of the work of the Montevideo team at JetBrains. The author thanks JetBrains for the compute that made it possible.

A sentence is compressed into one 1024-dimensional vector, and that vector alone is decoded back into the sentence.

This is the general-domain successor of brivangl/qwenar-0.6b: the same construction, trained on a 9.2-billion-sentence slice of SlimPajama instead of a biomedical mix. On held-out SlimPajama sentences it reconstructs 98.6% character-for-character, and 27 of 32 stress probes.

The idea is SONAR's and the Large Concept Model's: a sentence embedding lossless enough to decode, so the vector can stand in for the text. The construction is different — instead of training a seq2seq model from scratch, qwenar connects two off-the-shelf open checkpoints with a single learned linear bridge and adapts both with DoRA. The whole learned connector is about 30 lines of code.

Code, training pipeline and full evaluation: https://github.com/IvanDrokin/QwenAR

Requires transformers>=5.15, and older versions fail silently. Under transformers 4.x the model raises nothing and produces fluent, plausible text that is simply wrong (see the brivangl/qwenar-0.6b card for the numbers).

Usage

from qwenar.pipeline import Qwenar          # pip install qwenar

qw = Qwenar.from_pretrained("brivangl/qwenar-0.6b-montevideo")

vecs = qw.encode(["The city council approved the transit plan after two hours of debate."])
print(vecs.shape)                            # torch.Size([1, 1024])
print(qw.decode(vecs))                       # back to the sentence
print(qw.roundtrip(["..."]))                 # both at once

Or as a plain transformers model, with no extra dependency:

from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained("brivangl/qwenar-0.6b-montevideo", trust_remote_code=True)
enc_tok = AutoTokenizer.from_pretrained("brivangl/qwenar-0.6b-montevideo", subfolder="encoder_tokenizer")
dec_tok = AutoTokenizer.from_pretrained("brivangl/qwenar-0.6b-montevideo")

batch = enc_tok(["The city council approved the transit plan after two hours of debate."],
                return_tensors="pt", padding=True)
emb = model.encode(batch["input_ids"], batch["attention_mask"])     # (1, 1024)
ids = model.generate_from_embeddings(emb, max_new_tokens=64, do_sample=False)
print(dec_tok.batch_decode(ids, skip_special_tokens=True))

Two tokenizers, because the encoder and the decoder come from different checkpoints. The decoder tokenizer sits at the repository root so plain AutoTokenizer.from_pretrained(repo) gives you the one generation needs; the encoder tokenizer is in encoder_tokenizer/. Decoder targets must be right-padded — Qwenar enforces this.

Architecture

sentence ──► perplexity-ai/pplx-embed-v1-0.6b
             Qwen3 with the causal mask DISABLED → bidirectional
             mean-pool over non-pad tokens; no tanh, no INT8
                              │
                       vec (1024,)             ◄── this is the artifact
                              │
             EmbedToPrefix:  Linear(1024 → K·d_model) → GELU
                             → view(B, K, d_model) → RMSNorm      K = 2
                              │
                     prefix (B, 2, d_model)
                              │
             Qwen/Qwen3-0.6B-Base + DoRA(r=32, α=64) on q,k,v,o,gate,up,down
                              │
                  teacher-forced next-token cross-entropy
                  labels = -100 on the prefix and on padding

1194M parameters total. The encoder is adapted too (DoRA r=16, α=32, at a 10× lower learning rate), not frozen.

Training data

Sentence-level English from SlimPajama: 9,169,225,926 sentences (199.2B Qwen3 tokens) segmented from the deduplicated corpus, kept between 5 and 256 tokens, exact-deduplicated, and repacked by sentence length for training. Mean length 21.7 tokens, median 19, 95th percentile 45. Roughly 0.6% of sentences are code (rule-based estimate on the validation split) — code is in the data, but rare.

sentence length (tokens)	share
0-15	37.4%
16-31	42.8%
32-47	16.2%
48-63	3.0%
64-79	0.4%
80-95	0.1%

The run saw about 14% of the corpus — ≈1.28B sentences, ≈27.9B supervised tokens; it never completed a pass over the data. The corpus preparation pipeline (export, dedup, length repack) is in the repository under dataprep/.

Training procedure

Optimizer steps	600,000
Supervised tokens	≈27.9B (14% of the corpus)
Hardware	8×H100, bf16
Batching	token budget (5,888 tokens/batch, ≤1024 sentences), length-homogeneous batches
Adapters	DoRA — encoder r=16/α=32, decoder r=32/α=64
ReLoRA	merge + reinit every 100,001 steps (5 cycles)
Learning rates	bridge 1e-4, decoder LoRA 1e-4, encoder 1e-5, cosine, 500 warmup

ReLoRA periodically merges the adapters into the backbone, zeroes lora_B, clears the optimizer state and restarts the schedule; validation loss spikes after each merge, so checkpoints are taken at the end of a cycle.

Evaluation

Held-out SlimPajama sentences (200,000, never seen in training), evaluated with scripts/tech_report.py.

teacher-forced cross-entropy (whole split, token-weighted)	0.0047
perplexity	1.0047
next-token accuracy	99.85%
exact reconstruction, 8,000 sentences (greedy)	98.61%
character error rate (Levenshtein / length)	0.26%
word error rate	0.39%
cosine(enc(src), enc(reconstruction))	0.9995
reconstruction truncated at decoder max length	0.00%
exact reconstruction, 32 stress probes	27/32
throughput, 8×H100 bf16 (encode / decode)	14,986 / 1,001 sentences/s

By sentence length — reconstruction degrades past ~48 tokens, and the 4B encoder holds up markedly longer:

length (qwen3 tokens)	n	brivangl/qwenar-0.6b-montevideo exact / CER	brivangl/qwenar-4b-montevideo exact / CER	brivangl/qwenar-0.6b exact / CER
0-15	3,064	100.0% / 0.00%	100.0% / 0.00%	99.9% / 0.00%
16-31	3,426	99.7% / 0.02%	99.9% / 0.00%	99.0% / 0.09%
32-47	1,282	97.0% / 0.40%	99.3% / 0.07%	87.6% / 1.65%
48-63	208	79.3% / 3.77%	97.1% / 0.10%	57.2% / 10.45%
64-79	14	7.1% / 31.85%	28.6% / 13.36%	7.1% / 44.94%
80-95	3	0.0% / 40.72%	33.3% / 5.71%	0.0% / 52.92%
96-111	3	0.0% / 42.76%	0.0% / 24.60%	0.0% / 50.54%

Prose vs. code (rule-based detector; code is 0.7% of the split):

content	n	brivangl/qwenar-0.6b-montevideo exact / CER	brivangl/qwenar-4b-montevideo exact / CER	brivangl/qwenar-0.6b exact / CER
prose	7,941	98.7% / 0.23%	99.6% / 0.04%	96.3% / 0.65%
code	59	81.4% / 3.98%	94.9% / 0.92%	78.0% / 7.15%

The failure mode is the published one: the sentence frame survives while dense code syntax, identifiers, URLs and formula symbols scramble.

src: def flatten(xs): return [y for x in xs for y in (flatten(x) if isinstance(x, list) else [x])]
gen: def flatten(xs): return [y for x in xs for y in (flatten(x) if isinstance(x, list) else [x])]

src: for f in *.parquet; do python -c "import pyarrow.parquet as pq; print(pq.ParquetFile('$f').metadata.num_rows)"; done
gen: for f in *.parquet; do python -c "import parquet.py as pqarrow; print(pq('quetPFile.fd').meta.numrows) fscanf(d);
     (Levenshtein 40, CER 34.48%)

src: fn parse(input: &str) -> Result<Vec<u32>, ParseIntError> { input.split(',').map(str::trim).map(str::parse).collect() }
gen: fn parse(input: &str) -> Result<Vec<u32>, ParseIntError> { input.split(',').map(str|trim).parse(str::map().collect) }
     (Levenshtein 11, CER 9.32%)

src: SELECT u.id, COUNT(o.id) AS orders FROM users u LEFT JOIN orders o ON o.user_id = u.id WHERE u.active GROUP BY u.id HAVING COUNT(o.id) > 3;
gen: SELECT u.id, COUNT(o.id) AS orders FROM users u LEFT JOIN orders o ON o.user_id = u.id WHERE u.active GROUP BY u.id ORDER HAVING COUNT(o.id) > 1;
     (Levenshtein 7, CER 5.04%)

Comparison with the other qwenar models

model	encoder	d_emb	K	params	training data	steps	corpus seen	CE ↓	tok acc ↑	exact ↑	CER ↓	WER ↓	cos ↑	probes ↑
brivangl/qwenar-0.6b-montevideo	pplx-embed-v1-0.6b	1024	2	1194M	SlimPajama 9.2B sentences	600,000	14%	0.0047	99.85%	98.61%	0.26%	0.39%	0.9995	27/32
brivangl/qwenar-4b-montevideo	pplx-embed-v1-4b	2560	4	4629M	SlimPajama 9.2B sentences	1,000,000	9%	0.0012	99.96%	99.59%	0.05%	0.10%	0.9998	28/32
brivangl/qwenar-0.6b	pplx-embed-v1-0.6b	1024	2	1194M	PubMed / SlimPajama-6B / SYNTH mix	750,000	57% ¹	0.0141	99.58%	96.21%	0.69%	1.00%	0.9988	26/32

All numbers on the same held-out SlimPajama split (200,000 sentences), same greedy decoding, bf16. CE / tok acc: teacher-forced over the whole split. exact / CER / WER / cos: free-running reconstruction of 8,000 sentences — CER = character Levenshtein / source length, WER = word Levenshtein / source words, cos = cosine between encoder embeddings of source and reconstruction. probes: exact reconstructions of 32 fixed stress sentences (long sentences, code, numbers, URLs).

¹ brivangl/qwenar-0.6b was trained on a PubMed/textbook/SlimPajama-6B/SYNTH mix and saw ~57% of that corpus; on this general-domain split it is evaluated out-of-domain.

Limitations

Read this before drawing conclusions from the numbers above.

• Reconstruction is the only thing measured. There is no retrieval, STS, clustering, or downstream evaluation anywhere in this project. The model was trained purely to reconstruct, with no contrastive objective, so the embedding space is optimised to be decodable, not to be semantically well-shaped. Do not assume these vectors are good sentence embeddings for similarity tasks — nothing here tests that.
• Long sentences degrade. Past ~48 tokens exact reconstruction drops quickly (see the length table); the training cap is 256 tokens, and fewer than 1% of training sentences exceed 64.
• Code and high-entropy strings scramble. Identifiers, URLs, IBANs, formula symbols — 1024 floats hold a sentence's structure and content, not arbitrary character strings.
• English only. No multilingual training or evaluation.
• Sentences only, by construction of the training data.
• One run, no ablations. Nothing isolates the contribution of K=2, DoRA vs LoRA, ReLoRA vs a single cycle, or the encoder size beyond the single 0.6B-vs-4B comparison above.
• Less than one epoch. The run saw ~14% of the corpus.
• Trained on web text, so it reproduces whatever biases and inaccuracies that carries. It is a reconstruction model: it will faithfully re-emit harmful or false input text.

License and attribution

Apache-2.0. Built from perplexity-ai/pplx-embed-v1-0.6b (MIT) and Qwen/Qwen3-0.6B-Base (Apache-2.0); the bidirectional encoder code is derived from the former. Third-party code and model licenses: THIRD_PARTY.md.

Implements ideas from SONAR, Large Concept Models, ReLoRA, DoRA and LoRA+.

Citation

@software{qwenar,
  author = {Drokin, Ivan},
  title  = {qwenar: a sentence autoencoder built from open checkpoints},
  year   = {2026},
  url    = {https://github.com/IvanDrokin/QwenAR}
}