brivangl/qwenar-0.6b

A sentence is compressed into one 1024-dimensional vector, and that vector alone is decoded back into the sentence.

At its best checkpoint this model reconstructs 30 of 32 held-out probe sentences character-for-character from a single float vector.

The idea is SONAR's and the Large Concept Model's: a sentence embedding lossless enough to decode, so the vector can stand in for the text. The construction is different — instead of training a seq2seq model from scratch, qwenar connects two off-the-shelf open checkpoints with a single learned linear bridge and adapts both with DoRA. The whole learned connector is about 30 lines of code.

Code, training pipeline and full evaluation: https://github.com/IvanDrokin/QwenAR

Requires transformers>=5.5, and older versions fail silently. This model was trained on 5.5.0. Under transformers 4.x it raises nothing and produces fluent, plausible text that is simply wrong: teacher-forced cross-entropy goes from 0.0052 to 3.2614 and next-token accuracy from 100% to 56%.

Usage

from qwenar.pipeline import Qwenar          # pip install qwenar

qw = Qwenar.from_pretrained("brivangl/qwenar-0.6b")

vecs = qw.encode(["Serum ferritin remained within the reference range."])
print(vecs.shape)                            # torch.Size([1, 1024])
print(qw.decode(vecs))                       # back to the sentence
print(qw.roundtrip(["..."]))                 # both at once

Or as a plain transformers model, with no extra dependency:

from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained("brivangl/qwenar-0.6b", trust_remote_code=True)
enc_tok = AutoTokenizer.from_pretrained("brivangl/qwenar-0.6b", subfolder="encoder_tokenizer")
dec_tok = AutoTokenizer.from_pretrained("brivangl/qwenar-0.6b")

batch = enc_tok(["Serum ferritin remained within the reference range."],
                return_tensors="pt", padding=True)
emb = model.encode(batch["input_ids"], batch["attention_mask"])     # (1, 1024)
ids = model.generate_from_embeddings(emb, max_new_tokens=64, do_sample=False)
print(dec_tok.batch_decode(ids, skip_special_tokens=True))

Two tokenizers, because the encoder and the decoder come from different checkpoints. The decoder tokenizer sits at the repository root so plain AutoTokenizer.from_pretrained(repo) gives you the one generation needs; the encoder tokenizer is in encoder_tokenizer/. Decoder targets must be right-padded — Qwenar enforces this.

Architecture

sentence ──► perplexity-ai/pplx-embed-v1-0.6b
             Qwen3 with the causal mask DISABLED → bidirectional
             mean-pool over non-pad tokens; no tanh, no INT8
                              │
                       vec (1024,)             ◄── this is the artifact
                              │
             EmbedToPrefix:  Linear(1024 → K·d_model) → GELU
                             → view(B, K, d_model) → RMSNorm      K = 2
                              │
                     prefix (B, 2, d_model)
                              │
             Qwen/Qwen3-0.6B-Base + DoRA(r=32, α=64) on q,k,v,o,gate,up,down
                              │
                  teacher-forced next-token cross-entropy
                  labels = -100 on the prefix and on padding

1194M parameters total. The encoder is adapted too (DoRA r=16, at a 10× lower learning rate), not frozen.

Training data

Sentence-level English, from a 22.83B-token corpus. The published run saw about 13.0B supervised tokens — roughly 57% of one epoch; it never completed a pass over the data.

source	share of train tokens
MedRAG/pubmed	54.2%
DKYoon/SlimPajama-6B	30.0%
PleIAs/SYNTH	15.7%
MedRAG/textbooks	0.2%

Over half the training signal is PubMed abstracts. These proportions were not a design target — they are what the source directories happened to contain. Sentences were segmented with SaT (sat-3l-sm) and capped at 96 tokens, so the model has never seen a longer input.

The corpus itself is not released, and the dataset revisions were not recorded, so it cannot be rebuilt byte-for-byte. Full provenance, including what is missing, is in DATASET.md.

Training procedure

Optimizer steps	759,748
Supervised tokens	13,013,262,296
Hardware	4 GPUs, bf16 mixed precision
Wall clock	329h 52m
Batching	token budget (4096 tokens/batch), not a fixed batch size
Adapters	DoRA — encoder r=16, decoder r=32/α=64
ReLoRA	7 merge + reinit cycles, every 100k steps
Learning rates	bridge 1e-4, decoder LoRA 1e-4, encoder 1e-5, cosine

ReLoRA periodically merges the adapters into the backbone, zeroes lora_B, clears the optimizer state and restarts the schedule. One consequence matters for anyone picking a checkpoint: validation loss spikes after each merge, so the last checkpoint is not the best one. These weights are step 700,000, taken at the end of a converged cycle.

Evaluation

Reconstruction cross-entropy on held-out sentences, and the batching invariants.

best val/loss (= val/ce)	0.0126 at step 700,000
teacher-forced cross-entropy on the probe set	0.0052
next-token accuracy	285/285
exact reconstruction, 32 probe sentences	30/32
encode / decode batch invariance	~6e-07

Batch invariance is worth stating explicitly: encoding or decoding a sentence alone gives the same result as doing it inside a padded batch, to float precision. Batching is a performance choice here, not a behavioural one.

The failure mode is consistent and worth knowing: the sentence frame survives while dense numerals and rare nomenclature scramble.

src: Musculoskeletal injury causes pain and when chronic can affect mental
     health, employment and quality of life.
gen: Musculoskeletal injury causes pain and when chronic can affect mental
     health, employment and quality of life.

src: arbazone 14, 1,3-thiazolines 15a-c and 4-thiazolidinone 16 exhibited
     significant COX-2 inhibition ...
gen: arbazone 14, 13, 1-thiazolones4a-b and 15-cidine 2-thiazinoleic strain 60
     exhibited significant ...

1024 floats carry a sentence's structure and content, but not arbitrary high-entropy strings.

Limitations

Read this before drawing conclusions from the numbers above.

• Reconstruction is the only thing measured. There is no retrieval, STS, clustering, or downstream evaluation anywhere in this project. The model was trained purely to reconstruct, with no contrastive objective, so the embedding space is optimised to be decodable, not to be semantically well-shaped. Do not assume these vectors are good sentence embeddings for similarity tasks — nothing here tests that.
• Heavily biomedical. ~54% of training tokens are PubMed abstracts. Expect worse reconstruction on general text than the headline number suggests.
• English only. SONAR's defining property is a shared multilingual space; this model has no multilingual training or evaluation whatsoever.
• Sentences only, ≤96 tokens, by construction of the training data. Behaviour on longer inputs is untested.
• One run, no ablations. Nothing isolates the contribution of K=2, DoRA vs LoRA, ReLoRA vs a single cycle, or an unfrozen vs frozen encoder. These are the hyperparameters that were used, not ones shown to be necessary.
• Less than one epoch. The run saw ~57% of the corpus.
• Trained on web and biomedical text, so it reproduces whatever biases and inaccuracies those carry. It is a reconstruction model: it will faithfully re-emit harmful or false input text.

License and attribution

Apache-2.0. Built from perplexity-ai/pplx-embed-v1-0.6b (MIT) and Qwen/Qwen3-0.6B-Base (Apache-2.0); the bidirectional encoder code is derived from the former. Third-party code and model licenses: THIRD_PARTY.md.

Implements ideas from SONAR, Large Concept Models, ReLoRA, DoRA and LoRA+.

Citation

@software{qwenar,
  author = {Drokin, Ivan},
  title  = {qwenar: a sentence autoencoder built from open checkpoints},
  year   = {2026},
  url    = {https://github.com/IvanDrokin/QwenAR}
}