Akiki-MiniLM-L6-v2-arguana

A 22.86-million-parameter English counterargument retriever. A learned 384 × 384 affine weight and 384-element bias transform the input word embeddings of a completely frozen sentence-transformers/all-MiniLM-L6-v2. Only 147,840 parameters were trained. The output is a normalized 384-dimensional embedding, with no classifier.

The validation-selected epoch 45 achieves 57.85 nDCG@10 on the official ArguAna test set using the official BEIR scorer. This is an ArguAna-supervised specialist, not an ArguAna zero-shot result. It has not been evaluated across the full MTEB suite.

Results and the two frozen baselines

All locally measured rows below use the same 1,406 test queries, 8,674-document corpus, unchanged relevance labels, cosine retrieval, exclusion of matching query/document IDs, and official BEIR tie handling.

Model / preprocessing	nDCG@10 × 100	Recall@1 (%)	Recall@10 (%)	Full-corpus retrieval NLL ↓
Stock frozen MiniLM, truncate to 256 tokens	50.17	25.60	79.09	3.103040
Frozen MiniLM, our chunking pipeline	49.11	24.11	78.45	3.117224
Akiki epoch 45, our chunking pipeline	57.85	33.43	84.71	2.748142

The stock baseline reproduces the official MiniLM MTEB result: ndcg_at_10 = 0.50167 after official rounding. Our local unrounded value is 0.5016742612134475. The stock baseline truncates long text; the matched frozen baseline retains it through the exact same chunking as Akiki. The gain is 7.68 nDCG points over stock MiniLM and 8.73 points over the matched frozen baseline. Chunking by itself did not improve this benchmark.

For epoch 45, unrounded nDCG@10 is 0.5784773751814157, and the official BEIR routine rounds it to 0.57848. The Hugging Face benchmark value is 57.848 on its 0–100 scale. Recall@1 is 470/1,406; it is distinct from nDCG@10.

Validation at epoch 45: full-corpus retrieval NLL 2.098376987871397, nDCG@10 66.67, Recall@1 46.03%. Training stopped at epoch 55 after ten epochs without lower validation NLL. The test was evaluated at frozen and after each epoch for monitoring, but was never used for gradient updates, checkpoint selection, or early stopping.

Correction to the original monitor scores

The original monitor reported 58.08 for epoch 45 and 49.32 for the chunked frozen baseline. It broke exact score ties using ascending document IDs; pytrec_eval uses descending document IDs. Duplicate texts under different document IDs make the distinction material. Official rescoring changes these to 57.85 and 49.11. No weights, scores, relevance labels, or selected checkpoint were changed. Validation NLL is independent of ranking tie breaks.

results/original/ preserves the original monitor reports and full score matrices. results/official/ contains corrected BEIR metrics and per-query ranked results. results/tie-audit.json records the 83 epoch-45 queries and 74 frozen queries whose nDCG@10 or Recall@1 changes. results/stock-baseline/ contains the stock-truncation comparison. Use the official results for benchmark comparisons.

Architecture and frozen parameters

Token IDs → original MiniLM word embeddings
          → Akiki affine (384 → 384, W and b)
          → frozen MiniLM (6 transformer blocks)
          → masked token mean per chunk
          → chunk L2 normalization
          → content-token-count-weighted chunk mean
          → final L2 normalization → 384-dimensional embedding

Native position/type embeddings and embedding LayerNorm are applied inside MiniLM, after the affine. Every backbone parameter—including token/position/type embeddings, attention, feed-forward layers, norms and the unused pooler—has requires_grad=False and no accumulated gradient. Before every backbone forward, the implementation asserts that only the affine weight and bias have requires_grad=True. Gradients propagate through the frozen backbone into the affine on both query and document branches. Dropout remains disabled while training.

Component	Parameters
Frozen MiniLM	22,713,216
Trainable affine W and b	147,840
Total	22,861,056

The base revision is 1110a243fdf4706b3f48f1d95db1a4f5529b4d41. It was copied from the existing local cache. There is no Typed Decisions classifier or initialization from a previous trained specialist. Fresh training starts with identity W and zero b.

Long inputs

The tokenizer first retains the complete text. It splits it into non-overlapping chunks of at most 254 content tokens, then adds [CLS] and [SEP] to each chunk, giving 256 tokens maximum per backbone pass. Masked means include these special tokens and exclude padding. Each chunk vector is L2-normalized; the document vector is their content-token-count-weighted mean, L2-normalized again. Empty texts receive the two special tokens with weight one.

No text is silently truncated in the Akiki or matched frozen pipeline. Across the experiment, 14,549 distinct texts produced 19,636 chunks; the longest had 1,620 content tokens and seven chunks. A tokenizer warning about a raw text exceeding 512 tokens is harmless here: only bounded chunks reach the backbone. baseline.py deliberately uses truncation to reproduce stock MiniLM instead.

Data and holdout protocol

Supervision comes from the authors' ArguAna Counterargs corpus, using its training and validation debate splits. Evaluation uses the unchanged official BEIR ArguAna archive. The packaged evaluation texts and qrels were also checked against the current MTEB ArguAna dataset and match exactly.

Split	Queries	Corpus documents	Positive pairs
Cleaned training	1,973	3,948	1,974
Cleaned validation	958	1,917	958
Official test	1,406	8,674	1,406 judgments

To protect the holdout, entire validation debates overlapping original test texts or BEIR test queries/gold-answer bodies were removed. Entire training debates overlapping those protected texts or retained validation were then removed. Detection uses normalized exact matching and ≥85% five-word-shingle containment with at least 20 shingles. Text duplicates within train/validation were collapsed, merging positive labels. The audit is supplied in data/prepared/audit.json and excluded-overlaps.json.

The test retrieval corpus can contain training texts as distractors; test queries and gold-answer texts were protected. The five known missing gold documents in BEIR ArguAna remain unchanged: all 1,406 queries contribute to ranking metrics, with zero on missing answers. Retrieval NLL is computed only on the 1,401 answerable queries. NLL is the negative log gold mass under a full-corpus softmax at temperature 0.05, excluding the identical-ID document. It is not a calibrated probability of real-world relevance.

Queries use their text directly. Corpus inputs are (title + ' ' + text).strip(). There are no query/document prompt prefixes. JSON bundles, JSONL exports, raw archives, preparation code, and source hashes are included. See DATA_NOTICE.md for data attribution and licensing, separate from the model/code license.

Reproduce evaluation offline

Use Python 3.12 and a PyTorch build appropriate for the GPU. The reference environment was PyTorch 2.13.0+cu130, Transformers 5.17.0, FP32 deterministic SDPA, TF32 disabled, on an NVIDIA RTX PRO 6000 Blackwell Workstation Edition. Package versions are pinned in requirements.txt.

Run from this directory after installing the requirements. Installing Python software can use the network; model loading and all experiment scripts are offline and require no model or dataset download.

python -m pip install -r requirements.txt
export HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 HF_DATASETS_OFFLINE=1
export PYTHONHASHSEED=0 CUBLAS_WORKSPACE_CONFIG=:4096:8
python reproduce.py --device cuda:0 --epoch 45 --split all --verify
python reproduce.py --device cuda:0 --epoch frozen --split all --verify
python baseline.py --device cuda:0

For CPU-only rescoring of the saved predictions, with no model encoding:

python reproduce.py --scores-only --epoch 45 --split test --verify
python reproduce.py --scores-only --epoch frozen --split test --verify

reproduce.py calls the official BEIR 2.2.0 evaluator backed by pytrec-eval-terrier 0.5.10. It also records unrounded per-query metrics, NLL, and the historical monitor convention. This is a self-reported result independently rescored using the official BEIR implementation, not an HF-verified result or a claim to have run the full MTEB harness. The .eval_results/arguana.yaml entry documents that scope.

The packaged loader re-encoded every validation/test score for epoch 45 bit-for-bit identically on the reference GPU. A fresh first training epoch also reproduced both affine tensors bit-for-bit, with unchanged backbone hashes. checks/ contains the evidence. Across other hardware/software, small floating-point differences can alter rankings among nearly tied documents; --verify checks score agreement within 2e-6, not cross-platform bitwise identity.

Basic inference

This is a custom encoder package. Use AkikiEncoder; loading only base_model/ would omit the trained affine and chunk aggregation. No trust_remote_code or network lookup is required.

from model import AkikiEncoder

model = AkikiEncoder(device="cuda:0")  # use "cpu" if needed
queries = model.encode(["A proposed policy should be adopted because..."])
documents = model.encode([
    "A counterargument to that policy is...",
    "A different argument is...",
])
similarities = queries @ documents.T
print(similarities.argmax(axis=1))

The output consists of embeddings and cosine similarities, not ten-option classification probabilities. The same encoder processes queries and documents.

Reproduce training

training_config.json and experiment.yaml describe the recipe. Relative model/data/output paths resolve against this package directory. Set device in training_config.json if your GPU index differs.

export HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 HF_DATASETS_OFFLINE=1
export PYTHONHASHSEED=0 CUBLAS_WORKSPACE_CONFIG=:4096:8
python train.py --check-only
python train.py

The script saves resumable optimizer/state checkpoints every ten updates and at epoch boundaries. SIGTERM/SIGINT requests a clean pause; rerunning resumes from saved state. Use a server-side service or nohup python train.py > training.log 2>&1 & to disconnect while it runs. --stop-after-epoch 1 is an optional verification pause; normal training has no fixed epoch cap. Start with an empty output directory for an independent run. Original experiment history and per-step JSONL are in provenance/original/.

• Loss: query-to-document in-batch multi-positive contrastive loss, cosine/0.05; all known positives in the batch receive credit.
• Optimizer: AdamW, learning rate 1e-4, betas (0.9, 0.999), epsilon 1e-8, weight decay 0.
• Batch: 64 positive pairs, 32 chunks per backbone sub-batch, 31 optimizer updates per epoch.
• Gradient clipping: global affine norm 1.0 before each update.
• Initialization: seed 0, identity W, zero b. Each epoch shuffles pairs with Random(seed + epoch).
• Constant learning rate; no warmup. All computations FP32, dropout disabled.
• Selection: lowest full-validation-corpus retrieval NLL at temperature 0.05. Frozen epoch 0 is eligible.
• Stopping: at least 10 epochs, then 10 consecutive epochs without validation NLL improvement. Selected 45, finished 55. No lower-LR refinement was part of this run.

The training script retains the historical monitor's tie convention so the original history can be reproduced. Score selected checkpoints with reproduce.py for official ranking metrics. The selection loss is unaffected by that convention.

To regenerate the exact prepared splits from the included archives without downloads:

python prepare_data.py --raw data/raw --out reproduced_data

Verbatim original source and logs remain under provenance/original/ (including original internal naming and paths). The portable entry points at the package root use Akiki naming and package-relative paths. SHA256SUMS covers all released files except itself.

Scope and attribution

These results support counterargument retrieval on the held-out ArguAna test. General retrieval quality and other languages/tasks were not established. ArguAna training exposure must be disclosed when comparing this specialist with general-purpose or zero-shot models.

Model/code: Apache-2.0; see LICENSE and NOTICE. The base MiniLM model is from the Sentence Transformers authors. ArguAna is by Henning Wachsmuth, Shahbaz Syed, and Benno Stein, Retrieval of the Best Counterargument without Prior Topic Knowledge, ACL 2018 (paper). BEIR is by Nandan Thakur and colleagues, BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models (paper). MTEB (project) provides the benchmark and public result records.