[Case Study] Sequence truncation (512 vs 256) and passage chunking in Node.js / Transformers.js dense retrieval
Overview
We prepared a small reproducible case study with Xenova/all-MiniLM-L6-v2 in Node.js / Transformers.js. This is a usage example, not a model bug report or a general benchmark.
The upstream Sentence Transformers default is 256 wordpieces, while this Xenova revision's tokenizer has model_max_length=512 and its model configuration has max_position_embeddings=512. With truncation enabled, the total input length is 512, including special tokens such as [CLS] and [SEP].
Experimental setup
- Model revision: 751bff37182d3f1213fa05d7196b954e230abad9
- Runtime: Node.js v22.14.0, macOS Darwin x64, @xenova /transformers v2.17.2; quantized ONNX, mean pooling and L2 normalization.
- One synthetic query: "What is the secret deployment key for Project Lavenbee?"
- Target sentence: "The secret deployment key for Project Lavenbee is OMEGA-7749-DELTA."
- OMEGA-7749-DELTA is a deliberately invented dummy string, not a real credential.
- The target sentence is placed at offsets 1, 315, and 786 in three versions of an 804-token document. Each is ranked against five short technical distractor documents (six documents per comparison).
- Chunking: 80-word windows, 20-word overlap; document score is the maximum passage similarity (MaxP). Short distractors fit in a single passage. This design does not establish general retrieval superiority.
Observed results
| Target position | Preserved after truncation? | Unchunked cosine | Unchunked rank | Chunked MaxP cosine | Chunked rank |
|---|---|---|---|---|---|
| Front (offset 1) | Yes | 0.3027 | 1/6 | 0.5107 | 1/6 |
| Middle (offset 315) | Yes | 0.3177 | 1/6 | 0.5402 | 1/6 |
| End (offset 786) | No | 0.1759 | 4/6 | 0.5132 | 1/6 |
In the end-placement case, the target token sequence was absent from the truncated input_ids tensor. Chunking retained it in chunk 8 and restored rank 1 in this example. Rank 4 is a top-1 miss, not a claim that every top-k search would fail.
Reproduction package
https://github.com/castfox2021/all-minilm-truncation-study
The repository contains the script, dependency lockfile, README, raw structured results and model-file SHA-256 values recorded at runtime.
git clone https://github.com/castfox2021/all-minilm-truncation-study.git
cd all-minilm-truncation-study
npm ci
node reproduce.mjs
The default run downloads the pinned revision into an empty cache. Afterward, node reproduce.mjs --offline can repeat it from the populated cache.
Request for external reproduction
These results have been repeated locally on the same macOS x64 environment. Independent external cross-validation is still pending.
If you try this on Linux, Windows, or another Node version, please share your platform, runtime version and confirmation log—or any discrepancy. We'd appreciate feedback on the setup and interpretation.
Prepared while developing LAVENBEE's document-retrieval workflow.