Title: Exposing Attention Glitches with Flip-Flop Language Modeling
Benchmark: FFLM
Summary: This paper introduces FFLM, a synthetic benchmark designed to investigate "attention glitches" that cause Transformer-based language models to make sporadic reasoning errors.The benchmark tests models' ability to copy binary symbols across long distances while ignoring intervening tokens. The paper with this benchmark further explores regularisation and architectural tweaks to mitigate them.
Contribution: Provides large‑scale empirical study (10 k+ models) and mechanistic analysis.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Reasoning, (Robust long‑range) memory
Phenomenon defined by authors: Yes
Phenomenon definition: A flip-flop language modeling (FFLM) task is defined on sequences of write,
read, and ignore instructions: write sets the memory state to a certain value which is later
retrieved by read, while ignoring any contents in between. Authors are interested in whether language models can learn a flip-flop language from samples, which they define as processing the read operations perfectly to understand model output inaccuracies. (Sections 1 and 3.1)
Definition scope: Subset
Definition integrity: Single cohesive phenomenon
Definition integrity detail: No
Phenomenon contested: Contested
Purpose notes: They assume that the correct memory retrieval (subset) as the absence of incorrect responses.
Task definition: Predict the next token in sequences built from write/read/ignore instructions plus binary data so that every read must output the latest write bit.
Task item definition: A length‑T token sequence, evaluation checks all read positions.
Task source: Procedurally-generated task examples (e.g. Creating instances from a template)
Task ecology: Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Task ecology detail: Synthetic stress‑test (not user interaction scenarios)
Dataset size: 16M training sequences
Dataset size detail: 16M training sequences, 160K sparse o.o.d. sequences , and 4K dense o.o.d. sequences
Dataset metadata available: Yes
Dataset metadata detail: difficulty (fixed sequence lengths, writing probability, reading probability, computationally ignoring probability)
Sampling method: Random sample (creators defined a task space and sampled from it)
Split reviewed: Test, Train, Validation
Response format: Short free response (e.g. single word or number), Structured response (e.g. valid JSON, API call alone)
Response format detail: Deterministic read position index, binary digits
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: Mean error, scatter plots, attention heatmaps
Metric subscores: Yes
Metric subscores detail: Three canonical splits: (1) In-distribution, (2) sparse out-of-distribution token sets, (3) dense out-of-distribution sets where tokens are frequently appearing in read/write functions
Metric access: Outputs alone
Few-shot evaluation: Yes
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: No
Authors discuss validity: Yes
Author validity detail: 500 seed replicate study
Taxonomy: NLP / Long Context
New taxonomy: General Capability / Long Context