YAML Metadata Warning:empty or missing yaml metadata in repo card
Check out the documentation for more information.
BeyondRetrievalBench
Testing Functional Memory and Memory Updating in Long-Context LLMs (Pretrain version)
Pretrain-Model-Friendly Evaluation
This benchmark includes a version specifically adapted for evaluating pretrain (base) models, which do not rely on instruction-following capabilities. For a more detailed introduction, please refer to the original API-calling version: https://huggingface.co/alych9/BeyondRetrievalBench
Evaluation Method
Instead of prompting the model to generate a free-form answer, the pretrain-friendly version presents the model with multiple-choice options and selects the answer by computing the PPL (perplexity) / log probability of each option given the context. The option with the highest log probability is taken as the model's prediction.
This design removes the dependency on instruction-following ability and enables fair evaluation of base pretrain checkpoints.
Evaluation scripts:
7_eval_ppl_hf_load.py— loads models via HuggingFaceAutoModelForCausalLM7_eval_ppl_haillm_load.py— loads models via internal HAI-LLM interface
Option Design
Each task type uses carefully constructed distractors to probe distinct failure modes:
MemoryRet (3 options)
| Option | Type | Description |
|---|---|---|
| ✓ | correct |
The ground-truth answer |
| ✗ | retrieval_similar |
Output of the example whose input is most similar to the query input — tests whether the model confuses surface similarity for the right answer |
| ✗ | random_wrong |
A programmatically mutated wrong answer — baseline distractor |
MemoryCom (4 options)
| Option | Type | Description |
|---|---|---|
| ✓ | correct |
The ground-truth answer after applying all operators in the correct order |
| ✗ | surface_similar |
Output of an example with the most similar input — tests retrieval-over-induction bias |
| ✗ | wrong_rule |
Answer computed by applying operators in a wrong order — tests whether the model understands operator composition |
| ✗ | random_wrong |
A programmatically mutated wrong answer — baseline distractor |
MemoryRev (4 options)
| Option | Type | Description |
|---|---|---|
| ✓ | correct |
The ground-truth answer using the revised (latest) operator definitions |
| ✗ | retrieval_similar |
Output of the example with the most similar input — tests retrieval-over-update bias |
| ✗ | old_rule |
Answer computed using the original (pre-revision) operator definitions — tests whether the model successfully updates its memory after rule revision |
| ✗ | random_wrong |
A programmatically mutated wrong answer — baseline distractor |
Option Generation
Options are generated by 5b_gen_multiple_choice_options.py, which combines LLM-assisted generation (for semantically meaningful distractors) with programmatic mutation (for random wrong answers). Option labels (A/B/C/D) are randomly shuffled per case to prevent position bias.
The dataset files containing options follow the naming convention:
{domain}_{task}_[k{K}_][editk{editK}_]M{M}_que_ans.json
where M is the number of sampled examples shown in context (default: 70).