Title: Beyond Prompts: Dynamic Conversational Benchmarking of Large Language Models
Benchmark: LTM Benchmark
Summary: This paper introduces the LTM benchmark designed to evaluate the Long-Term Memory (LTM) and Continual Learning (CL) capabilities of conversational agents. The LTM Benchmark engages agents in a single, prolonged conversation, incorporating multiple tasks and distractions to simulate realistic and meaningful interactions.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Long Term Memory
Phenomenon defined by authors: Yes
Phenomenon definition: The ability to recall and effectively utilize past information. LTM encompasses several skills related to the generation and management of memories, which include but are not limited to recall, information integration, and the handling of conflicting information.
Definition scope: Subset
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Widely-agreed
Task definition: An LLM engages in a prolonged, dynamic conversation where multiple tasks are interleaved. Within this conversation, specific pieces of information, termed 'needles,' are included amidst unrelated content ('haystack'). These needles are essential for completing subsequent tasks. The LLM is later queried on these tasks, requiring it to retrieve and integrate the relevant needles from the conversation history, thereby assessing its long-term memory and information integration capabilities.
Task item definition: A dynamic, multi-turn conversation containing interleaved messages from different tasks, within which specific "needles" (relevant sentences) are included. 
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Dataset size: 33
Dataset metadata available: No
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test
Response format: Short free response (e.g. single word or number)
Metric definition: Exact Match (accuracy, F1, precision, recall), n-gram (BLEU, ROUGE, chrF), LLM post-processing (extracting answers, reformatting for automated scoring)
Metric aggregation: Simple Mean
Metric subscores: Yes
Metric subscores detail: They include the separate runs for each test scenario subset in the Appendix.
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Industry
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Taxonomy: NLP / Long Context
New taxonomy: General Capability / Long Context