Title: Evaluating Very Long-Term Conversational Memory of LLM Agents
Benchmark: LOCOMO
Summary: The paper introduces LOCOMO, a dataset created through a machine-human pipeline that generates high-quality, very long-term dialogues by grounded LLM-generators in personas and temporal event graphs. Across 10 conversations (each averaging 600 turns and 16K tokens across up to 32 sessions), they present an evaluation benchmark measuring long-term memory in models through question answering, event summarization, and multi-modal dialogue generation tasks.
Contribution: The dataset is significantly longer than previous conversational datasets (16x longer than MSC, with 10x more turns and 5x more sessions on average). The conversations include multimodal elements through image-sharing and image-response behaviors. Note quite a small dataset (10 items), even though each item is very rich.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: long-term conversational memory
Phenomenon defined by authors: No
Phenomenon definition: They provide the following def: "This long term evaluation is crucial for refining engaging chatbots capable of remembering key information from past interactions, to generate empathetic, consistent, and useful responses" but this is more of a motivation than a definition. They talk a lot about _very_ long term memory but as far as I can see, don't explicitly define what counts as short vs long memory.
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Purpose notes: The authors frame conversational memory as a composite capability and design their evaluation benchmark with three distinct tasks (question answering, event summarization, and multi-modal dialogue generation) to measure different aspects.
Task definition: Three tasks: 1) a question answering task to assess memory recall from conversations, 2) an event summarization task to measure comprehension of causal and temporal connections, and 3) a multi-modal dialogue generation task to evaluate consistency in responses based on past context.
Task item definition: For the QA task, items are questions categorized into five reasoning types (single-hop, multi-hop, temporal, open-domain knowledge, and adversarial). Example: Input = A long context conversation, Q: "Whose birthday did X celebrate?"/ "Would X Likely enjoy The Four Seasons by Vivaldi?" --> Answer = multiple choice (A)

 For event summarization, items are prompts to summarize events within designated timeframes. Example: Input = long context convo, Q: "Summarize the significant events that have occured in X's life". 

For multimodal dialogue generation, items are prompts to continue conversations based on prior context. Example: Input = long context convo, Q: "Please generate conversation with appropriate image"
Task definition detail: Authors designed the tasks to measure different aspects of long-term memory in conversation. The QA task directly tests factual recall, the event summarization task tests causal and temporal understanding, and the dialogue generation task tests the ability to maintain consistency over time.
Task source: Crowd-sourced task examples (e.g. Prolific-created tasks), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task source detail: The dataset was created through a hybrid pipeline where first LLM-based agents generated conversations based on personas and event graphs, then human annotators edited these conversations to fix inconsistencies, replace irrelevant images, and ensure alignment with event graphs. The authors note that annotators edited approximately 15% of dialog turns and 19% of images.
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Task ecology detail: While the conversations are synthetic, they aim to mirror real-world online interactions between people over extended time periods. The authors tried to ensure ecological validity by grounding conversations in personas and realistic temporal event graphs. But still not real conversational data.
Dataset size:  1986 for QA, unclear about other tasks.
Dataset size detail: The dataset consists of 10 very long conversations. The QA benchmark includes 1,986 questions: 841 single-hop (42.3%), 282 multi-hop (14.2%), 321 temporal reasoning (16.1%), 96 open domain knowledge (4.8%), and 446 adversarial (22.4%). Each conversation contains an average of 35.8 ground truth events for summarization.
Dataset size notes: For multimodal dialogue, they generated 50 conversations as training data
Dataset metadata available: Yes
Dataset metadata detail: QA subcategory (e.g., single-hop, multi-hop)
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test
Response format: Multiple choice, Short free response (e.g. single word or number), Free response (e.g. summary paragraph, executable code)
Response format detail: For the QA task, the expected response format is primarily short free responses, where they match exact wording. However, in a figure, they also show the answer as "A) xxxx" which is confusing that it could be multiple choice. For the event summarization task, the format is free response summarization. For the dialogue generation task, the response is a free-form continuation of a multimodal dialogue.
Metric definition: Exact Match (accuracy, F1, precision, recall), n-gram (BLEU, ROUGE, chrF), FactScore (Min et al., 2023), a method that evaluates the factuality of generated text by decomposing both the reference and hypothesis into atomic facts; MMRelevance
Metric definition detail:  For the QA task, they say they use F1 score for exact matches after normalizing predicted and ground truth answers. For event summarization, they say they employ both ROUGE scores for lexical similarity and FactScore (which decomposes reference and hypothesis into atomic facts) to measure precision and recall of factual content. For multimodal dialogue generation, they say they measure alignment to groundtruth dialogues through MMRelevance and standard NLG metrics.
Metric aggregation: Simple Mean
Metric subscores: Yes
Metric subscores detail: For the QA task, scores are broken down by reasoning types (single-hop, multi-hop, temporal, open-domain knowledge, and adversarial). For event summarization, scores are provided for both ROUGE (ROUGE-1, ROUGE-2, ROUGE-L) and FactScore (Precision, Recall, F1) metrics. The multimodal dialogue generation results are analyzed by length of dialog history in tokens.
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: Yes
Authors discuss validity: A bit (but not a strong Yes)
Author validity detail: They test whether long-context LLMs perform differently than base models on the benchmark, confirming it measures the intended capability. They also analyze event summarization errors in detail, identifying five distinct error categories (missing information, hallucinations, misunderstanding dialog cues, speaker attribution errors, and mistaken salience). For multimodal dialog generation, they demonstrate that performance decreases with increased dialog history length, validating that the task measures long-term memory challenges.
Taxonomy: NLP / Long Context
New taxonomy: General Capability / Long Context