Learning through experience:Episodic memory representation for cognitive agents

Authors: Shweta Singh, Shraddha Seshadri
Primary area: transfer learning, meta learning, and lifelong learning
Keywords: Episodic Memory, Bio inspired Robot learning, incremental Memory structures

Abstract:
As the demand for intelligent robots and cognitive agents rises, the ability to retain and utilize past experiences through episodic memory has become crucial, especially for social companion robots that rely on previous interactions for task execution. To address this, we introduce Episodic Memory for Cognitive Agents (EMCA), a novel framework that advances knowledge representation by integrating real-world interactions. EMCA enables agents to adapt to complex environments by learning from tasks, interacting with humans, and processing multimodal data—such as speech, vision, and non-verbal cues—without pre-training on specific scenarios.
EMCA models episodic memory through a graph-based structure , allowing for incremental storage and retrieval of experiences. Each interaction or event enriches the memory graph, supporting continuous learning and adaptation without extensive retraining. This human-like memory formation optimizes the agent’s ability to retrieve relevant information for tasks like localization, planning, and reasoning based on prior experiences.Unlike conventional models relying on temporal markers or recurrent patterns, EMCA encodes data like human memory, allowing reasoning across diverse scenarios regardless of temporal patterns. The framework dynamically builds a memory graph with semantic and temporal connections based on the agent’s experiences, promoting flexible temporal reasoning. It also introduces mechanisms for clustering new memories and a dynamic retrieval policy that adjusts based on context or query type, ensuring robustness even in unpredictable scenarios. Empirical tests show EMCA adapts effectively to real-world data, offering reliability and flexibility in dynamic environments.

Decision: reject
Reviewer scores: 3, 3, 5, 3

Reviews and discussion:
<thread>
creation date:
1730051899843

thread_type:
official_review

<thread_object>
creator: reviewer_yt1b

<content>
summary:
The paper proposes a novel method for storing multi-modal episodic memories. The design takes inspiration from a model of human memory. More specifically, the paper introduces a novel type of memory graph with different types of connections. The method compares favorably against baselines in experiments.

strengths:
The paper tackles and interesting problem. The architecture is described in an intuitive way and seems sound and novel. The experiments are reasonably extensive with comparisons to baselines, multiple datasets, and ablations and show very promising results.

weaknesses:
- I found the paper to have some "false advertising":
  - Some of the introduction, motivation, and discussion circles around 'robots', I didn't see anything specific to robots in this paper. Yes, it could be integrated into a robot, but the method would equally well work for a body cam, social agent, etc. In robotics there is quite an extensive literature on 'lifelong learning' that covers some of the same challenges: what to memorize, how to store and to retrieve, how to generalize, what to forget, etc.
  - The title says 'learning'. At least according to my definition there is no learning in the paper. It proposes a way to store information and to retrieve it, so that would correspond to 'memorization' (=rote learning), while learning implies understanding the information and being able to apply it to new situations. The method could serve as a starting point for learning, but in the current paper that doesn't seem to be present in the method nor in the experiments.
  - The introduction makes it sound like a general method. But I have a few doubts about that. There are quite a lot of design choices, and some of choices in Sect. 3 seem rather specific. In the end the method is evaluated with a question answering task. What would need to change for a different task, say for a robot learning low-level movement control? Please clarify the generalizability of your method. Specifically, it would be nice if you could discuss how your approach might be adapted for different tasks (such as robot movement control), or explain any limitations in its applicability to other domains.
- Section 3 describes the method. It remains however very much on the level of HOW, rather than providing many insights in the WHY (reasoning behind design choices, consequences of design choices) and it isn't always clear what is a core part of the method and what is an implementation detail. Please provide more explanation for the key design decisions, discussing the rationale behind these choices and their potential implications. Additionally, please clearly distinguish between core methodological components and implementation details.
- Fig. 5 isn't very convincing (except for Arigraph)
- Memory requirement would be another interesting metric for comparing methods.
- The paper mentions the missing forgetting mechanism as a major limitation. Related to that it also leaves the question on 'what to store' unanswered. It reads like everything is stored, even if it is effectively a duplicate. I believe the real challenging question that needs to be solved is the memory management: what to store, what to consolidate/merge, what to forget, etc. Without those a memory representation is of limited value, and it remains unclear to me how suitable the proposed architecture is for extending it in that way - or if we would be better off redesigning it from scratch.
- The method relies on various models to extract features and to summarize things before storing them. I don't believe there is a 'one size fits all approach' but how to best do that depends on the retrieval task and the type of downstream tasks you have. The paper does not provide any indications on how to deal with that.
- There are a whole lot of design choices in this paper, an ablation on only the modalities and search methods seems a bit limited. There also is no sensitivity analysis for e.g. the clustering thresholds
- This seems to be a rather complex system, which makes reproducing results very difficult, with probably quite lot of additional implementation and setting details. I couldn't find any promises to release code (or at least a detailed appendix), which would have alleviated this concern.

questions:
- The paper seems to miss all details on how the responses are generated, but the whole experiments are based on evaluating the answers. Please provide a detailed explanation of your response generation process, as this is crucial for understanding and evaluating the experimental results.
- Sect. 3.2.4: This seems to result in just 'jumping' nodes. Conserving the chronological structure is a nice property, but the real question/challenge rather is on the side of the module that determines if there is any harm in 'skipping a step', e.g. when thinking about some of the uses-cases 'predict ... next activity' you mention
- Quite a lot of unclear details
  - Eq (1) vs (2): the difference between S_T and T_voiced is unclear
  - How is T_acoustic generated
  - Sect. 3.1.2, 3.1.3, and 3.2, l. 218: at places the paper sounds like everything is represented as 'text', at others it seems to be a mixture of text and other embeddings. It would be great to explain earlier on what it stored as what
  - l. 184: "organized by place, characters, and events" raises the question where those come from - that is explained later in the text
  - l. 220: "relevance of these tasks is assessed" - again HOW?
  - l. 212: what are the implications/limitations resulting form using a simple metric like cosine similarity?
  - Eq (17) sim seems undefined
- Fig. 2 and Sect. 4.1: I didn't get the terminology "master". Maybe that term can be avoided altogether (similar to https://learn.microsoft.com/en-us/style-guide/a-z-word-list-term-collections/m/master-slave )
- The paper comes across as unpolished
  - missing and extra spaces - e.g. in the title and abstract
  - the template uses natbib, not using correct commands for references \citep \citet (and resulting repeats of names) makes it painful to read
  - LaTeX has different opening and closing quotation marks
  - paragraph above "Contributions" is double
  - broken sentences in Sect. 5.1.2
  - a few references with incomplete info (e.g. l. 300 "as shown in 4", l. 191, l.468)

##After rebuttal
I appreciate the very detailed replies. However, I still believe the paper is not ready to be published and will maintain my score (see message below).
</content>
</thread_object>

</thread>
<thread>
creation date:
1730577470801

thread_type:
official_review

<thread_object>
creator: reviewer_1ixu

<content>
summary:
This paper presents Episodic Memory for Cognitive Agents (EMCA), a novel framework that enables AI systems to retain and utilize past experiences through a graph-based memory structure. The key innovation lies in its ability to: 1) Process multimodal data (vision, speech, non-verbal cues) without requiring pre-training; 2) Dynamically build and update memory representations through a graph structure with semantic and temporal connections; 3) Adapt to complex environments through continuous learning from interactions.

strengths:
originality: graph-based episodic memory structure multimodal processing without pre-training, dynamic clustering

quality: comprehensive testing on multiple datasets, benchmarking against exsiting methods, systematic component analysis in ablation study

clarity: clear problem formulation, good visual aids explaining complex concepts

significance: adresses crucial challenges in cognitive AI, social robot applications, potential impact on memory assistance systems

key innovations: 
1) removes pre-training requirements
2) enables continuous learning
3) provides real-time processing campability

weaknesses:
1. Scalability limitations: Does the graph structure grow exponentially with experiences? What are the computational costs?
2. More details about the Big Bang Theory dataset are needed.
3. More implementation details about the method in section 5.0.2 should be provided.
4. Forgetting is one of the key problems in memory systems - how does the paper assess and handle memory 5. retention and decay?
5. What is the real-time performance?
6. The paper tries to claim episodic memory for agents and robots, but robotic interaction with the environment is different from and much harder than agent interaction in a virtual environment. It is better to make a clear definition and scope. For instance, in L391, "robot's episodic memory" should be "agent's episodic memory".
7. Can authors provide the code and dataset for evaluation?
8. Repeated paragraph: L054-L062
9. Subtitle formatting issue in line 480

questions:
1. Will the code and datasets be made publicly available?
2. How does the system maintain temporal consistency without explicit time markers? Can you provide quantitative results comparing temporal reasoning accuracy with and without explicit timestamps?
3. What are the specific model architectures and hyperparameters used?
4. What are the computational costs for memory retrieval at different scales?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730669981771

thread_type:
official_review

<thread_object>
creator: reviewer_jsod

<content>
summary:
The authors introduce Episodic Memory for Cognitive Agents (EMCA) that models episodic memory based on a graph-structure. This allows them to incrementally store memories and retrieve experience. They also can cluster memories, have dynamic retrieval, and can handle temporal reasoning. Memory is structured as a graph, where each episode contains characters, temporal elements, location, and events. The edges of this graph can be temporal or semantic. They then build a retrieval system that can handle contextual, temporal, and spatial queries.

strengths:
- The Big Bang Theory and the Agent datasets seem very useful! 
- Their results indicate that their method performs better than other graph-based approaches. 
- This area is a growing field, especially as robots and agents become more capable and need better ways to scale their context. And their graph-based approach seems to be a meaningful contribution
- Table 2 is interesting, as it showcases that some of their questions require access to vision, acoustics, and/or dialogues. This result would be better if we knew what the Big Bang Theory and Agent dataset contained.

weaknesses:
## Main weaknesses
- After reading the full paper, I am not sure what the actual task is. What commands does the master ask? I understand the approach, but not what task it is specifically solving. Is the output the location of a specific memory? Or is it a free-form text answer?
- In terms of the structure of the paper, I do not think that the text space used for discussing how signals are captured (section 3.1) is very useful. I would argue this is the case with a chunk of this paper's equations, where it simply adds space when it does not need to, and it makes the paper more difficult to read. I would recommend condensing the information and matching ICLR's 9-page recommendation as opposed to 10. Similarly, there is an excessive use of new lines at arbitrary positions.
- There is no discussion on how the Big Bang Theory and Agent datasets were constructed. This on its own seems like a major contribution on its own. I would recommend the authors remove much of the superfluous equations and newlines (and move that into the appendix), and put more of an emphasis on this dataset component.
- I think I like what Section 4.1 is implying about combining a "master's" memory with that of an agent's, but it is not presented very clearly, and I do not see a connection with temporal graphs like the subsection title suggests
- Nor do I see how this section is a "theoretical comparison"
- "Master" terminology in line 346 is confusing, and should be introduced earlier in section 4.1. Also, the term "master" is generally frowned upon in these settings, so I would recommend a different term
- Results are poorly presented


## Formatting/Clarity issues:
- Check for spaces after periods or colons throughout the paper.
- Line spacing is odd in much of the paper
- Figure placement should ideally be on the top or bottom of a page, not in the middle with paper text above and below the paper. This makes the paper difficult to follow
- Figure 4, the legend has oddly shaped circles
- Figure 5's result is good, but it should not be a line graph with x axis being method and lines being the dataset. Instead it should be datasets on the x-axis and methods on the y-axis
- The results in section 5.1.1 are all discussed in a single paragraph. Is that all the main results? I would recommend splitting this up into a few bolded mini-sections and showing the main takeaway of each figure along with highlights on how the method performed, possibly with qualitative results.
- The focus of the introduction on falls a bit flat. Rather than focusing on the historical definition of episodic memory, focus more on how people have been engineering and building these kinds of systems. Focus on why other systems do not work, and why yours does.
- I would recommend larger fonts for the figures; they are difficult to read in the paper.
## Citations
Other relevant concurrent work on memory in robotics, some of which use graphs while others do not.
[1] Xie, Quanting, et al. "Embodied-RAG: General Non-parametric Embodied Memory for Retrieval and Generation." arXiv preprint arXiv:2409.18313 (2024).
[2] Anwar, Abrar, et al. "ReMEmbR: Building and Reasoning Over Long-Horizon Spatio-Temporal Memory for Robot Navigation." arXiv preprint arXiv:2409.13682 (2024).
[3] Bärmann, Leonard, et al. "Episodic Memory Verbalization using Hierarchical Representations of Life-Long Robot Experience." arXiv preprint arXiv:2409.17702 (2024).

questions:
- How was the dataset constructed? This is a major contribution that is not discussed.
- What does a successful or unsuccessful example look like? I would recommend looking at [2] or [3] above to see how to discuss dataset creation in such a setting.
- In Table 1, what metric is being used? It does not say in the caption or the table. It should be re-iterated in the table itself.
- Is a forgetting mechanism necessary, or would it be more like a memory aggregation mechanism so that retrieval is still efficient?
- How does the incremental storage/retrieval scale as the number of episodes change? This result is not displayed in the paper but I would argue is very important. If you only used say 10 episodes of EM instead of 181, is there a difference in performance? This would directly support contribution number 2 in Section 1 of your paper
- What is "Time complexity" in Table 3? Time complexity of BFS is O(V+E), but here a number is used instead. Authors should use "retrieval time" or something similar instead of time complexity.
- Can the authors describe the main takeaways of Section 4? I still do not understand the insights this section is supposed to provide.
- Also, there are some questions/concerns in the weaknesses section


Overall, I like the paper. But I think there are a lot of issues with how to content is shown to the reader that makes the paper's contributions fall flat. In its current state, I would recommend rejecting the paper, but if the authors address my concerns above, I believe I would lean more towards accept.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730704406742

thread_type:
official_review

<thread_object>
creator: reviewer_zt5v

<content>
summary:
The paper presents Episodic Memory for Cognitive Agents (EMCA), a framework designed to support memory retention and retrieval in  cognitive agents. EMCA models episodic memory using a graph-based structure that incrementally stores and organizes multimodal experiences—such as speech, vision, and non-verbal cues—without pre-training on specific scenarios. This approach enables agent to keep adding new experiences continuously from data. This supposedly allows for flexible temporal reasoning across different datasets. EMCA’s dynamic memory graph builds semantic and temporal connections, enabling context-aware retrieval and clustering of memories based on query relevance. The framework aims to improve task execution , and reasoning by recalling contextually significant past events. Empirical tests reported indicate that EMCA adapts to real-world data, demonstrating good recall in unpredictable settings.

strengths:
I think the paper is trying to address a very important problem of how can an autonomous agent keep memoring new experienes and the recall those flexibly based on the context, question or query. Specifcally I see two main strengths.  
Pretrained Models: The model builds on pretrained models that help with extraction of compenents, but this also means the system does not require pre-training on every specific scenario from scratch, which is a significant strength as it allows flexibility across contexts.
Dataset Diversity: The authors evaluated the system on multiple datasets, demonstrating a broad application range, although it should be noted that most of these datasets were developed by the authors.

weaknesses:
Despite tackling an important problem, the paper suffers from serious clarity and coherence issues that obscure its contributions and weaken its scientific rigor. The presentation is fragmented, key concepts are inadequately explained, and essential technical details are missing, all of which make it challenging to assess the model’s validity and potential impact. Specifically, the weaknesses include:
1. The authors claim EMCA encodes data in a way that resembles human memory, but there is no evidence or detailed explanation to support this claim from a neural encoding way. Such claims should be toned down. You should instead  emphasize on the 'what', 'where' 'when' organisation from a psychological perspective of episodic memory.  

2. Insufficient Motivation: The introduction section does not adequately establish the necessity of this system or why it improves upon existing learning frameworks for cognitive agents. Additional motivation for the need of an episodic memory for a cognitive agent would help contextualize EMCA's contributions.


3. Minimal Related Work Discussion: Essentialy the model is an encode and retrival model with some dynamic reorganistion. The related work section is sparse and lacks comparisons to key formal methods like Hopfield networks or other established models in episodic memory encoding and retrieval. A more rigorous comparison to established human memory models would also strengthen the paper.

4. Unclear Implementation and Integration Details: Although multiple models and methods are mentioned, the paper lacks a cohesive description of how these components integrate within the system. Critical details such as model architecture, parameter settings, and processing pipelines are absent, making it difficult to assess or replicate the work. A system architecture diagram, a table of key parameters, or pseudocode for the main processing pipeline would help.

Vague Statistical Estimation Methods: The paper mentions the use of statistical methods for estimating missing timestamps but does not specify which methods were used, leaving an important aspect of the framework unexplained.

Surface-level Comparison with Temporal and Knowledge Graphs: The comparisons with temporal and knowledge graph structures are brief and lack depth, offering limited insight into how EMCA differs from or improves upon these existing approaches.

Undefined Terminology and Variables: Certain terms and variables (e.g.,"key events", "location weight," "subjective temporal timescales") are introduced without sufficient explanation or definition, reducing clarity.

Overreliance on Custom Datasets: While the use of various datasets to evaluate EMCA is a strength, most of these datasets were developed by the authors, which could indicate potential biases in testing and validation.

Limited Explanation of Retrieval Policy: The retrieval policy and memory clustering mechanisms, while central to EMCA’s functionality, are described only briefly. A more detailed explanation would clarify how these mechanisms adapt to different query types and scenarios.

questions:
Statistical Estimation Methods for Timestamps: Which specific statistical methods are used for estimating timestamps in the absence of explicit temporal markers?

Union of Tacoustic and Tvoiced: In combining Tacoustic and Tvoiced, what is the methodology for performing this union? Is Tvoiced identical or related to another variable, such as St?

Format and Extraction of Visual Details: What is the format of the visual scene details (e.g., Vscene, Vplace, Vtime), and how are these extracted and integrated into the memory graph?

Defining Key Events and Hierarchical Organization: How are key events identified, and what hierarchical structure is used to organize these events?

Relation between Taudio and Tcombined: Is Taudio equivalent to Tcombined, or is there another relationship between these variables?

Task Categories for Text Summarization: How are text summaries grouped into broader task categories (e.g., meetings, lunches)? What criteria and process are used to define these categories?

Similarity Calculation in Equation 8: Equation 8 is intended to measure similarity between an event and multiple episodes, but it isn’t clear how it accomplishes this. Could you clarify how this calculation works?

Location Weight Definition: How is "location weight" defined, and how does it differ from location similarity in the model?

Temporal Parameter in Equation 14: In Equation 14, should the parameter be (t-k) instead of just t? If not, what purpose does the current form of the equation serve?

Meaning of "Agent Comprehends": In line 274, it says the "agent comprehends" something. Does this imply processing by a language model, and if so, could you clarify which model is used?

Definition of the Set Du: How is the set Du defined in the context of the framework?

Similarity Function in Line 283: Which similarity function is used in line 283, and what factors are considered?

Role of w and l Functions: In line 287, the w and l functions are mentioned. Could you elaborate on their roles within the memory retrieval mechanism?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732117378764

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Updated version of Introduction

comment:
Episodic memory, as introduced by Tulving (1972) [^1], represents the capacity to recall personal experiences embedded within specific temporal and spatial contexts. Unlike semantic memory, which holds general knowledge, episodic memory encompasses detailed information about events, integrating aspects such as time, place, characters, and context. Tulving’s framework organizes these components into cohesive episodes.
Drawing from this foundational concept, we propose a model where episodic memories are structured as a graph. Each episode \( \text{Episode}_i \) functions as a node:
$$
\[
\text{Episode}_i = \{ \mathbf{C}_i, \mathbf{T}_i, \mathbf{L}_i, \mathbf{e}_i \}
\]
$$
In this model,$$ \( \mathbf{C}_i \) $$stands for characters, $$\( \mathbf{T}_i \)$$ represents temporal markers, $$\( \mathbf{L}_i \)$$ signifies location, and $$\( \mathbf{e}_i \)$$ encapsulates events. Semantic edges $$\( \mathbf{S}(v_i, v_j) \)$$ connect nodes that share common elements, while temporal edges $$\( \mathbf{T}(v_i, v_j) \)$$ map the chronological flow of experiences. To enhance retrieval efficiency, we apply a dynamic clustering mechanism that organizes similar episodes based on both temporal and contextual similarities.

Our retrieval system supports three query types: "what" (contextual), "when" (temporal), and "where" (spatial), as outlined by Stephen et al., and Holland and Smulders (2011), enabling human-like memory recall. This is particularly valuable for applications in social companion robotics, aiding elderly or memory-impaired individuals.

For such a cognitive agent, the ability to recall episodic memories is essential for human cognition, linking personal experiences to specific temporal and spatial contexts. Existing memory models often struggle with continuous, time-series data, which limits their ability to simulate episodic recall effectively. Many of these systems fail to store dialogues as multimodal data, preventing them from capturing the rich, context-dependent nature of human memory. Additionally, most existing approaches store a single experience as one isolated episode and lack a mechanism for retrieving information across multiple experiences, hindering their ability to integrate knowledge over time.

Furthermore, current episodic memory systems are typically restricted to performing a specific, predefined task, limiting their flexibility and adaptability. In contrast, our system is designed to be more versatile, capable of handling a variety of tasks and dynamically adapting to new scenarios, making it far more suited for real-world applications that require memory integration across different contexts and time periods. 

By integrating multimodal data and time-related information into the episodic memory framework, our model extends experience memory localization, recommendation, and question answering. It provides a robust foundation for adaptable, scalable systems capable of operating without frequent retraining, applicable to real-world scenarios such as social companion robots and autonomous task planning systems.

### **Contributions:**
1. Temporal connections are managed without complex pattern learning, enabling adaptive reasoning and retrieval of subgraphs from past experiences.
2. The system incrementally stores and retrieves episodic memories, dynamically clustering them based on temporal and contextual affinities.
3. A multi-edge graph framework optimizes path traversals for dynamic memory retrieval and personalized recommendations across subjective timescales.
4. A new dataset is introduced to improve episodic memory question answering, enhancing the agent's ability to respond to queries based on past events.

Our model’s versatility is demonstrated through comparisons with existing systems that require retraining. It handles various dataset types, including visual, multimodal, and text-based data, and excels in temporal reasoning even without explicit timestamps, addressing complex memory retrieval tasks across diverse applications.

[^1]: Tulving, E. (1972). *Episodic and semantic memory*. In *Organization of Memory* (pp. 381-403). Academic Press.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732119094958

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Dataset Section

comment:
We propose a comprehensive dataset framework designed to evaluate and enhance episodic memory systems in artificial agents. This framework integrates multiple datasets, including a custom set of episodic questions based on the TV series *The Big Bang Theory*, spanning all nine seasons (181 episodes). The aim is to assess memory recall and narrative understanding in complex scenarios.

We introduce the *Agent Dataset*, a 10-episode time-series dataset created in Unity3D, where a virtual agent performs tasks and interacts with characters in realistic environments, simulating the role of companion robots. This dataset emphasizes the importance of multi-sensory inputs and task execution, challenging the agent to process and integrate information from *dialogues* and *visual cues* to maintain task order and achieve context-driven objectives.

Additionally, we adapted the **Ego4D dataset**, restructuring its activity sequences into simulated chronological episodes to address the original absence of time-series data—portraying an agent performing a series of activities over 30 days. We also combined group activity videos designed for active speaker recognition. This transformation enables episodic queries such as "Where did I place the agricultural tool on the last day of farming?", enhancing the ability to localize and retrieve temporal experiences effectively.

Together with the **PerLTQA** [du2024perltqapersonallongtermmemory] and **LLQA** [dolan-brockett-2005-automatically] datasets, which test essential episodic memory dimensions—**"what"** (context), **"when"** (time), and **"where"** (place)—this framework forms a robust benchmark for evaluating advanced episodic memory capabilities in AI systems.

**Data Annotation**: The data was carefully annotated to tag scene information and identify characters in dialogues, ensuring that the model could recognize character presence and understand related events. This included explicitly tagging scene details for location identification and differentiating characters present in the scene versus those mentioned. Events within dialogues were also meticulously annotated to capture key details, facilitating effective memory representation beyond simple summaries. Capturing these essential details is crucial for episodic memory tasks, as it allows the agent to recall past experiences accurately. Each episode was annotated with 10 *what*, *when*, and *where* questions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732119448095

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Implementation Details

comment:
Implementation Details
For event extraction within dialogues, we utilized a Transformer-based BART model initialized with pre-trained weights to effectively extract and summarize events within contextual boundaries. The architecture options included BARTBASE, featuring a 6-layer encoder-decoder with approximately 140 million parameters, and BARTLARGE, which has a 12-layer encoder-decoder and 400 million parameters. Both configurations use a hidden size of 1024 and a feed-forward filter size of 4096, with a fixed dropout rate of 0.1 across layers. We employed the Fairseq toolkit for training, using the **Adam optimizer** with a warmup strategy. Learning rates were set to $4 \times 10^{-5}$ for BARTBASE and $2 \times 10^{-5}$ for BARTLARGE, with a maximum batch token limit of 1100 tokens. Contrastive objectives were supported by a margin coefficient of 1, and hyperparameters for coherence and sub-summary objectives were tuned using a validation set. Our method demonstrated substantial performance improvements compared to publicly available models trained on datasets like **SAMSUM** and **DialogueSUM**.

For visual data processing, we used a **Vision Transformer (ViT)** as the vision encoder, specifically adapted for video frame analysis from the **MSR-VTT dataset**. The encoder processed $224 \times 224$ video frames, segmented into 16-sized patches and embedded into a 512-dimensional latent space. The 12-layer encoder, with a width of 768, was equipped with **LayerScale** (initialized at 0.1) for training stability. Advanced regularization techniques, including stochastic depth with a configurable \texttt{drop\_path\_rate}, were applied. The encoder was based on the "eva-clip-b-16" model, which proved effective in extracting detailed spatial and temporal features essential for multimodal tasks.
For models based on **LLaMA** that integrate vision and dialogue for character and place tagging, a multimodal configuration was used. **ViT** processed visual data, while **LLaMA** managed dialogue input. Training included cross-entropy loss for character tagging and contrastive loss for image-text alignment, incorporating **episodic memory** for QA tasks. The training process leveraged the **AdamW optimizer**, a dropout rate of 0.2, and a **cosine annealing scheduler** for efficient learning.

**Temporal tagging** was optimized with key hyperparameters for best performance: a maximum sequence length of 128, a batch size of 32, and a learning rate of $5 \times 10^{-5}$. A dropout rate of 0.1 was used to mitigate overfitting, and a weight decay of 0.01 improved generalization. The training process spanned 10 epochs to ensure sufficient learning while preventing overfitting.
For extracting dialogue from audio, the **Whisper-large model** was employed. This model was utilized for its robustness in transcribing and converting spoken content into text for further processing.

For text detection, the **TextSnake** model was trained on the **SCUT-CTW1500** dataset using **SGD with Momentum** as the optimizer. The architecture combined **ResNet** and **FPN\_UNet**, configured with a training batch size of 64 and 8 workers for data loading. The validation batch size was set to 1, with 4 workers and persistent workers enabled. Training was conducted over 200 epochs with validation checks every 10 epochs.

The **QA system** was built using a **BERT** model fine-tuned on concatenated datasets, including **SQuAD**, **Wikipedia**, and **Reddit**, to improve contextual comprehension. The hyperparameters included a learning rate of $1 \times 10^{-5}$, a maximum sequence length of 512, and a document stride of 512. The training batch size was 8, with gradient accumulation steps of 2 over 2 epochs. Mixed-precision training was used with `fp16` at the **O2 optimization level** for better efficiency. The final model outputs were stored in the `bart-squadv2` directory, without intermediate model saving.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732167086593

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We would like to sincerely thank all the reviewers for their insightful feedback and constructive comments. Your thoughtful suggestions have significantly contributed to improving the clarity and quality of our paper. We have carefully addressed each of the points raised and made the necessary revisions to enhance the presentation and strengthen the content of our work.  
We appreciate your time and effort in reviewing our manuscript, and we believe that the revisions made will help in making the paper more polished and comprehensive. Thank you again for your valuable input.  

In response to the feedback, we will revise the relevant sections and include an appendix section:  

1. Following are the sections we will be adding to the paper:  
   - **Dataset Section** containing details of the dataset used in this paper.  

2. **Appendix Section** containing:  
   - Detailed Methodology  
   - Implementation Details  
   - Additional Results  

**The code and dataset will be shared upon the acceptance of the paper to ensure proper access and usage.**
</content>
</thread_object>

</thread>
<thread>
creation date:
1732452355149

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Dear "Reviewers",

We sincerely thank you for your thoughtful and constructive feedback on our manuscript, . Your suggestions have been valuable in improving the quality and clarity of our work. We have uploaded a detailed manuscript with changes mentioned and have also added a detailed appendix section addressing all your concerns.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper addresses the problem of storing and retrieving multimodal past experiences with a graph-based memory structure. The memory graph can be dynamically updated and enables temporal reasoning. 

Reviewers agree that the datasets and results are interesting. However, reviewers raised significant concerns about the presentation of this work and find it hard to follow. The problem is not well motivated and the design choices are not backed up theoretically or empirically. Comparisons with reasonable baseline methods are missing to provide useful insights. 

The reviewers unanimously agreed that this paper should be rejected.