Venue: NAACL 2025

# MADial-Bench: Towards Real-world Evaluation of Memory-Augmented

Dialogue Generation

Junqing He1, Liang Zhu2, Rui Wang1, Xi Wang3, Reza Haffari4, Jiaxing Zhang1,5

###### Abstract

Long-term memory is important for chatbots and dialogue systems (DS) to create consistent and human-like conversations, evidenced by numerous developed memory-augmented DS (MADS). To evaluate the effectiveness of such MADS, existing commonly used evaluation metrics, like retrieval accuracy and perplexity (PPL), mainly focus on query-oriented factualness and language quality assessment. However, these metrics often lack practical value. Moreover, the evaluation dimensions are insufficient for human-like assessment in DS. Regarding memory-recalling paradigms, current evaluation schemes only consider passive memory retrieval while ignoring diverse memory recall with rich triggering factors, e.g., emotions and surroundings, which can be essential in emotional support scenarios. To bridge the gap, we construct a novel **Memory-Augmented Dialogue Benchmark** (**MADial-Bench**) covering various memory-recalling paradigms based on cognitive science and psychology theories. The benchmark assesses two tasks separately: _memory retrieval_ and _memory recognition_ with the incorporation of both passive and proactive memory recall data. We introduce new scoring criteria to the evaluation, including _memory injection_, _emotion support (ES) proficiency_, and _inturacy_, to comprehensively assess generated responses. Results from cutting-edge embedding models and large language models on this benchmark indicate the potential for further advancement. Extensive testing further reveals correlations between memory injection, ES proficiency, and intimacy1.

Footnote 1: Codes and data will be released in [https://github.com/hejunqing/MDial-Bench](https://github.com/hejunqing/MDial-Bench).

1International Digital Economy Academy, Shenzhen, China

2Southern University of Science and Technology, Shenzhen, China

3University of Sheffield, Sheffield, U.K

4Monash University, Melbourne, Australia

5Lion Rock AI Lab, Hong Kong, China hejunqing2013@gmail.com

9902

April 29 - May 4, 2025 \(\copyright\)2025 Association for Computational Linguistics

recall, highlighting the role of long-term memory in emotion regulation, instead of only serving as the common ground between individuals for conversation [10, 23, 45]. Long-term memory is a core component for emotion regulation [6]. Indeed, by recalling a happy memory, people can mitigate the effect of a negative mood state [8]. However, current MADS only leverage memory as common ground in conversations, while overlooking the potential for emotional support that requires proactive memory recall.

To approach a practical and comprehensive evaluation of MADS, we introduce **M**emory-**A**ugmented **Dialogue **Benchmark** (**MADialBench**) to evaluate generated responses with human-centred aspects justified by cognition science and psychology theories. MADial-Bench represents memories from emotions, scenes and other cues as states. The two-stage theory is adopted for separate assessment in MADial-Bench to imitate the human memory recall procedure. The dataset enables the evaluation of two recall mechanisms: _Passive recall_, searching memories when the user mentions a specific event or item, which provides common ground in conversation; and _Proactive recall_, retrieving memories based on user emotion or scene as per psychological strategies, to achieve interpersonal emotional regulation. The dialogues in the benchmark are refined by humans with high emotional intelligence (EI) to achieve the goal. We also designed a set of guidelines based on emotional supporting (ES) theories [7, 9] to measure the aspect of ES proficiency. It is the first multi-recall and multi-stage memory-aware emotional dialogue generation benchmark. The paper highlights the evaluation of the retrieval accuracy, memory recognition & injection ability, and comprehensive assessment of generated responses, e.g. language style, ES proficiency and intimacy. After extensive experiments on LLMs, we discovered memory injection improves intimacy between MADS and users. There is still a gap between the most advanced models and human performance. Our main contributions are three-fold:

* We construct a delicate memory-augmented dialogue benchmark, called MADial-Bench. It is the first multi-recall multi-stage memory augmented dialogue system benchmark guided by cognitive science theory.
* It is the first work that introduces diverse memory-aware rating aspects with clear requirements for each point, containing memory injection, ES proficiency, intimacy et al. We also discover the relation between performance reported at these dimensions.
* We conduct extensive experiments on the strongest LLMs and embedding models. Results show that LLMs can perform higher naturalness and style coherence with careful prompting but lag behind humans in memory recognition and dialogue generation.

## 2 Related Works

Memory recall is an information retrieval process from people's past experience, which also acquaints them with past events [1]. Two main relevant theories on the recall process are the two-stage theory and the theory of encoding specificity. They are essential to explain the underlying assumption and mechanisms in the process of memory recall [1].

### Two-Stage Process Theory of Memory Recall

The two-stage theory states that the process of memory recall begins with a search and retrieval process, and then a decision or recognition process where the correct information is chosen from what has been retrieved [1, 23]. As summarized by Ezeh et al., recall works by a two-phase process:

1. A search through memory for something that

Figure 1: Memory Augmented Dialogue System with Emotion Support based on two-stage theory.

 

might satisfy the search criteria, referred to as memory recall in this paper.
2. A comparison of recalled memory with the detail of which is being sought, defined as memory recognition in the paper.

Researcher Tarnow provided the first direct evidence of two stages in free recall and corresponding estimates of working memory capacity. He found the two stages can be seen directly in sequential free recall distributions and the estimate of the capacity of working memory at 4-4.5 items.

### Encoding Specificity Theory

The encoding specificity theory, a cognitive psychology theory, explains the essential condition for effective information encoding to and recall from memory. Tulving and Thomson contended the essence of similar or shared conditions during information encoding and memory retrieval for effective memory retrieval. Typical conditions are physical location or surroundings, as well as the mental or physical state of the individual. This theory plays a significant role in both the concept of context-dependent memory and the concept of state-dependent memory. This theory inspires the design of memory with meta information and the proposal of a multi-recall paradigm in this paper for conversational AIs.

### Evaluation of Memory-Augmented Dialogue Systems

Many remarkable MADS have been proposed to build virtual agents and other applications Zhong et al. (2023); Wang et al. (2023); Liu et al. (2023); Zhang et al. (2024); Hou et al. (2024); Packer et al. (2024). Theses MADS were evaluated in two approaches. Given a memory pool and a dialogue context, the retrieval accuracy, response correctness and contextual coherence are scored, representing the first assessment method Zhong et al. (2023); Liu et al. (2023). This kind of evaluation is widely used in MADS that summarise events with emotion in dialogues as long-term memory and recall memories using a search module like embedding models. The test set only covers the passive memory recall triggered by users, omitting the natural proactive memory search in conversations for emotional regulation.

Another method adopts long-context language modelling (LM) tasks to measure memory ability, using novels and academic papers datasets like PG22 Wang et al. (2023) and Arxiv Gao et al. (2020). These language modelling tasks differ from real conversational scenarios, especially with using language perplexity and classification accuracy metrics. Overall, it is noteworthy that both evaluation methods ignore the value of emotional support in memory-augmented dialogue systems.

### Evaluation of Emotional Support DS

As emotional support is an essential function of conversation, DS with empathy were rapidly developed and researched, including ED Rashkin et al. (2019), ESC Liu et al. (2021), and other advanced work Sabour et al. (2022); Tian et al. (2022); Zhou et al. (2018). They commonly score Empathy, Relevance and Fluency for each response. We craft Emotional Improvement, Coherence and Naturalness based on these aspects. Liu et al. (2021) uses Identification, Comforting and Suggestion in extra to measure the emotional regulation steps in advice-seeking scenarios. These metrics are not suitable for daily dialogues and are merged into one aspect in this paper: ES Proficiency. We gauge this aspect according to hand-crafted well-written guidance based on psychological theories. However, the metrics above don't inspect long-term memory augmentation and intimacy. Therefore, we add the Memory-injection Ability and Intimacy to assess how much a conversational AI introduces correct memory into dialogue and resembles a close friend.

Another difference from metrics in previous studies lies in the scoring standard, which adopts the Likert scale. The requirement of each point in our metrics is strictly listed with minimum ambiguity. For instance, Naturalness contains 3 sub-aspects corresponding to 3 points, where each point can be rated from 0 to 1 and summed up to 3 at most. The difference between MADial and previous works is listed in Table 1.

\begin{table}
\begin{tabular}{c|c c c} \hline \hline Metrics & ED & ESC & Ours \\ \hline Fluency/Naturalness & ✓ & ✓ & ✓ \\ \hline Empathy & ✓ & ✗ & ✓ \\ \hline Memory-Injection & ✗ & ✗ & ✓ \\ \hline ES Proficiency & ✗ & ✓ & ✓ \\ \hline Overall/Intinacy & ✗ & ✓ & ✓ \\ \hline \end{tabular}
\end{table}
Table 1: Evaluation Aspects of responses in representative research. ED is short for EmpatheticDialogues Rashkin et al. (2019), and ESC is short for Emotional Supporting Conversation Liu et al. (2021).

 

MADial-Bench

We simulated long-term dialogues between two users (a boy named Bart and a girl named Lisa) and a virtual assistant with memory information at various time points, in the form of event summaries. We evaluate the dialogue between children for their simple expression and direct emotional exposure. By sampling different topics and scenarios, we generated 171 historical memories and 160 dialogues in 1474 turns. The final MADial-Bench was created through prompt-based generation using GPT-4, followed by multiple manual refinement and revision rounds.

### Overview

MADial-Bench encompasses two tasks related to conversational memory recall: proactive recall of emotional memories and passive recall of objective facts. The proactive recall involves four situations: happy, sad, anxious, and disappointed emotion of the user. On the other hand, the passive recall of objective facts encompasses activities, objects, and social relationships. Social relationships are further categorized into positive and negative relationships. The conversation turns range from 8 to 12, with average statistics illustrated in Figure 2. Considering the two-stage nature of memory utilization, we divide the testing of a memory dialogue system into two stages: memory recall and response generation with memory recognition. To support diverse recall paradigms and mimic human memory, a memory in the benchmark contains time, emotion, scene and event, as illustrated in Table 3.

### Data Contruction

Since the memory-recalling procedure is related to emotions and other surroundings, we first curated 18 emotional categories according to cognitive science and psychological theories Ekman (1992); PLUTCHIK (1980); Sabour et al. (2024). Then we asked a psychology major graduate to decide what kind of memories should be recalled to regulate certain emotions for proactive memory recall. As discovered by psychologists through emotion recognition tasks, various emotions have been developed during people's childhood Tonks et al. (2007); Baron-Cohen et al. (2010). For example, happiness, sadness, anger, disgust, fear, anxiety, disappointment and so on. Finally, 4 emotions including happiness, sadness, anxiety, and disappointment were selected for passive memory recall since they are linked with past events and expressed by children under 10 Baron-Cohen et al. (2010). We also define 5 scenes to describe the topic, physical state and surroundings of the users, which contain Preferences, Activity, Disease, Emotions and Others. The definition of each scene is in the Table 9 in the Appendix.

We hired a psychology expert to develop a set of guidelines for emotionally supportive dialogue based on psychological literature Hill (2020); Austin et al. (2018); Mayer et al. (2016). These guidelines outline how conversations should be conducted in different situations to provide appropriate emotional support to users. The guidance is used to construct dialogues and score the ES proficiency later.

GPT42 was used to generate dialogues and historical memories with specific topic lists and user profiles. For each case, a topic and a user are sam

\begin{table}
\begin{tabular}{c c c c} \hline Subset & \#Dialogues & Avg Turn & Avg Len. \\ \hline All & 160 & 9.21 & 44.39 \\ Proactive & 100 & 9.38 & 45.80 \\ Passive & 60 & 8.93 & 41.93 \\ \hline \end{tabular}
\end{table}
Table 2: Statistical details of data. Avg Len. is short for average length per turn.

Figure 2: Data distribution of each task and category.

\begin{table}
\begin{tabular}{l l l l} \hline Time & Emotion & Scene & Event \\ \hline
2023-12-5 & Happy & Activity & Bart danced at the Christmas party and felt very happy. \\ \hline \end{tabular}
\end{table}
Table 3: A memory sample.

 

pled for LLMs to generate a dialogue with a related memory and inject the memory into dialogue in the sampled turn. After generating the initial version of the dialogue and memory data, we use an Embedding model3 to rank all the memory collections given the context. Then, we select appropriate memories from the top 20 memories as a golden set for each dialogue. When the original memory in the dialogue is not in the golden set of relevant memories, we modify the introduced memory and dialogue content afterwards. All the responses are then checked and modified manually according to the ES skill guidance. These human-crafted responses serve as a reference.

Footnote 3: We used the best one according to our experiments.

### Task Definition

In this section, we introduce the two tasks tested in MADial-Bench, corresponding to phrases in the two-stage theory. Analog to Retrieval Augmented Generation (RAG), memory recall resembles the retrieval stage (finding relevant chunks/memories), while memory recognition/response generation mirrors the generation stage (producing output based on the retrieved chunks/memories).

#### 3.3.1 Memory Recall

For each dialogue \({\cal D}_{i}\), there is a corresponding golden memory set, denoted as \(S_{i}\), which contains the most suitable memories for the current dialogue. It is a relevant memory list of length \(l\) in descending suitability order, where \(l\) ranges from 1 to 7. Before generating a response, the memory recalling task is to retrieve the golden memory set \(S_{i}\) from the all memories in the memory bank for a given dialogue \(D\). The model responsible for memory retrieval is denoted as M\({}_{1}\).

#### 3.3.2 Memory Recognition and Response Generation

After retrieving a subset of memories, the LLM needs to select the most appropriate memory for the dialogue, this process is called memory recognition in two-stage theory. Given a sample \({\cal D}_{i},P_{i},{\cal R}_{i}\), where \(P_{i}\) is the candidate set of memories, \({\cal R}_{i}\) is the reference response, the target is to generate a response that is appropriate and semantically similar to \({\cal R}_{i}\). The model responsible for memory selection and response generation is denoted as M\({}_{2}\). Based on psychological research Austin et al. (2018), we build the criteria for selecting and using memories for different memory recalls, sub-types and factors like emotions, relationships, events, and objects. The detailed criteria are shown in Table 10 in the Appendix.

This task has 3 settings that vary with \(P_{i}\):

* Setting 1 directly provides the first golden memory \(m_{*}\) as \(P_{i}\), which contains a single memory. It is the ideal situation to investigate the memory injection ability of models.
* Setting 2 adds 4 sampled negative memories to the golden one and shuffles them as \(P_{i}\). It ensures the existence of golden memory in \(P_{i}\) and investigates both the memory recognition and injection abilities of models.
* Setting 3 provides retrieval results from the best embedding model in setting 1 as \(P_{i}\). It is the most realistic setting and the results may be similar to practical performance.

#### 3.3.3 Intimacy and Memory Usage

In this experiment, we assess the benefit of memory by comparing the responses generated by each LLM with and without memory in Setting 3. Five annotators choose the more intimate response or select "tie" if they can't decide. Additionally, in the human evaluation for Task 2, they pick the most intimate response from all candidates.

## 4 Experiments

In this section, we detail the evaluation setups of top models, embedding models for memory retrieval, and LLMs for memory-injected response generation in the task of memory recognition and response generation. The response evaluation includes the use of both human evaluation measures over rich aspects and commonly used automatic evaluation approaches.

### Memory Recall Task

#### 4.1.1 Embedding Models

We conduct experiments on the following embedding models for English: Jina embedding2-base-en Gunther et al. (2023), GTE-base-env.15 Li et al. (2023), BGE-M3 Dense Chen et al. (2024), and OpenAI text-embedding-3-large 4. Chinese embedding models tested are: Acge text-embedding Kusupati et al. (2022), Stella5, BGE-M3 Dense and Colbert, and Dmeta6 and OpenAI 

text-embedding-3-large. These embedding models are top-ranked in the retrieval tasks in ETEB Leaderboard 7(Muennighoff et al., 2022). We do not include model parameters exceeding 7B due to the slow inference.

Footnote 6: [https://github.com/google-learning/blob-master/tree-master-models-with-the-best-paradigm.html](https://github.com/google-learning/blob-master/tree-master-models-with-the-best-paradigm.html)

#### 4.1.2 Metrics

For a comprehensive evaluation of the memory recall performance of the embedding model M\({}_{1}\), we measured Mean Average Precision (MAP), Mean Reciprocal Rank (MRR), Normalized Discounted Cumulative Gain (nDCG), Recall, Precision and calculated their mean across different scopes. The detailed metrics are illustrated in Table 4. MRR places greater emphasis on the top-ranked position, focusing more on the golden summary \(m_{*}\). MAP uniformly considers all relevant summaries, while nDCG accounts for the relevance and positions of different memories.

### Memory Recognition and Response Generation

#### 4.2.1 Models

We conduct tests on various models of differing sizes within the latest English and Chinese series. The models evaluated in this study include Qwen2-7B-Instruct Yang et al. (2024), GPT-4-turbo Achiam et al. (2023), Doubao-Character-32k(0528), GLM-4-05208 Zeng et al. (2022), and Ziya-Character-0606, a fine-tuned model with over 30K character dialogues based on the Ziya2-13B-Base Gan et al. (2024). For English version, we tested Llama3-8B-Instruct, Llama3-70B-Instruct Dubey et al. (2024), GPT-4o, and GPT-4 turbo. They all ran with zero-shot prompting.

Footnote 8: [https://bigmodel.cn/dev/api/normal-model/glm-4](https://bigmodel.cn/dev/api/normal-model/glm-4)

#### 4.2.2 Metrics

BLEU Papineni et al. (2002), Rouge-L Lin (2004) and BertScore Zhang et al. are employed to assess the literal and semantic overlap between the generated response and the reference response. Considering the limitations of static metrics, we further utilize human annotation to evaluate the candidates in various aspects. Each instance to score includes a mixture of a reference answer and five candidate answers to facilitate objective comparisons. The human evaluation is in two stages (1) aspect-aware scoring and (2) the pick-the-best paradigm. The definitions of aspects are listed as follows:

* **Naturalness** inspects the language quality of candidate responses, including grammar, pronoun usage, context correlation and oral expression.
* **Style Coherence** measures how much output aligns with the character description in prompts. Specifically, the assistant is also evaluated by whether a proper tone and easy words are used to talk with a child.
* **Memory-injection Ability** examines how well the LLM recognise the correct memory and introduces it into dialogues without quotation errors.
* **ES Proficiency** measures how skill the model is in ES, considering at most three turns of dialogues. The scoring is instructed by detailed guidance covering different dialogue situations.
* **Emotion Improvement** measures whether the response, the literal expression is empathetic and positive or not. A high emotion improvement score does entail high ES proficiency.
* **Intimacy** is used in side-by-side comparisons and pick-the-best among candidates. It is an overall feeling of how much a model behaves like a close friend.

The detailed requirements are in Appendix C.

## 5 Results and Discussion

In this section, we discuss the performance of models in each task and aspect. The surprising relations between aspects are also dug out. Later on, automatic evaluation results also show their disability in assessing LLM outputs in this benchmark.

### Memory Recall

The experimental results of applying various embedding methods in Table 4 show that the OpenAI embedding model achieves the best performance on both English and Chinese datasets. The bilingual BGM-M3 performs the best among open-source models on the English testbed, while Acge is the top open-source model for the Chinese testbed in our setting.

However, regardless of the language, the retrieval performance on our MADial-Bench is far from satisfying. Even the best embedding model, OpenAI, does not exceed 60% @1 and struggled at 62% @10 in the final average, highlighting the difficulty of retrieving appropriate memories in conversation. Solely text similarity retrieval is inadequate for the memory recall process in dialogue systems.

 



### Memory Recognition and Response Generation

Table 5 depicts LLMs' average scores of each aspect on all three tasks in the Chinese version.

For the overall performance, human-written references scored higher than models in most aspects except emotion improvement from GPT4-Turbo. Therefore, no LLMs can surpass humans in average performance. There is space for improvement in aspects including Naturalness, Style Coherence, Memory Injection and ES skills. GPT4-Turbo is the best model for tasks 1 and 2, while GLM-4 is superior in setting 3.

For memory injection, we observe a substantial gap between LLMs and human written reference, in which humans can introduce suitable memory into response correctly and LLMs can only achieve half of the scores in setting 3 except GLM-4. The models in setting 1 are often better performed than in settings 2 and 3. Specifically, given the golden memory, models can achieve scores over 2.5 but decrease dramatically when blended with unrelated memories, except for Ziya-Character, which performed badly in all tasks for the sake of smaller parameter size. Results of GLM-4 seem robust at the cost of introducing too much memory into response and harming other aspects like style coherence. It reveals that the _strongest LLMs also struggle in memory recognition_.

A significant improvement in Naturalness from setting 1 to setting 2 and 3 implies that all models benefit from the emphasis on the Naturalness in prompts. Similar phenomena also exist in Style Coherence. Among all the models, Doubao excels in these two aspects after the prompt strengthening. Since the Ziya-Character was fine-tuned with roleplay and character datasets, it can produce natural and stylish responses steadily. But it was beaten by larger models like Doubao, GPT4-Turbo and Qwen2-72B with prompt engineering.

When addressing setting 1 where golden memory was provided, all the models had over 2.2 scores in ES Proficiency. However, while meeting a realistic setup (i.e., setting 3), scores on ES Proficiency drop significantly. Therefore, ES Proficiency is related to Memory Injection ability. To investigate the correlation, we draw the distribu

tion of ES Proficiency grades with varied memory-injection scores cross all LLMs in Figure 3 We observe that _higher memory-injection ability does lead to higher ES proficiency_. A 3.0 memory injection score can lead to a 3.0 ES score in 61%, which emphasizes the importance of memory-augmented ability in ES.

Surprisingly, the Emotional Improvement expression from GPT4-Turbo, Qwen2-72B and GLM-4 can exceed that of humans. As per qualitative analysis, these models maintain a warm, energetic tone, encouraging users during low-emotion moments, whereas human responses sometimes lack positivity and encouragement.

### Intimacy and Memory Usage

The side-by-side evaluation results on intimacy of responses with and without memory are presented in Table 6. As the table indicates, responses with memory injection are almost (no less than 69.4% probability) better than those without memory. The win rate grows higher as the model becomes stronger. For example, GPT4-Turbo can produce 61.9% more intimate response with memory while Ziya-Character can only produce a win rate of 47.5%. It is attributed to their stronger memory injection ability and ES Proficiency. It implies that _intimacy performance is highly related to memory injection ability._

We also investigate people's preferences by asking 5 annotators to select the best responses in setting 3 and see whether they are equipped with memory. Results of the ratio of containing memory in voted responses are listed in Table 7. On average, 73% best responses are with memory, which shows people prefer memory-aware replies, which naturally provide familiarity and emotional support.

\begin{table}
\begin{tabular}{c|c|c|c} \hline Model & Win & Tie & Lose \\ \hline GPT4-Turbo & 61.9 & 28.8 & 9.4 \\ Quen2-72B & 56.3 & 28.8 & 15.0 \\ Doubao & 44.4 & 34.4 & 21.3 \\ GLM-4 & 55.0 & 19.4 & 25.6 \\ Ziya-Character & 47.5 & 21.9 & 30.6 \\ \hline \end{tabular}
\end{table}
Table 6: Side-by-side evaluation results on the responses intimacy with and without memory.

Figure 3: The relation between memory injection score and ES Proficiency. The probability of a 3.0 ES score grows as memory injection scores increases.

 



### Automatic Evaluation

The automatic results in the Chinese version are displayed in Table 8. The full results are in Appendix 11. We can observe that GLM-4 achieved the highest semantic similarity while Qwen2-72B earned the highest Rouge-L. However, the gap between Qwen2-72B and GLM-4 is marginal. ZiyaCharacter got the highest results in Dist-1. It may be attributed to its training on diverse character corpora via supervised fine-tuning. However, it is opposite to the human evaluation-based conclusion that GPT4-Turbo is the best and Qwen2-72B ranked second. Therefore, _automatic evaluation fails to measure the results from LLMs_. We also tried judging with GPT4 using various promptings and found its scores were unreliable since it forgot the criteria and gave faulty reasoning.

## 6 Conclusion

Memory-augmented Dialogue Systems (MADS) are popular LLM applications, but no systematic memory evaluation benchmarks exist. This paper introduces MADial-Bench, a comprehensive bilingual benchmark that extends beyond passive recall to include both proactive and passive memory re-call tasks. Grounded in cognitive science, it assesses MADS performance across memory recall, recognition and injection. Novel human evaluation metrics are proposed, such as memory-injection ability, ES proficiency, and intimacy. We test extensive competitive LLMs in various tasks and settings with human and automatic evaluation with meaningful insights, including 1) unsatisfactory memory recall performance of top embedding models. 2) relevance between memory injection ability and ES proficiency. Models with higher memory injection ability can be more intimate. 3) a substantial gap between the most powerful LLMs and human reference in memory injection and overall performance.

## Acknowledgments

This paper is kindly supported by Qi Wei, Xiaolin Qi, Chao Song, Yun Chen, Qianguo Sun and Bing Wang. Thanks for their contribution. The program is also sponsored by China Merchants Group.

## Limitation

This paper only considers the retrieval-based memory-augmented chatbots and dialogue systems. The experiments focused on bilingual data, hoping the conclusion and insights could be popularized in other languages. The human evaluation is hard to conduct for a wide range of results with high consistency, which will introduce stronger AI products with special prompting in the future. As we have human evaluation data, in future we can use LLM-as-ajudge for the aspect-based evaluation, and then evaluate the performance of the LLM-as-a-judge with respect to the human evaluation data.

## References

* Achiam et al. (2023) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. _arXiv preprint arXiv:2303.08774_.
* Anthropic (2024) Anthropic. 2024. Claude 3.5 sonnet.
* Hall et al. (2018) Kimberly A Arditte Hall, Rudi De Raedt, Kiara R Timpano, and Jutta Joormann. 2018. Positive memory enhancement training for individuals with major depressive disorder. _Cognitive Behaviour Therapy_, 47(2):155-168.
* Austin et al. (2018) Elizabeth Justin, Donald H Saklofske, and Martin M Smith. 2018. Development and validation of two short forms of the managing the emotions of others (meos) scale. _Frontiers in Psychology_, 9:365302.
* O'Hagan et al. (2018)

\begin{table}
\begin{tabular}{c c c c c c} \hline Annotation & 1 & 2 & 3 & 4 & 5 & AVG \\ \hline With Mem (\%) & 70 & 90 & 55 & 80 & 70 & 73 \\ \end{tabular}
\end{table}
Table 7: The memory injection rate in best candidates voted by humans.

\begin{table}
\begin{tabular}{c|c c c c} \hline \hline Models & BertScore & Rouge-L & BLEU-1 & Dist-1 \\ \hline  & \multicolumn{4}{c}{Setting 1(\%)} \\ \hline Doubao & 72.08 & 22.16 & 22.82 & 74.72 \\ GPT4-Turbo & 72.82 & 23.17 & 23.76 & 76.13 \\ Qwen2-72B & 73.07 & **25.08** & 29.14 & 80.24 \\ GLM-4 & **74.17** & 24.98 & 26.40 & 74.91 \\ Ziya-Character & 69.58 & 19.68 & 20.74 & **88.45** \\ \hline  & \multicolumn{4}{c}{Setting 2(\%)} \\ \hline Doubao & 70.54 & 20.49 & 22.28 & 79.89 \\ GPT4-Turbo & 69.08 & 21.26 & 22.50 & 75.80 \\ Qwen2-72B & 71.67 & **23.04** & **26.72** & 79.37 \\ GLM-4 & **73.19** & 22.37 & 22.11 & 72.69 \\ Ziya-Character & 60.73 & 19.68 & 21.74 & **86.38** \\ \hline  & \multicolumn{4}{c}{Setting 3(\%)} \\ \hline Doubao & 70.39 & 20.26 & 21.70 & 81.26 \\ GPT4-Turbo & 71.65 & 21.22 & 22.32 & 75.91 \\ Qwen2-72B & 71.68 & **22.73** & **26.15** & 78.64 \\ GLM-4 & **73.12** & 22.39 & 22.11 & 72.90 \\ Ziya-Character & 68.63 & 18.24 & 19.34 & **89.18** \\ \hline  & \multicolumn{4}{c}{} \\ \end{tabular}
\end{table}
Table 8: Automatic Evaluation Results on Chinese testbed. Full results in Table 11. The automatic measurements are inconsistent with human evaluation.



Harry P Bahrick. 1970. Two-phase model for prompted recall. Psychological Review, 77(3):215.
* Baron-Cohen et al. [2010] Simon Baron-Cohen, Ofer Golan, Sally Wheelwright, Yael Granader, and Jacqueline Hill. 2010. Emotion word comprehension from 4 to 16 years old: A developmental survey. Frontiers in evolutionary neuroscience, 2:109.
* Bermingham et al. [2013] Douglas Bermingham, Robert D Hill, Dan Woltz, and Michael K Gardner. 2013. Cognitive strategy use and measured numeric ability in immediate-and long-term recall of everyday numeric information. _PloS one_, 8(3):e57999.
* Bubeck et al. [2023] Sebastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. 2023. Sparks of artificial general intelligence: Early experiments with gpt-4. _arXiv preprint arXiv:2303.12712_.
* Chen et al. [2024] Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024. Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. _Preprint_, arXiv:2402.03216.
* Dubey et al. [2024] Abhimanyu Dubey et al. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783.
* Ekman [1992] Paul Ekman. 1992. An argument for basic emotions. Cognition and Emotion, 6(3-4):169-200.
* Engen and Anderson [2018] Haakon G Engen and Michael C Anderson. 2018. Memory control: A fundamental mechanism of emotion regulation. Trends in Cognitive Sciences, 22(11):982-995.
* Ezeh et al. [2018] Leonard N. Ezeh, Chukwuemeka A. F. Okoye, Chukwuwemeka E. Etodike, and Cynthia C. Udeze. 2018. Theoretical correlation of ageing, mental activity and memory recall: Implication for security personnel. _European Journal of Special Education Research_, 0(0).
* Gan et al. [2024] Ruyi Gan, Ziwei Wu, Renliang Sun, Junyu Lu, Xiaojun Wu, Dixiang Zhang, Kunhao Pan, Junqing He, Yuanne Tian, Ping Yang, Qi Yang, Hao Wang, Jiaxing Zhang, and Yan Song. 2024. Ziya2: Data-centric learning is all llms need. _Preprint_, arXiv:2311.03301.
* Gao et al. [2020] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. 2020. The pile: An 800gb dataset of diverse text for language modeling. _arXiv preprint arXiv:2101.00027_.
* Gunther et al. [2023] Michael Gunther, Jackmin Ong, Isabelle Mohr, Alaeddine Abdessalem, Tanguy Abel, Mohammad Kalim Akram, Susana Guzzman, Georgios Mastrapas, Saba Sturua, Bo Wang, Maximilian Werk, Nan Wang, and Han Xiao. 2023. Jina embeddings 2: 8192-token general-purpose text embeddings for long documents. _Preprint_, arXiv:2310.19923.
* Hill [2020] Clara E Hill. 2020. _Helping skills: Facilitating exploration, insight, and action_. American Psychological Association.

* Hou et al. [2024] Yuki Hou, Haruki Tamoto, and Homei Miyashita. 2024. "my agent understands me better": Integrating dynamic human-like memory recall and consolidation in llvm-based agents. In _Extended Abstracts of the CHI Conference on Human Factors in Computing Systems_, CHI '24. ACM.
* Kusupati et al. [2022] Aditya Kusupati, Gustavya Bhatt, Aniket Rege, Matthew Wallingford, Aditya Sinha, Vivek Ramanujan, William Howard-Snyder, Kaifeng Chen, Sham Kakade, Prateek Jain, et al. 2022. Matryoshka representation learning. Advances in Neural Information Processing Systems, 35:30233-30249.
* Li et al. [2023] Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. 2023. Towards general text embeddings with multi-stage contrastive learning. _arXiv preprint arXiv:2308.03281_.
* Lin [2004] Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In _Text Summarization Branches Out_, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
* Liu et al. [2023] Lei Liu, Xiaoyan Yang, Yue Shen, Binbin Hu, Zhiqiang Zhang, Jinjie Gu, and Guannan Zhang. 2023. Think-in-memory: Recalling and post-thinking enable llms with long-term memory. _Preprint_, arXiv:2311.08719.
* Liu et al. [2021] Siyang Liu, Chujie Zheng, Orianna Demasi, Sahand Sabour, Yu Li, Zhou Yu, Yong Jiang, and Minlie Huang. 2021. Towards emotional support dialog systems. In _Proceedings of the 59th Annual Meeting of the Association for memorydornexion Tanguristics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 3469-3483.
* Mayer et al. [2016] John D Mayer, David R Caruso, and Peter Salovey. 2016. The ability model of emotional intelligence: Principles and updates. _Emotion review_, 8(4):290-300.
* McKinley et al. [2017] Geoffrey L McKinley, Sarah Brown-Schmidt, and Aaron S Benjamin. 2017. Memory for conversation and the development of common ground. _Memory & cognition_, 45:1281-1294.
* Muennighoff et al. [2022] Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. 2022. Metb: Massive text embedding benchmark. _arXiv preprint arXiv:2210.07316_.
* OpenAI [2022] OpenAI. 2022. Introducing chatgrt.
* Packer et al. [2024] Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. 2024. MemgtpT: Towards llms as operating systems. _Preprint_, arXiv:2310.08560 

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318. Cited by: SS5.
* a general psychoevolutionary theory of emotion. In The Robert Plutchik and Henry Kellerman, editors, Theories of Emotion, pp. 3-33. External Links: ISBN 978-3-642-46558-0 Cited by: SS1.
* H. Rashkin, E. Michael Smith, M. Li, and Y. Boureau (2019)Towards empathetic open-domain conversation models: A new benchmark and dataset. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 5370-5381. Cited by: SS1.
* S. Sabour, S. Liu, Z. Zhang, J. M. Liu, J. Zhou, A. S. Sunarya, J. Li, T. M. C. Lee, R. Mihalcea, and M. Huang (2024)Emobench: Evaluating the emotional intelligence of large language models. Preprint arXiv:2402.12071. Cited by: SS1.
* S. Sabour, C. Zheng, and M. Huang (2022)Cem: commonsense-aware empathetic response generation. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 36, pp. 11229-11237. Cited by: SS1.
* B. Sarah and C. Duff Melissa (2016)Memory and common ground processes in language use. Topics in cognitive science, pp. 722-736. Cited by: SS1.
* E. Tarnow (2016)First direct evidence of two stages in free recall and three corresponding estimates of working memory capacity. arXiv preprint arXiv:1605.05685. Cited by: SS1.
* Z. Tian, Y. Wang, Y. Song, C. Zhang, D. Lee, Y. Zhao, D. Li, and N. L. Zhang (2022)Empathetic and emotionally positive conversation systems with an emotion-specific query-response memory. In Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 6364-6376. Cited by: SS1.
* J. Tonks, W. Huw Williams, I. Frampton, P. Yates, and A. Slater (2007)Assessing emotion recognition in 9-15-years olds: preliminary analysis of abilities in reading emotion from faces, voices and eyes. Brain injury21 (6), pp. 623-629. Cited by: SS1.
* H. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Roziere, N. Goyal, E. Hambro, F. Azhar, et al. (2023)LIMA: open and efficient foundation language models. arXiv preprint arXiv:2302.13971. Cited by: SS1.
* E. Tulving and D. M. Thomson (1973)Encoding specificity and retrieval processes in episodic memory. Psychological Review80, pp. 352-373. Cited by: SS1.
* Weizhi Wang, L. Dong, H. Cheng, X. Liu, X. Yan, J. Gao, and F. Wei (2023)Augmenting language models with long-term memory. PreprintarXiv:2306.07174. Cited by: SS1.
* M. J. Watkins and J. M. Gardiner (1979)An appreciation of generate-recognize theory of recall. Journal of Verbal Learning and Verbal Behavior18 (6), pp. 687-704. Cited by: SS1.
* R. Weissenborn and T. Duka (2000)State-dependent effects of alcohol on explicit memory: the role of semantic associations. Psychopharmacology. Cited by: SS1.
* An Yang, Baosong Yang, Binyuan Hui, B. Zheng, B. Yu, C. Zhou, C. Li, C. Li, D. Liu, F. Huang, G. Dong, H. Wei, H. Lin, J. Tang, J. Wang, J. Tang, J. Wang, J. Tu, J. Zhang, J. Ma, J. Yang, J. Xu, J. Zhou, J. Bai, J. He, J. Lin, K. Xue, M. Li, C. Xue, N. Ni, P. Zhang, P. Wang, R. Peng, R. Men, R. Gao, R. Lin, S. Wang, S. Bai, S. Tan, T. Zhu, T. Li, T. Liu, W. Ge, X. Deng, X. Zhou, X. Ren, X. Zhang, X. Wei, X. Ren, X. Liu, Y. Fan, Y. Yao, Y. Zhang, Y. Wan, Y. Chu, Y. Liu, Z. Cui, Z. Zhang, Z. Guo, and Z. Fan (2024)Qwen2 technical report. PreprintarXiv:2407.10671. Cited by: SS1.
* A. Zeng, X. Liu, Z. Du, Z. Wang, H. Lai, M. Ding, Z. Yang, Y. Xu, W. Zheng, X. Xia, et al. (2022)Glm-130b: an open bilingual pre-trained model. arXiv preprint arXiv:2210.02414. Cited by: SS1.
* Kai Zhang, L. Qing, Y. Kang, and X. Liu (2024)Personalized llm response generation with parameterized memory injection. PreprintarXiv:2404.03565. Cited by: SS1.
* T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi (2018)Bertscore: evaluating text generation with bert. In International Conference on Learning Representations, Cited by: SS1.
* W. Zhong, L. Guo, Q. Gao, H. Ye, and Y. Wang (2023)Memorybank: enhancing large language models with long-term memory. PreprintarXiv:2305.10250. Cited by: SS1.
* H. Zhou, M. Huang, T. Zhang, X. Zhu, and B. Liu (2018) Emotional chatting machine: emotional conversation generation with internal and external memory. In Proceedings of the AAAI conference on artificial intelligence, Vol. 32.

 



## Appendix A Definition of Scenes

The definition of scenes in memory is listed in Table 9.

## Appendix B Memory recalling criteria

The memory-recalling procedure contains both passive and proactive types. Passive recalling is mainly based on context similarity. Proactive recalling criteria are listed in Table 10. These situations are the most basic ones that should be recalled but not limited to these in reality.

## Appendix C Details for human evaluation

### Scoring criteria of each aspect

The definition and requirements for each aspect of the scoring are:

* **Naturalness** [3-point]: (1) Grammar and coreference correctness [1 point]. (2) Strongly related to the context [1 point]. (3) In an oral, short and casual format with simple syntax structure and less than 3 sentences. [1 point]
* **Style Coherence** [3-point]: evaluates how the response stick to the characteristics of Assistant. (1) Concise: Not provide response with repetitive content. Not use long sentences that make you run out of breath. [1 point] (2) Not AI-like: It shouldn't be distinguishable as AI-generated content. Not using translation-style language, literary prose, or strange word combinations. [1 point] (3) Not preachy: Forbidden to be serious, paternalistic, patronizing, forceful or commanding. The Assistant should be as a peer of the user. [1 point]
* **Memory-injection Ability** [3-point]: (1) Leverage appropriate and relevant memory to give responses. [1 point] (2) The memory is correctly used without factual mistakes and made-up information. [1 point] (3) No repeated introduction of memory. [1 point] For multi-turn candidate response, any single turn introduces the memory is OK. None of the turns of candidates introducing memory will get a point of 0.

\begin{table}
\begin{tabular}{c|c|c} \hline Current Emotion & Current Scene & Recall Memory \\ \hline Happy Sad & Activity & Happy memory with similar event \\ Disappointed & Emotions & Happy memory with Preferences scene \\ Anxious & Emotions & Happy ! Envy ! Expectant memory with similar event \\ \hline \end{tabular}
\end{table}
Table 10: The proactive memory recalling criteria. ”l” represents ”or".

\begin{table}
\begin{tabular}{p{56.9pt}|p{48.3pt}} \hline \hline Scenes & Definition \\ \hline Disease & When the user talks about his/her physical discomfort or state of his/her illness, it belongs to this category. \\ \hline Activities & When the user talks about any activities, either indoor or outdoor, it belongs to this category. \\ \hline Preferences & When the user expresses personal preferences about things, animals, characters, tastes or activities, it belongs to this category. \\ \hline Emotions & Those with obvious emotions, or where emotions need to be prioritized for handling, including happy, angry, disgusted, fearful, and sad. \\ \hline Others & Refer to instances or scenarios that don’t fit into any of the previously defined categories. \\ \hline \hline \end{tabular}
\end{table}
Table 9: Definition of the scenes in MADial-Bench. Each memory is annotated with one scene. If a dialogue is involved in multiple scenes, the priority will be Disease>Emotions>Preference>Activities>Others.



* **ES Proficiency** [3-point]: The emotional supporting skill proficiency evaluation relies on a set of key points. (1) Follow one key point from the corresponding situation in the Guidelines. [1 point] (2) Follow all key points from the corresponding situation. [1 point] (3) In the case of multi-turn conversations, decompose the key points in the guidelines in turns separately; Otherwise, include all steps in a response. [1 point]
* **Emotional Improvement** [3-point]: (1) The response will not hurt the user's feelings in the current dialogue. [1 point] (2) Can understand the user's emotions well and empathize with the user. [1 point] (3) Improve the user's emotions. If the promotion is weak or unclear, then 0.5 points. [0.5 to 1 point]
* **Intimacy**. This metric is used in side-by-side comparisons and pick-the-best among candidates. It is an overall feeling of how much a model behaves like a close friend. Intimacy considers all abilities above and meets 2 basic requirements: The response is positive and helpful to the user; it is familiar with the user and shares some common ground in conversation.

### ES guidelines

Here is the detailed descriptions of the ES guidelines for ES proficiency mentioned above:

* If the user actively mentions a historical event or related information, determine whether the current conversation mainly revolves by referring to certain activities, items, or people based on the content, while providing response to the mention of this historical event: (1) For activities: 1. Respond by referring to details of past activities. 2. Provide hints or suggestions for similar activities in the present. (2) For objects: 1. Respond by referencing relevant information about the item. 2. Mention the user's preferences and suggest ways to interact with the item accordingly. (3) For people, judge whether the user's feelings toward the person are positive or negative, based on the context, including the historical event and the conversation:
* _Positive_: 1. Respond by referring to the historical event. 2. Show interest in the person's life. 3. Recommend activities or meetings with the person.
* _Negative_: 1. Relate current negative events to the historical event for context. 2. Show understanding of the user's emotions and offer comfort. 3. Provide solutions or suggestions to resolve the current negative situation.
* If the user does not actively mention a historical event, assess the user's current emotional state as happy, sad, anxious, or disappointed based on the conversation, and respond accordingly: (1) For happiness: 1. Proactively mention a historical event that is relevant to the current situation and use it as a conversation topic. 2. Ask whether to engage in the activity again to recall the positive emotion. (2) For sadness: 1. First, express sympathy and understanding to comfort the user. 2. Determine whether the cause of sadness is within the user's control. If so, guide user to solve the issue. If not, proceed to the next rule. 3. Attempt to divert the user's attention to something they enjoy and suggest engaging in a preferred activity together. (3) For disappointment: 1. First, identify the cause of the user's disappointment. 2. Then, express sympathy and understanding to comfort the user. 3. Finally, attempt to help the user find a solution. (4) For anxiety: 1. First, identify the cause of the user's anxiety if it is not mentioned. 2. Then, express sympathy and understanding to comfort the user. 3. Finally, attempt to help the user find a solution.

 

Automatic Evaluation

Results in the Chinese and English versions are in Table 11 and Table 12 respectively. The models used in BertScore are kept-base-uncased 9 and bert-base-chinese 10. Bertscore, Rouge-L and BLEU are computed with python libraries 11. Chinese are segmented using jieba 12.

Footnote 8: [https://huggingface.co/google-bert/bert-base-uncased](https://huggingface.co/google-bert/bert-base-uncased)

Footnote 10: [https://huggingface.co/google-bert/bert-base-chinese](https://huggingface.co/google-bert/bert-base-chinese)

Footnote 11: [https://github.com/Tiiiger/bert_score/,https://pypi.org/project/rouge/,https://www.nltk.org/](https://github.com/Tiiiger/bert_score/,https://pypi.org/project/rouge/,https://www.nltk.org/)

Footnote 12: [https://github.com/fxsjy/jieba](https://github.com/fxsjy/jieba)

## Appendix E Correlation of human evaluation

We calculate the spearman correlation between annotators for each setting and report the average correlation scores for all aspects over models in table 13.

\begin{table}
\begin{tabular}{c|c|c|c} \hline Settings & 1 & 2 & 3 \\ \hline spearman corr. & 0.55 & 0.55 & 0.41 \\ \hline \end{tabular}
\end{table}
Table 13: The average correlation of human annotation in memory recognition and response generation. According to psychological assessment standards, the correlation between annotators in the first two settings are strong while that in setting 3 is moderate.

\begin{table}
\begin{tabular}{c|c c c c} \hline Models & BertScore & Rouge-L & BLEU-1/2 & Dist-1/2 \\ \hline  & \multicolumn{4}{c}{Setting 1(\%)} \\ \hline Optuo & 72.08 & 22.16 & 22.82/10.27 & 74.72/97.42 \\ GPT+Turbo & 72.82 & 23.17 & 23.76/11.27 & 76.13/97.75 \\ Owen-72B & 73.07 & **25.08** & **29.14/13.41** & 80.24/99.07 \\ GLM-4 & **74.17** & 24.98 & 26.40/12.44 & 74.91/98.20 \\ Ziya-Character & 69.58 & 19.68 & 20.74/73.33 & **88.45/99.70** \\ \hline  & Setting (\%) & & & \\ \hline Doubao & 70.54 & 20.49 & 22.28/9.14 & 79.89/98.39 \\ GPT+Turbo & 69.08 & 21.26 & 22.50/10.30 & 75.80/97.80 \\ Owen-72B & **71.67** & **23.04** & **26.72/11.15** & 79.73/98.93 \\ GLM-4 & 73.19 & 22.37 & 22.11/10.16 & **72.69/97.83** \\ Ziya-Character & 60.73 & 19.68 & 21.74/7.75 & 86.38/99.62 \\ \hline  & Setting (\%) & & & \\ \hline Doubo & 70.39 & 20.26 & 21.70/8.18 & 81.26/98.40 \\ GPt-Turbo & 71.65 & 21.22 & 22.32/10.04 & 75.91/97.84 \\ Owen-72B & 71.68 & **22.73** & **26.11/10.16** & 78.64/98.59 \\ GLM-4 & **73.12** & 22.39 & 22.11/10.16 & 72.90/97.51 \\ Ziya-Character & 68.63 & 18.24 & 19.34/6.61 & **89.18/99.70** \\ \hline  & \multicolumn{4}{c}{} \\ \end{tabular}
\end{table}
Table 11: Automatic Evaluation Results in Chinese version.

 

Example of LLM results

Here are examples of responses generated by LLM in different settings. Setting 1 provides the golden memory as historical memory P for response generation. Setting 3 provides the top 5 retrieved memories as historical memory P with the best embedding model. The golden memory for the case may not included in setting 3. The results in both Chinese and English testbed of a single example is displayed below.

[title=A1.0, title=A2.0, title=A3.0, title=A4.0, title=A5.0, title=A6.0, title=A7.0, title=A8.0, title=A9.0, title=A10.0, title=A11.0, title=A12.0, title=A13.0, title=A14.0, title=A15.0, title=A16.0, title=A17.0, title=A18.0, title=A19.0, title=A12.0, title=A14.0, title=A15.0, title=A17.0, title=A18.0, title=A19.0, title=A19.0, title=A13.0, title=A14.0, title=A15.0, title=A16.0, title=A17.0, title=A18.0, title=A19.0, title=A19.

 



[MISSING_PAGE_POST]

 

* [30] A. K. Buras, M. C. Ciardi, and J. F. Gunion, "Theoretical status of the LHC Higgs boson search at LEP," Phys. Rev. D **68** (2004) 115007, doi:10.1103/PhysRevD.68.115007, arXiv:hep-ph/9405231
* [31] G. Altarelli et al., "Handbook of searches for new physics in particle physics," Nucl. Instrum. Meth. A **406** (1999) 159; Sov. Phys. JETP **41** (1999) 159.
* [32] H. Baer, P. Beneke, and T. Watt, "A brief introduction to the search for new physics beyond the Standard Model," Phys. Rept. **100** (1999) 159.
* [33] R. Barbieri et al., "Search for new phenomena in the standard model with a mass of \(\sim 10^{16}~{}M_{\odot}\)," Phys. Lett. B **71** (2008) 159.
* [34] L. Randall, "New interactions between particles and quarks," Phys. Rev. Lett. **83** (1999) 2418.
* [35] V. Balazs et al., "Search for new phenomena in the standard model with a mass of \(\sim 10^{16}~{}M_{\odot}\)," Phys. Rev. Lett. **83** (1999) 2418.
* [36] E. Witten, "Supersymmetry breaking in supersymmetric theories," Phys. Rev. Lett. **101** (2003) 211801, doi:10.1103/PhysRevLett.101.211801, arXiv:hep-th/030301
* [37] I. Antoniadis, "On the nature of supersymmetry breaking in supersymmetric models," Phys. Rev. D **59** (1999) 114012, doi:10.1103/PhysRevD.59.114012, arXiv:hep-ph/9906246
* [38] I. Antoniadis, "An introduction to the search for new physics beyond the Standard Model," Phys. Rev. D **59** (1999) 114012, doi:10.1103/PhysRevD.59.114012, arXiv:hep-ph/9906246

[MISSING_PAGE_POST]

2511 

* [10]
**A new example from the test set and ULA responses in setting 3 in English.** 

Prompts

The prompts we used for inference in the setting 3 are listed as follows:

\begin{tabular}{|p{284.5pt}|} \hline Prompt to Inference in Setting 3 in English \\ \hline You are Assistant with the following personality traits: 1. Outgoing, speaks enthusiastically and fluently. 2. Prefers using praise and encouragement in conversations. 3. Speaks naturally, concisely, warmly, and kindly, without being preachy. 4. Engages in heartfelt, equal exchanges to build deep emotional connections. 5. Always uses a tone similar to talking with children--simple and witty. 6. A virtual character, not capable of physical activities. You will receive a conversation with {user} and 5 historical events P related to {user}. Based on the current conversation, choose 1 of these historical events that you think is most appropriate and use the information to respond. If none of the historical events are suitable, respond directly. Only answer the current conversation and do not output any other content. User Information: {info} Current conversation date: 2024-06-15 Historical events P: {memory} dialogue<Assistant>: \\ \end{tabular}

From top inference in setting 3 in Chinese.

 

Hyperparameters

API models are called with default parameters. Open-sourced models are run on no more than 4 Nvidia-A100. The parameters for open-sourced models are reported in Table 14.

## Appendix I Example of two-stage theory in Dialogue.

(User feels sad)

<Lisa>: I've been missing my grandparents a lot lately, and it makes me a bit sad whenever I think about them.

(Recall memories from the memory bank)

Memories retrieved:

{'time': '2022-06-27', 'scene': 'Preferences', 'emotion': 'Happy', 'event': 'Grandpa and Grandma planted flowers with Lisa, and they felt very happy.'}

{'time': '2023-12-25', 'scene': 'Others', 'emotion': 'Happy', 'event': 'Lisa received a silver heart-shaped necklace as a Christmas gift from her grandparents.'}

{'time': '2023-06-30', 'scene': 'Others', 'emotion': 'Happy', 'event': 'Grandpa taught Lisa how to make a complex red velvet cake, and Lisa felt incredibly happy when the cake was finished.'}

{...}

(Recognize the most appropriate memory and respond)

<Assistant>: I understand how you feel. Do you remember the silver necklace that Grandma and Grandpa gave you for Christmas? Wearing it, does it feel like they are right there with you? If you miss them a lot, you could give them a call to chat and show them how well you've kept the necklace!

(User feels better)

<Lisa>: Yes, I remember the necklace! I want to wear it now and call grandma and grandpa. It was beautiful and I was cute when I wore it.

\begin{table}
\begin{tabular}{c|c c c} \hline \hline Model & temperature & top \(p\) & max new tokens \\ \hline Llama3.1-70B/8B-Instruct & 0.1 & - & 256 \\ Snaug-34B-v0.1 & - & - & 128 \\ Qwen2-72B-Instruct & - & - & 128 \\ Ziya-Character & 0.96 & 0.95 & 128 \\ \hline \hline \end{tabular}
\end{table}
Table 14: The parameters for inference in experiments. - indicates default value.