Both Matter: Enhancing the Emotional Intelligence of Large Language Models without Compromising the General Intelligence (2024)

Abstract: Emotional Intelligence (EI), consisting of emotion perception, emotion cognition and emotion expression, plays the critical roles in improving user interaction experience for the current large language model (LLM) based conversational general AI assistants. Previous works mainly focus on raising the emotion perception ability of them via naive fine-tuning on EIrelated classification or regression tasks. However, this leads to the incomplete enhancement of EI and catastrophic forgetting of the general intelligence (GI). To this end, we first introduce EIBENCH, a large-scale collection of EI-related tasks in the text-to-text format with task instructions that covers all three aspects of EI, which lays a solid foundation for the comprehensive EI enhancement of LLMs. Then a novel Modular Emotional Intelligence enhancement method (MoEI), consisting of Modular Parameter Expansion and intra-inter modulation, is proposed to comprehensively enhance the EI of LLMs without compromise their GI. Extensive experiments on two representative LLM-based assistants, Flan-T5 and LLaMA-2Chat, demonstrate the effectiveness of MoEI to improving EI while maintain GI. 1

heading: 1 Introduction
text: The question is not whether intelligent machines can have any emotions, but whether machines can be intelligent without any emotions.
– Marvin Minsky
Emotional intelligence (EI), a pivotal concept in the field of human intelligence, holds significant importance in the context of the current large language models (LLMs) (Brown et al., 2020; Raffel et al., 2020; Touvron et al., 2023) exhibiting great
* Equal contribution † Corresponding author 1Our source code is available at https://github.com/
circle-hit/MoEI.
Emotion Perception Emotion Cognition Emotion Expression General Reasoning World Knowledge
10
20
30
40
50
60
70
80
Pe rfo
rm an
ce
Emotional Intelligence
General Intelligence
Flan-T5 (XL) Flan-T5-Emo (XL)
Figure 1: Comparison of EI and GI before (in orange) and after (in blue) the EI-enhancement via naive finetuning based on the Flan-T5-XL (3B) backbone.
general intelligence (GI) to serve as conversational general AI assistants (Minsky, 2007). It involves effectively deal with EI-related downstream tasks to accurately perceive, understand users’ emotional states and respond properly, which is necessary for fostering effective communication and facilitating smooth social interactions (Mayer et al., 2001).
To enhance the EI, many researchers have started performing the naive fine-tuning of LLMs on EIrelated tasks (Zhang et al., 2023a,b; Lei et al., 2023; Liu et al., 2024). However, there are two significant limitations in the current state of these efforts.
On one hand, the EI-enhancement in these studies is of narrow scope because they only focus on emotion perception regarding emotion classification tasks. However, EI is a broad concept that also includes emotion cognition (e.g. emotion cause reasoning (Poria et al., 2021)) and expression (e.g. empathethic response generation (Rashkin et al., 2018)). Therefore, the comprehensive enhancement of EI for LLMs is ignored in previous works.
On the other hand, they all overlook the catastrophic forgetting (McCloskey and Cohen, 1989) of the GI of LLM backbones during the process of
11157
enhancing EI via naive fine-tuning. In our preliminary experiments depicted in Figure 1, we find that solely focusing on improving the EI of an LLM backbone would result in a significant decline in its GI, such as world knowledge and general reasoning. Desirable intelligent assistants are supposed to be of both high GI and high EI, where any performance loss on either front would significantly undermine the user experience.
To this end, we study how to comprehensively enhance the EI of current LLM-based assistants while preserving their inherent GI from being compromised. According to three aspects of EI defined by Mayer et al. (2001), named emotion perception, emotion cognition and emotion expression, we first construct EIBENCH, a large curated collection of EI-related tasks converted into a text-to-text format, covering 15 tasks with 88 datasets. Moreover, motivated by the promising gains from fine-tuning with task instructions (Sanh et al., 2021; Wang et al., 2022; Longpre et al., 2023), we also manually write related instructions for each dataset. Thus, EIBENCH lays a solid foundation for the comprehensive EI enhancement of LLMs.
Further, we propose a novel Modular Emotional Intelligence enhancing method (MoEI) with two collaborative techniques, Modular Parameter Expansion (MPE) and Intra-Inter Modulation (I2M), to comprehensively improve the EI of LLMs while maintaining most of their GI. Specifically, in MPE, a set of modular parameters are introduced to endow additional capacity to handle various tasks within the above three aspects of EI. For the sake of computation and resource efficiency, these expanded modular parameters are instantiated with parameter-efficient LoRA blocks (Hu et al., 2021), named MoLoRA. During the process of EI-enhancement, only the EI-specific MoLoRA is updated, thereby reducing the impact on parameters of the LLM backbone representing its GI. In addition, a router is devised in I2M to exert the modulation on the two separate parameters. More specifically, intra-modulation is performed within MoLoRA, leveraging the weighted combination of different LoRA blocks to deal with various EIrelated tasks. And inter-modulation functions to strike the balance between MoLoRA and the whole LLM backbone to achieve the goal of protecting GI, where the GI-related samples are navigated to be processed by the LLM backbone while reducing the influence from EI-specific MoLoRA.
We conduct extensive experiments on two repre-
sentative open-source LLM-based assistants, FlanT5 (Chung et al., 2022) and LLaMA-2-Chat (Touvron et al., 2023). Results demonstrate that MoEI not only helps significantly enhance all three aspects of their EI but also ensures that their GI, including world knowledge, general reasoning, commonsense reasoning and reading comprehension, are hardly compromised. Moreover, such EI-enhanced models could exhibit better performance when addressing various OOD tasks in the presence of emotional stimuli (Li et al., 2023b).
The main contributions of this work are summarized as follows: (1) We take the first step to study how to develop an LLM-based assistant that possesses both high GI and EI, a challenging direction for the more practical application of LLMs. (2) We introduce EIBENCH, a comprehensive collection of EI-related tasks to support the EI enhancement of LLM backbones. Then a novel method MoEI is proposed to comprehensively improve the EI of LLM backbones without compromising their GI. (3) Experiments on various EI and GI benchmarks demonstrate the effectiveness of MoEI.

heading: 2 EIBENCH
text: We first introduce EIBENCH, a large-scale collection of EI-related tasks with instructions that describe them in plain language. Figure 2 shows the overview of the benchmark by category and task.
Taxonomy of Emotional Intelligence. According to Mayer et al. (2001), the EI of LLMs includes three aspects: (1) Emotion Perception, (2) Emotion Cognition and (3) Emotion Expression. Please
refer to Appendix A for their specific meanings.
Task Collection. Based on the aforementioned EI taxonomy, we systematically gather and organize existing open-source tasks and datasets related to EI within this framework. The resulting depiction of all EI-related tasks in EIBENCH is illustrated in Figure 2, while Table 6 (in Appendix B) provides a comprehensive list of datasets associated with each task. This leads to 15 tasks and 88 datasets in total. To be more specific, in the aspect of emotion perception, the primary focus is on classification tasks such as emotion recognition and stance detection. Shifting to the area of emotion cognition, the prioritized task involves extracting emotion causes. Additionally, we incorporate more advanced cognitive challenges, including irony and metaphor recognition, aiming to comprehensively enhance the LLMs’ emotional cognitive capabilities. Finally, in the realm of emotion expression, the main tasks involve empathetic response and emotional support, empowering the model to offer improved comfort and guidance to users, thereby enhancing the overall interactive experience.
Task Schema. Motivated by the promising gains from fine-tuning with task instructions (Sanh et al., 2021; Wang et al., 2022; Longpre et al., 2023), we manually construct one piece of instruction for each dataset in our EIBENCH. Specifically, the schema contains the following components: (1) Text Input: the input sentence X . (2) Instruction: the detailed guidance on how the model should process X to complete the current task. (3) Option (for classification task only): including all the candidate labels and serving as both a constraint and a hint. We hire 5 annotators who are proficient in English to write these instructions. In addition, to guarantee the quality of these instructions, 1 or 2 reviewers are also assigned to each dataset. Their task is to confirm that whether the instructions are clear, fluent and comprehensive enough for an average language speaker to successfully complete the given task. Examples of instances from our EIBENCH is displayed in Table 7 in Appendix B.

heading: 3 Methodology
text: Instead of merely focusing on enhancing specific capabilities of LLMs, we present a novel approach MoEI, a model-agnostic EI-enhancing method that is compatible with any transformer-based LLM, which could not only comprehensively boost the
EI of LLMs but also safeguards their GI from being compromised. As shown in Figure 3, MoEI consists of two collaborative techniques, namely Modular Parameter Expansion and Intra-Inter Modulation. The subsequent section will offer a detailed introduction to both of them.

heading: 3.1 Preliminary
text: Low Rank Adaptation. We adopt a representative PET method LoRA (Hu et al., 2021) in MoEI. Specifically, in LoRA, the pre-trained weight matrix of LLMs is expanded with a low-rank decomposition. For any linear layer h = W0x, the forward pass with LoRA is modified to be:
h = W0x+BAx (1)
where W0 ∈ Rd×k, B ∈ Rd×r, A ∈ Rr×k and the rank r ≪ min(d, k). The pre-trained weight W0 remains fixed during training, while A and B are trainable parameters.

heading: 3.2 Modular Emotional Intelligence Enhancement
text: Modular Parameter Expansion. To harness two advantages of LoRA, which includes isolation from the entire parameters of LLMs and promising training efficiency, we take a step further by extending it to modular designs, endowing more capacity to accomodate various newly acquired EI-related knowledge. This leads to the Mixture-of-LoRA (MoLoRA) architecture.
MoLoRA inherits the flexibility of LoRA and could be applied on any linear layer of the transformer-based LLM backbone (mainly in the
attention and FFN layer) to expand the model capacity with multiple pairs of low-rank matrixes. To be more specific, the singular pair of low-rank matrixes BA in Equation (1) is now replaced by a set containing multiple ones {BiAi}Ni=1 and the forward process of the MoLoRA layer can be mathematically expressed as follows:
h = W0x+ N∑
i=1
BiAix (2)
where N is the number of modular LoRA blocks.
Intra-Inter Modulation. To effectively navigate different types of inputs to be properly processed by its corresponding parameters, a router is introduced in I2M to exert the intra- and inter-modulation on weighted contributions of the separate MoLoRA and the LLM backbone:
h = αW0x+
N∑
i=1
βi BiAix
[α; {βi}Ni=1] = G(x) = softmax(Wx) (3)
where G(·) represents the router and W ∈ Rd×N+1 is the trainable parameter of it. α and {βi}Ni=1 are the intensity weights of inter and intra-modulation, respectively. In the upcoming contents, we will elaborate the concept and delve into the optimization process associated with them.
More specifically, intra-modulation {βi}Ni=1 is only performed within EI-specific MoLoRA to leverage the weighted combination of different LoRA blocks, which leads to more powerful capability in dealing with various EI-related tasks compared to the single LoRA. As shown in Figure 3, the optimization of intra-modulation is driven by the EI-related inputs with the task loss:
Ltask = − ∑
(x,y)∈EI logP (y | x; θm, θE , θG) (4)
where θm, θE and θG are parameters of the LLM backbone, the MoLoRA and the router, respectively. EI is the training data samples from EIBENCH. Only parameters of θE and θG are updated during the training.
And inter-modulation α functions to strike the balance between MoLoRA and the whole LLM backbone, ensuring the GI-related samples to be processed only by the LLM backbone and eliminating the influence from EI-specific MoLoRA.
This is achieved through the minimizing of a KL divergence loss:
LKL = ∑
(x,y)∈GI DKL(G(x)||I) (5)
where GI is the replayed GI-related samples from the previous training corpus of the LLM backbone, which is a subset of Flan collection (Longpre et al., 2023) in our experiments. I is the one-hot vector with only the position of α setting to 1.
And it is worth to mention that, in our experiments, although α does involve in the softmax function, it is exactly set to 1 in the forward pass because we believe the large-scale knowledge stored in the contemporary LLMs are the key to handle downstream tasks, which is also verified by our main experiments in Table 1. In general, the intermodulation ensures the given input, either EI- or GI-related, to fully leverage the powerful LLM backbone, while intra-modulation determines to what extent the incremental EI-enhanced MoLoRA are activated to complete the current input.
Finally, a multi-task learning fashion is adopted to jointly minimize the task loss and the KL loss:
L = Ltask + λLKL (6)
where λ functions to balance the two parts.

heading: 4 Experiments

heading: 4.1 Dataset and Evaluation Metrics
text: Emotional Intelligence. We split EIBENCH into two subsets: one for training and the other for evaluation. For the training set, we sample a maximum of 5,000 instances from each dataset, resulting in 268,234 training instances in EI. For an efficient evaluation, a maximum of 100 instances are sampled from the remaining sets of each dataset, leading to 5,600 instances as the evaluation under the supervised settings. We report the average accuracy for datasets from Emotion Perception (Emo.Prc) and Rouge-L (Lin, 2004) for those from Emotion Cognition (Emo.Cog) and expression (Emo.Exp). Furthermore, to comprehensively assess the impact of EI-enhancement, we extend our evaluation to include EQ-Bench (Paech, 2023) for cross-task zero-shot evaluation, where models are tasked with predicting the intensity of emotions of characters in a dialogue with a set of 60 English questions.
General Intelligence. To evaluate a model’s GI, following prior research (Wang et al., 2023a), we conduct evaluations across four crucial dimensions: (1) World Knowledge (WK): Employing the Massive Multitask Language Understanding dataset (MMLU) (Hendrycks et al., 2020), with questions spanning 57 subjects, ranging from elementary to professional levels. Following LLaMa-2 (Touvron et al., 2023), we report 5-shot accuracy. (2) General Reasoning (GR): The evaluation involves BigBench-Hard (BBH) (Suzgun et al., 2022), featuring 23 tasks derived from Big-Bench (Ghazal et al., 2013). Few-shot prompting, provided with 3-shot in-context examples, are utilized and EM scores are reported. (3) Commonsense Reasoning (CR): Our assessment incorporates PIQA (Bisk et al., 2020). Following LLaMa-2 (Touvron et al., 2023), we report 0-shot accuracy. (4) Reading Comprehension (RC): BoolQ (Clark et al., 2019) is adopted for evaluating reading comprehension, with the focus on reporting 0-shot accuracy.

heading: 4.2 Baselines and Comparison Models
text: Based on two representative LLM-based assistants of different architectures, Flan-T5 (encoderdecoder) (Chung et al., 2022) and LLaMA-2-Chat (decoder-only) (Touvron et al., 2023), we evaluate MoEI against the following EI-enhancing methods: (1) FT: directly fine-tunes the LLM backbone with the whole parameters of it updated. (2) LoRA (Hu et al., 2021): updates additional parameter-efficient low rank matrixes and weights of the LLM back-
bone remain frozen, which could be viewed as the method of singular parameter expansion without intra-inter modulation. Moreover, both baseline methods are seamlessly integrated with Replay (Lopez-Paz and Ranzato, 2017), a widely adopted continual learning method to mitigate the catastrophic forgetting of previous knowledge, where the model is trained with a mixture of EI and GI data in the multi-task learning fashion.

heading: 4.3 Implementation Details
text: In MoEI, the low rank r of MoLoRA is 4 and N is 8. And the r of the single LoRA-tuning is 32 for the fair comparison. MoEI and LoRA are both implemented upon the query and value projection in the attention layer, and the last linear transformation in the FFN layer. All methods are trained for 5 epochs with 3 random runs. For more detailed settings, please refer to the Appendix C.

heading: 5 Results and Analysis

heading: 5.1 Overall Results
text: Table 1 and Figure 4 demonstrate the performance comparison of MoEI and baselines in terms of EI and GI. Our findings are as follows:
MoEI could effectively improve EI and maintain GI simultaneously across different architures and sizes of LLM backbones. Compared to naive EI-enhancing techniques such as FT and LoRA on both Flan-T5 and LLaMA-2-Chat backbones, MoEI demonstrates superior performance in enhancing all three aspects of EI, while effectively safeguarding the GI from compromise across all four dimensions. This trend is still consistent when we apply MoEI on larger Flan-T5-XXL (11B, shown in Figure 4) and LLaMA-2-Chat-13B (shown in Figure 7 in Appendix D). Furthermore, the parameter-efficient LoRA and MoEI even outperform FT in enhancing EI, highlighting the significant potential of these lightweight methods to be more effective in aiding LLMs in adapting to specific domains (Ding et al., 2022).
Simply replaying the previous data is not sufficient to maintain GI. Although the Replay method is widely adopted in the domain adaption of LLMs, directly applying it for EI-enhancement could not reach the expected outcomes. This can be ascribed to the negative task transfer (Zhang et al., 2022; Jang et al., 2023). Benefiting from the explicit separation of different parameters representing EI and GI via modular parameter expansion
and our intra-inter modulation, MoEI attempts to navigate the EI- and GI-related samples to be processed by the proper parameters, offering a novel perspective to leverage replay-based methods in the enhancement of LLMs. The training instances in replayed set GI is 5,000. We also perform additional experiments with the varied size of replayed data to display the efficiency and resource-friendly of MoEI. Please refer to Appendix F for details.

heading: 5.2 Ablation Study
text: We conduct ablation studies to verify the effectiveness of different components proposed in MoEI. Results are shown in Table 2.
Effect of Modular Parameter Expansion. After replacing the MoLoRA with a singular LoRA block, although the GI could still be largely protected via intra-inter modulation, the effect of EIenhancement is largely hindered, which manifests the importance of additional capacity to accomodate various aspects of EI.
Effect of Intra-Modulation. Without the intramodulation, each pairs of modular parameters in the expanded MoLoRA is equally activated, resulting the decline of performance in terms of EI. This manifests the critical role of intra-modulation to navigate various EI-related inputs to be properly and effectively processed.
Effect of Inter-Modulation. When we remove the inter-modulation which serves to balance the utilization of the EI-specific MoLoRA and the
whole LLM backbone representing the GI, the significant degradation of all four dimensions of GI demonstrates its crucial role in preserving the GI from being compromised. Although the GI could be partly recovered through the incorpration of Replay, it is still limited in offering clear navigation for the given inputs to lead them be properly processed. In addition, the more superior results on aspects of Emo.Prc and Emo.Cog remind us that there is a trade-off between EI-enhancing and GI-maintaining, encouraging more advanced intrainter modulation mechanisms to achieve better optimization for future works in this direction.
In general, Modular Parameter Expansion and Intra-Modulation play important roles in achieving more effective EI enhancement, while InterModulation focuses on protecting GI. The three of them achieve a relative balance in MoEI.

heading: 5.3 Results of EI on Cross-Task Settings
text: To comprehensively assess the impact of EIenhancement, we include EQ-Bench for cross-task zero-shot evaluation. Due to the requirement for models participating in the EQ-Bench evaluation to possess a certain level of instruction-following capability, we conduct experiments using the Flan-T5XXL (11B) and the LLaMA-2-Chat-13B. Results are shown in Figure 5 (Appendix G). Compared to the original models, MoEI still demonstrates the effective improvements in EI. And we do not include the FT- and LoRA-version baselines because the resulting models can not follow input prompts to complete the evaluation with valid outputs, which further verifies the effectiveness of MoEI to largely protect the GI of LLM backbones.

heading: 5.4 Impact of EI on OOD Tasks
text: Li et al. (2023b) propose EmotionPrompt (EP) to explore EI to enhance the performance of LLMs on other OOD downstream tasks, which is performed with the incorporation of emotional stimulus into regular prompts. An example of EP is in Figure 6 in
Appendix E. Here, we explore how EP would perform based on the LLMs with enhanced-EI. To be more specific, following the experimental settings in Li et al. (2023b), we evaluate EP on eight tasks of Instruction Induction (Honovich et al., 2022): Sentiment Analysis (SA), Sentence Similarity (SS), Cause Selection (CS), Sum, Word in Context (WC), Starting With (SW), Larger Animal (LA) and First Letter (FL). Details on those tasks can be found in Table 8 and designs of all 11 types of EP are in Table 9. For each task, 100 samples are randomly selected, except for Cause Selection, including 50 examples in total. And the prompting strategy includes both zero-shot and few-shot ways with incontext demonstrations chosen from the remaining part of the data. Each column in Table 3 is the average performance of all 11 types of EP. Interestingly, EP exhibits more powerful performance on the LLMs with EI-enhanced by our MoEI, especially on the few-shot learning ability. This further demonstrates the importance of both GI and EI in assisting LLMs to accomplish specific tasks.

heading: 5.5 Qualitative Analysis on MoEI
text: Here we qualitatively demonstrate the activation status (i.e. α and β values) of the Intra-Inter Modulation of MoEI when facing different EI- and GIrelated inputs. Specifically, these values are derived from the router G(·) corresponding to the FFN block from the last Transformer layer, averaged over all samples in each test dataset. Here are 8 values because our experimental setup utilizes 8 modular LoRA blocks. Results on Flan-T5-3B and LLaMA-2-Chat-7B are shown in Table 4 and Table 5, respectively. There are two interesting phenomena worthy of discussion: (1) When facing different types of EI-related tasks, intra-modulation does not exhibit distinct task-specific patterns (Only tasks from Emo.Exp require more modular LoRA to participate in). This is consistent with the conclusion drawn by Jiang et al. (2024), that modular training does not necessarily result in task-specific func-
tional partitions. (2) Significant improvement in model’s EI capability can be achieved with relatively small activation values. This also corroborates an intriguing recent finding by Yu et al. (2023), indicating that making very small delta increments in parameters of LLMs can lead to significant enhancement in the model’s ability to handle specific tasks. We hope that our findings can provide more inspiration for future work on EI enhancement.

heading: 6 Related Works

heading: 6.1 Emotional Intelligence of LLMs
text: The current study on EI of LLMs is primarily centered around two key directions. Firstly, researchers are delving into the integration of psychological theories or scales, proposing a public evaluation benchmark to evaluate the emotional understanding capabilities of LLMs (Wang et al., 2023b; Paech, 2023; Huang et al., 2023). Secondly, efforts are directed towards fine-tuning LLMs for specific EI-related downstream tasks, with a predominant focus on classification and regression tasks, aiming to improve their proficiency in handling such challenges (Zhang et al., 2023a,b; Lei et al., 2023; Liu et al., 2024; Li et al., 2024).
In contrast to existing works, our study stands out in the following aspects: (1) Rather than exclusively evaluating the emotional understanding capabilities of LLMs, based on our proposed EIBENCH, we seek to comprehensively enhance all three facets of their EI. (2) We recognize the GI and EI as equally vital capabilities of LLMs, and our design of MoEI aims to boost EI while simultaneously maximizing the preservation of their GI.

heading: 6.2 Parameter-Efficient Tuning
text: Recently, there has been a growing interest in parameter-efficient tuning (PET) (Ding et al., 2022). This research area aims to minimize computational resources when adapting LLMs to specific tasks through the introduction of additional parameters
that are much fewer compared to the LLM backbones (Houlsby et al., 2019; Lester et al., 2021; Li and Liang, 2021; Zaken et al., 2022). Among existing PET methods, LoRA (Hu et al., 2021) has stood out for its superior performance. Hence, our modular parameter expansion in MoEI is primarily instantiated with it as a representative method.

heading: 6.3 Mixture-of-Experts for LLMs
text: Another line of related works involves the integration of the Mixture-of-Experts (MoE) architecture (Jacobs et al., 1991) with LLMs through the expansion of the FFN layer, which have exhibited appealing performance in pretraining (Lepikhin et al., 2020; Fedus et al., 2022; Jiang et al., 2024; Dai et al., 2024), continual pretraining (Chen et al., 2023; Wu et al., 2024a) and instruction tuning (Shen et al., 2023) of LLMs. In addition, another line of attempts focus on achieving extended capacity in a more computationally efficient manner using PET blocks (Zadouri et al., 2023; Dun et al., 2023; Liu et al., 2023). The major distinction between our proposed MoEI and these PET-based MoE structures lies in their emphasis solely on enhancing the model’s ability to learn new tasks through modular designs. In contrast, we take a step further by exploring how to empower the model not only to improve its learning of new abilities but also to prevent compromising its previously acquired ones. Therefore, the problems addressed in this work are more challenging and demanding. Concurrently, Dou et al. (2023) propose to maintain world knowledge during the alignment of LLMs, while we focus on protecting more aspects of the GI to take the capabilities of reasoning and reading comprehension into account. At the same time, the greater heterogeneity between EI and GI also poses a more challenging setting for this work.

heading: 6.4 Continual Learning for LLMs
text: On one hand, the notion of parameter expansion in our MoEI is partly inherited from the parameter-
isolation-based continual learning (CL) methods, which dynamically expand model capacity or isolate existing model weights to mitigate interference between new and old tasks (Rusu et al., 2016; Fernando et al., 2017). On the other hand, the incorporation of GI-related samples in the process of intrainter modulation aligns with the Rehearsal-based CL methods, where a fixed memory is utilized to store real samples of previous tasks (Lopez-Paz and Ranzato, 2017; Isele and Cosgun, 2018; Rolnick et al., 2019; de Masson D’Autume et al., 2019). Thus, this study lies in an emerging research direction to integrate CL techniques into the adaptation of LLMs (Wu et al., 2022; Song et al., 2023; Zhao et al., 2024a,b; Wu et al., 2024b).

heading: 7 Conclusion and Future Work
text: In this paper, we take the first step to study the challenging and demanding research topic of enhancing the EI of current LLM-empowered assistants while maintaining their GI. We introduce EIBENCH, a comprehensive collection comprising large-scale tasks that encompass all facets of EI: emotion perception, cognition, and expression. Our innovative approach, MoEI, ingeniously integrates two collaborative techniques, Modular Parameter Expansion and Intra-Inter modulation, to introduce additional modular parameter-efficient LoRA blocks for the accomodation of newly acquired EI-related competencies, and automatically navigate EI- and GI-related inputs to be properly processed by the corresponding parameters. Extensive experimental results demonstrate the applicability of MoEI on LLM backbones of varying scales and architectures, highlighting its versatility.
For future work, exploring the quality and quantity of EI- and GI-related data in the process of EI enhancement is an intriguing direction to further enhance the effectiveness and efficiency of EI enhancement and GI maintenance.

heading: 8 Limitation
text: There are several limitations to consider for future directions of EI-enhancement of large language models. Firstly, exploring the quality and quantity of data related to EI and GI during the enhancement process is an intriguing avenue to amplify the effectiveness and efficiency of EI improvement and GI maintenance. Secondly, in MoEI, we assume that previous training data (such as pre-training data and SFT data) of the LLM backbones is ac-
cessible for Inter-Modulation. However, this assumption does not apply to current closed-source black-box commercial LLMs. Therefore, exploring how to achieve EI enhancement in LLMs under restricted data privacy could be considered as a future research direction. Finally, MoEI necessitates the identification of EI- or GI-related tasks during training to establish distinct modulation strategies for each task. Investigating training techniques that is independent of task identification could prove to be a promising avenue for future research, which could favor the application of continually enhancing the EI upon on the online streams of data. We also acknowledge that there are larger LLM-based assistants that we are not able to train due to the limitations of our computational budget.

heading: 9 Ethics Statement
text: In our pursuit to enhance emotional intelligence (EI) in large language models (LLMs), it is imperative to underscore that our primary objective is to bolster their capacity to address and tackle downstream tasks related to EI. This endeavor aims to elevate user experiences by facilitating more nuanced interactions. It is essential to emphasize that our intention is not to anthropomorphize LLMs or imbue them with emotions akin to humans. We are committed to augmenting the ability of LLMs to comprehend and respond to emotional cues within the context of specific NLP tasks or applications. This approach ensures that EI enhancements serve pragmatic purposes, such as improving conversational agents’ ability to recognize and appropriately respond to users’ emotional states. And we maintain a clear distinction between the capabilities of LLMs and the complexities of human emotions. Our research focuses on equipping LLMs with advanced techniques to analyze and respond to emotional cues without ascribing human-like emotions or consciousness to them.
In addition, all of the tasks in our EIBENCH and experiments are based on widely-used open-source datasets. We uphold the principle of informed consent in our research involving human annotation of task instructions. We ensure that annotators are fully informed about the nature and purpose of our research, including any potential risks or benefits, and that they have the opportunity to provide voluntary and informed consent before participating. In addition, all the annotators participate in our research with reasonable wages paid.

heading: Acknowledgements
text: We thank the anonymous reviewers for their insightful comments and suggestions. This work was supported by the National Key RD Program of China via grant 2021YFF0901602.

heading: A Taxonomy of Emotional Intelligence
text: According to Mayer et al. (2001) and recent investigation for emotional dialog ablity of LLMs (Zhao et al., 2023), the EI of LLMs can be defined in three aspects: emotion perception, emotion cognition, and emotion expression. Their specific meanings are as follows:
• Emotion Perception: the ability to detect and decipher emotion of users.
• Emotion Cognition: the ability to comprehend emotion situation and to reason complicated relationships among emotions.
• Emotion Expression: the ability to convey the emotional response properly.
Thus, emotion perception is the fundamental component of EI, while emotion cognition involves a more comprehensive and profound understanding of emotions. Emotion expression, built upon these two aspects, involves conveying emotional information and facilitating user interaction.

heading: B Tasks in EIBENCH
text: In Table 6 we present the list of tasks with datasets used in each task. Table 7 displays examples of instances from our EIBENCH.
C Implementation Details
Our experiments are implemented with PyTorch (Paszke et al., 2019) and Transformer library (Wolf et al., 2020). All models are trained with the AdamW optimizer. The Flan-T5-XL and LLaMA2-Chat-7B is trained on 4 NVIDIA Tesla A800 GPU while the larger ones Flan-T5-XXL and LLaMA-2-Chat-13B are performed on 8 NVIDIA Tesla A800 using DeepSpeed repository (Rasley et al., 2020) with ZeRo-2 optimization. And the evaluation for GI of enhanced LLMs is performed with lm-evaluation-harness (Gao et al., 2023).
The hyper-parameter of LoRA is set with lowrank r to 32, alpha to 32 and dropout to 0.1. For the fair comparison in the LoRA part, the modular expanded parameters in our MoEI is set to 8 pairs of LoRA in with low-rank r to 4. All models is trained for 5 epoches with 3 random runs. And the learning rate for MoEI and LoRA is 3e-4 with the batch size of 32, while that for FT is 5e-5 with the bach size of 256. As for the hyper-parameter λ in Equation (6), it functions to balance the process of
intra-modulation for the EI-related tasks and intermodulation of GI-related ones. The larger λ means that the inter-modulation contributes more to protect GI. However, excessive λ can impair the performance of EI, thereby weakening EI-enhancement. For the Flan-T5 series, the balancing factor λ is 0.1 and instances from GI is replayed every 200 training steps, while λ is 2 for LLaMA-2-Chat family.

heading: D MoEI on Larger LLM Backbones
text: MoEI still exhibits consistent superiority in enhancing EI while maintaining GI when we apply it on larger LLaMA-2-Chat-13B (shown in Figure 7).

heading: E More Details of EmotionPrompt
text: Taking inspiration from psychology, Li et al. (2023b) propose EmotionPrompt (EP) to incorporate psychological insights to improve the effectiveness of LLMs. As illustrated in Figure 6, the implementation of EmotionPrompt is remarkably straightforward, requiring only the addition of emotional stimuli to the initial prompts. To be more specific, as shown in Table 9, such emotional stimuli is designed base on three types of well-established psychology theories, named Social Identity theory, Social Cognition theory and Cognitive Emotion Regulation theory, leading to 11 types in total.

heading: F Experiments with Varied Size of Replayed data
text: The replayed data to assist the maintainence of GI is from the Flan-collection (Longpre et al., 2023), containing a large-scale high-quality samples of various tasks with the corresponding instructions. And the Flan-T5 (Chung et al., 2022) is exactly trained on it. As reported in Touvron et al. (2023), the fine-tuning data of LLaMA-2-Chat includes publicly available instruction datasets, as well as over one million new human-annotated examples. However, this part of fine-tuning data is not publicly available, so we also adopt Flan-collection as an alternative. Due to the high volume of samples in Flan, following (Wang et al., 2023c), we only use a subset of it, which contains 100k samples in total. For our main experiments, considering the training efficiency, we only randomly sample 5k instances from it. Here, we scale the replayed instances to 10k, 50k and 100k for baseline methods LoRA and MoLoRA, while those used in our MoEI is always kept 5000, trying to exploring the efficiency and resource-friendly of MoEI. Results
0 10 20 30 40
LLaMA-2-Chat
LLaMA-2-Chat MoEI
Flan-T5
Flan-T5 MoEI
30.94
36.91
37.99
41.76
Figure 5: Results of the EI-enhancement on the EQBench. The LLM backbones are Flan-T5-XXL (11B) and LLaMA-2-Chat-13B.
for the Flan-T5-XL (3B) and LLaMA-2-Chat-7B are displayed in Figure 8 and Figure 9, respectively. We have the following two observations:
(1) MoEI demonstrates excellent computational efficiency, achieving better performance on GI preservation even with just 5k replay data than baseline methods using 100K data. This further showcases the tremendous potential of MoEI for EI enhancement and GI preservation of LLM backbones in low-resource scenarios. (2) For baseline methods, increasing the amount of replay data may improve the preservation of GI, but it still encounters bottlenecks. Moreover, the effectiveness of EI enhancement may be compromised (illustrated in Figure 9). In contrast, MoEI achieves a better balance, yielding optimal results in both enhancing EI and maintaining GI simultaneously.

heading: G Results of EI on Cross-Task Settings
text: To comprehensively assess the impact of EIenhancement, we include EQ-Bench (Paech, 2023) for cross-task zero-shot evaluation. Results are shown in Figure 5.

heading: Future Work
text: There are several limitations to consider for future directions of EI-enhancement of large language models. Firstly, exploring the quality and quantity of data related to EI and GI during the enhancement process is an intriguing avenue to amplify the effectiveness and efficiency of EI improvement and GI maintenance. Secondly, in MoEI, we assume that previous training data (such as pre-training data and SFT data) of the LLM backbones is ac-
cessible for Inter-Modulation. However, this assumption does not apply to current closed-source black-box commercial LLMs. Therefore, exploring how to achieve EI enhancement in LLMs under restricted data privacy could be considered as a future research direction. Finally, MoEI necessitates the identification of EI- or GI-related tasks during training to establish distinct modulation strategies for each task. Investigating training techniques that is independent of task identification could prove to be a promising avenue for future research, which could favor the application of continually enhancing the EI upon on the online streams of data. We also acknowledge that there are larger LLM-based assistants that we are not able to train due to the limitations of our computational budget