EmoGrowth: Incremental Multi-label Emotion Decoding with Augmented Emotional Relation Graph
Authors: Kaicheng Fu
Changde Du
Jie Peng
Kunpeng Wang
Shuangchen Zhao
Xiaoyu Chen
Huiguang He
Venue: ICML 2025 poster (2025)
Keywords: Emotion decoding， Class incremental learning， Multi-label learning
Primary area: applications->health_medicine

Abstract: Emotion recognition systems face significant challenges in real-world applications, where novel emotion categories continually emerge and multiple emotions often co-occur. This paper introduces multi-label fine-grained class incremental emotion decoding, which aims to develop models capable of incrementally learning new emotion categories while maintaining the ability to recognize multiple concurrent emotions. We propose an Augmented Emotional Semantics Learning (AESL) framework to address two critical challenges: past- and future-missing partial label problems. AESL incorporates an augmented Emotional Relation Graph (ERG) for reliable soft label generation and affective dimension-based knowledge distillation for future-aware feature learning. We evaluate our approach on three datasets spanning brain activity and multimedia domains, demonstrating its effectiveness in decoding up to 28 fine-grained emotion categories. Results show that AESL significantly outperforms existing methods while effectively mitigating catastrophic forgetting. Our code is available at https://github.com/ChangdeDu/EmoGrowth.

EmoGrowth:
Incremental Multi-label Emotion Decoding with Augmented Emotional Relation Graph
Kaicheng Fu 1
Changde Du
Jie Peng 12 Kunpeng Wang' Huiguang He 123
Shuangchen Zhao'
Xiaoyu Chen 12
Abstract
Class-incremental leaming
V
Emotion recognition systems face significant chal- lenges in real-world applications, where novel emotion categories continually emerge and multi- ple emotions often co-occur. This paper intro- duces multi-label fine-grained class incremen- tal emotion decoding, which aims to develop models capable of incrementally learning new emotion categories while maintaining the abil- ity to recognize multiple concurrent emotions. We propose an Augmented Emotional Semanties Learning (AESL.) framework address two crit- ical challenges: past- and future-missing partial label problems. AESL incorporates augmented Emotional Relation Graph (ERG) for reliable soft label generation and affective dimension- based knowledge distillation future-aware feature learning. We evaluate our approach on three datasets spanning brain activity and mul- timedia domains, demonstrating its effective- ness in decoding up to 28 fine-grained emo- tion categories. Results show that AESL signif- icantly outperforms existing methods while ef- fectively mitigating catastrophic forgetting. Our code is available at https: //github. com/ ChangdeDD/EmoGroon
Task
Adoration
Task 2
Awe
Adoration, Aie
Task
Adoration, Jay
Training Dataset
Test Dataset
Figure 1. The class-incremental learning process is demonstrated through three sequential tasks: Task begins with emotion adora- tion, Task 2 introduces "awe". and Task 3 adds joy. The unified model & with parameters & 2², 4 evolves across tasks while maintaining the ability to recognize all previously learned tions.
conventional deep learning approaches have demonstrated promising performance (Li & Deng, 2020; Poria et al., 2017). they struggle to adapt to the evolving nature of real- world scenarios, where novel emotion categories continu- ously emerge to capture increasingly nuanced emotional experiences. As eloquently expressed by the renowned nov- elist Jeffrey Eugenides (Eugenides, 2003), "Emotions, in my experience, are not covered by single words. I do not believe in 'sadness', 'joy', or 'regret"" " this profound observation underscores the inherent complexity of human emotions. In- deed, individuals typically experience sophisticated blend of multiple emotions simultaneously when responding to emotional stimuli (Fu et al., 2022). Motivated by these in- sights, we introduce a novel research paradigm: multi-labed fine-grained class incremental emotion decoding.
1. Introduction
Accurately decoding human emotional states remains fun- damental challenge in affective computing research. While
"Equal contribution SState Key Laboratory Brain Cognition and Brain-inspired Intelligence Technology, Institute of Automa- tion, Chinese Academy of Sciences, Beijing, China "School of Artificial Intelligence, University of Chinese Academy of Sci- ences, Beijing, 100049, China "School of Biomedical Engineering, ShanghaiTech University, Shanghai, China. Correspondence to: Huiguang He <huiguang.he ia.ac.cn>>
As illustrated Figure 1, multi-label class incremental emotion decoding aims to develop unified model capable of incrementally learning and integrating knowledge from both existing and emerging emotion classes while compre- hensively decoding multiple concurrent emotional states. Unlike traditional single-label class incremental Iearning (SLCIL), the multi-label class incremental learning (ML- CIL) faces unique challenges in addressing catastrophic for- getting, primarily stemming from past- and future-missing
Proceedings of the nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

EmoGrowth: Incremental Multi-label Emotion Decoding with Augmented Emotional Relation Graph
partial label problems. Consider the past-missing partial label scenario: the left screenshot (The corgi is diving) in task 3's training dataset contains the label Adoration, yet this emotion remains imperceptible to the model during the current task. Similarly, in the future-missing partial label case, the right screenshot (The father is combing his daugh- ter's hair) in task I's dataset encompasses the label Joy, which is inaccessible to the model in the current task. Ex- isting MLCIL approaches either rely on storing historical instances (Kim et al., 2020; Liang & Li, 2022), limiting their practical applicability, or overlook the critical future- missing partial label problem (Du et al., 2022), leading to suboptimal performance.
2. Related Work
Class Incremental Learning. Class incremental learning has gained significant attention in machine learning research (De Lange et al., 2021; Masana et al., 2020). Traditional methods mainly focus on preventing catastrophic forgetting through regularization (Kirkpatrick et al., 2017), knowledge distillation (Li & Hoiem, 2017), or memory replay (Re- buffi et al., 2017). Recent advances have explored more sophisticated approaches such as parameter isolation (Serra et al., 2018) and dynamic architecture adaptation (Yan et al., 2021).
Multi-label Class Incremental Learning. MLCIL ad- dresses the crucial challenge of simultaneously handling incremental class learning and multi-label classification. Re- cent works have explored various approaches: (Dong et al., 2023) proposed an attention- based knowledge restore and transfer framework, while AGCN (Du et al., 2022) em- ployed GCN for label relationship learning. Online class incremental learning has been addressed through specialized replay buffer designs in PRS (Kim et al., 2020) and OCDM (Liang & Li, 2022). Additional studies have investigated prototype learning (Zhang et al., 2021) and knowledge dis- tillation (Liu et al., 2022) for MLCIL. However, MLCIL specifically focused on emotion decoding remains largely unexplored. particularly concerning the challenges of partial label problems and emotional semantic preservation.
To address these challenges, we propose a novel Augmented Emotional Semantics Learning (AESL) framework. First, we tackle the past-missing partial label problem by introduc- ing augmented Emotional Relation Graph (ERG) module with graph-based label disambiguation. Upon encountering new tasks, this module not only generates reliable soft labels for existing emotion classes but also constructs an enhanced ERG by integrating historical ERG with new data, thereby preserving crucial emotional label correlations. Second, to resolve the future-missing partial label problem, we lever- age the affective dimension space an alternative emotion model capable of representing infinite emotion categories (Russell & Mehrabian, 1977)- to provide complementary domain knowledge (Le et al., 2023) for continuous emotion learning. This insight leads to our development of relation- based knowledge distillation framework that aligns model features with the affective dimension space. Furthermore, we utilize the ERG to design emotional semantics learn- ing module incorporating a graph autoencoder, which learns emotion embeddings to facilitate semantic-specific feature decoupling, crucial for enhanced multi- learning. We conduct extensive evaluations across three datasets: a hu- man brain activity dataset (Brain27) with 5 subjects and two multimedia datasets (Video27 and Audio28), implementing multiple incremental learning protocols that encompass up to 28 fine-grained emotion categories. Our key contributions are threefold:
Class Incremental Emotion Decoding. The continuous learning of new emotion categories has emerged as crucial research direction in affective computing. (Churamani & Gunes, 2020) proposed CLIFER, combining a generative model with a complementary learning- dual-memory model for continual facial expression recognition. (Ma et al., 2022) developed GCN-base approach for few-shot class-incremental classification across emotion categories, while Huménezz-Gurneet et al., 2022) introduced weight alignment to address bias in new emotion classes. More recent works have explored adaptive architectures (Wang et al., 2023) and meta-learning approaches (Zhang et al., 2022) for emotion recognition. However, these studies are primarily limited to small number of coarse -ranied emo- tion categories and fail to address the complexity of human emotional expression.
We pioneer the investigation of multi-label class incre- mental emotion decoding, advancing emotion recogni- tion capabilities in dynamic real-world environments.
We develop an innovative augmented emotional se- mantics learning framework that enhances emotion de- coding performance while effectively mitigating catas- trophic forgetting in MLCIL scenarios.
3. Methodology
3.1. Problem Formulation
In the MICIII scenario, we have a sequence of B train- ing tasks (D), D2 ....... , ,DB) without overlapping emotion classes, where Db = {(x)), ) is the b-th incremen- tal task with n² training instances. x E IR is an instance with classes Y C Chh Ch is the label set of task b, where
We demonstrate the superior effectiveness of our ap- proach through comprehensive experiments across three datasets and multiple incremental learning proto- cols.
2

---

EmoGrowth: Incremental Multi-label Emotion Decoding with Augmented Emotional Relation Graph
AEG-D
Task
Task b
A
D
Fmotion embeddings
A
x
Y
A
GIN-based encoder
ERG
A
Mi
B
W
Lie
Augmented FRG
Augmented GAF
GLD
R
Importance vectors
Le
Semantic-specific features
Bayes
Pair-wise dccoder
Emotion embeddings
Q
rule
X
AFG-D
W
Wh
A
b-1
W
Hadamard
A
Product
GAE
(a)
(b)
Figure 2. The framework of AESL for multi-label class incremental emotion decoding. (a) Emotional semantics learning and semantic- guided feature decoupling procedure in incremental learning scenario. We omit semantic-guide guided feature decoupling module in task 6 6- 1 for clarity. (b) The process of constructing augmented ERG with label disambiguation in task b.
n² = Ø for * V. Only data from D is accessible during task 6 training. y R c*1 is the multi-hot label vector where sic {0,1} indicates whether emotion is relevant to instance x². After task b, the model is evaluated over all seen emotion classes Ch = C Ch.
given class set C is defined based on label co-occurrence:
Nij N,
Aij = P(4 E C/C, E C)liss =
(I)
where Ni is the number of instances with both class E and fj. N, is the number of instances with class Ej- When the task b is coming, the augmented adjacency matrix A can be formulated as the following block form (Du et al., 2022):
3.2. Framework Overview
As shown in Figure 2(a), AESL processes each incremental task through four interconnected modules. The ERG mod- ule first constructs and maintains the emotion relationship graph, providing the foundation for emotional semanties learning. The graph autoencoder (GAE) then learns emotion embeddings from ERG, which guide the feature decoupling module to extract label-specific features. Finally, relation- based knowledge distillation preserves previously learned knowledge while accommodating new emotion categories. These components work jointly to address both past-missing and future-missing partial label problems MLCIL.
A
Rd B
Old-Old
Old-New
A
(2)
New-Old New-New
A can be directly inherited from task b, and B can be easily computed from D'. However, R2 and Q' involve the inter-task label relationship between old classes in past tasks and new classes in task b. We should first assign soft labels in the past label set Ch for the instances in new dataset D for subsequent calculation. For clarity, we use (xx A; b denote the procedures of emotional semantics learning and semantic-guided feature decoupling, with 4 = {0, Φ. W). Although s (xxx, bb-1, gb-1) is a feasible solution for the soft labels construction, this kind of soft labels contain a significant amount noise and fail to utilize the correlation among instances.
3.3. Augmented Emotional Relation Graph
Here, we first introduce the Augmented Emotional rela- tion Graph module with label Disambiguation (AEG-D), as shown in Figure 2(b). At the beginning of task have access to new labels Cp. Compared with the existing emo- tional relation graph gh-1, we need to augment the node set and adjacency matrix to 1 and A respectively. For the former, we only need to sample (Cal vectors from the standard Gaussian distribution. For the latter, it is difficult infer A directly from statistical label co-occurrence due the partial label problem. The adjacency matrix the
To tackle this problem, we adopt Graph-based Label Disambiguation (GLD) module to the label confidence score sh Firstly, the similarity between two instances is calculated with Gaussian kernel (omit b without ambiguity) Pij
exp(- |xx,-x,112,, ), in which x; and x, are two different sam- 20² ples in Db. Following the label propagation procedure, let P. PD-1 be the propagation matrix by normalizing
3

---

EmoGrowth: Incremental Multi-label Emotion Decoding with Augmented Emotional Relation Graph
weight matrix P in column, where D [di :d,]]
in which H441 E RCC* xde1 is the updated feature matrix of nodes, [1+1 0t41) refers to fully-connected neural network. Additionally, E1+11 is a learnable parameter which regulates the importance of the node's own features during the process of neighborhood aggregation. Unlike object labels in image classification (Chen et al., 2019), emotion category labels are difficult to obtain initial word embed- dings directly from language models. Consequently, the initial feature matrix of nodes H E |x is initialized by standard Gaussian distribution and we set do = |C*] (can also be other task-agnostic constants). After stacking L GIN layers, we use E = H. E R|C" xdz as the final emotion label semantic embeddings in task b for further semantic-specific feature extraction.
is the diagonal matrix with d; Σi= P Pij- Assume that we have access to a past label confidence matrix using (: 1 for Dd. which denotes as S E _= g=*x|c*='1, And we set the initial label confidence matrix Fo- S. For the 2-th iteration, the refined label confidence matrix is updated by propagating current labeling confidence over P:
F.- = 3 P' + (1- - 3) Fo-
(3)
The balancing parameter 3 E [0. 1] controls the amount of labeling information inherited from iterative label propaga- tion and F9. Let F be the final label confidence matrix and also serve the soft labels after disambiguation, which means S = F*. We set the balancing parameter 3 to 0.95 during label disambiguation according to (Chen et al., 2020).
Furthermore, we introduce a pairwise decoder to reconstruct the adjacency matrix A which can ensure that the obtained label embeddings capture the topological structure of the label semantic space well. The loss function of the pairwise decoder can be written as:
With the dataset Do and soft label matrix S, we are able to compute Rè E - as follows:
"|x|c*I
= Ex 8,03 (4)
R = E C ECO)=
N
in which &; denotes the value of class i corresponding to instance x in the soft label matrix, and y/ refers to the value of class j corresponding to the same instance in the label matrix derived from DP Naturally, following the Bayes' rule, we can obtain the Qº E &R @RI""" by:
|c*1 |c*1
(c [e
o (c)) 6²) et ell - e0|l
Lie cc
AbP. (7)
where @b = [. corresponds to the average emotion embeddings in task b, ef denotes the i-th row of E and A = A + Ih, with Is being an identity matrix. Over- all, emotional semantics learning aims to fit the function [- 0²), leading to E = J(H), A1;00) in task b.
Q¹;- = P(e, E C" E 1
P(e: E C0- E Ch) ECa)_ C')
R
E(
Σx $
(5)
3.5. Semantic-Guide Feature Decoupling
Above all, we have constructed the adjacency matrix A and achieved continual learning of new emotion categories in the multi-label scenario. It is noticeable that, in our experiments, we actually utilize a symmetric adjacency matrix for model training by applying the A+A" operation.
A multi-label classification model with semantic-guided fea- ture decoupling can be regarded as the composition of a semantic-specifie feature extractor g and a classification head W (omitting bias for simplicity), where g(: d²) R x |c'" + Rlc'|xd and Wh E Rdx|c") in task b. For a specific emotion label le E Cb, the semantic-specific mapping 9k can be formulated as (x(, e d") E Rd, The linear layer can be further decomposed into the combina- tion of classifiers W [W1,'.. w W/ in which each classifier corresponds one emotion embedding. The clas- sification head will be expanded for new classes as the continual learning progresses. The key issue is to design the semantic-speciffe feature extractor gk-
2
3.4. Emotional Semantics Learning
Now, we focus on how to obtain the emotion embeddings in task b. We construct an augmented ERG G = (Vo, {b)). F²), where 5 represents nodes corresponding to class labels C³ and Eb refers to edges. Then, we adopt a GAE to project emotion labels into label co-occurrence semantic space with the ERG. We exploit Graph Isomorphism Network (GIN) (Xu et al., 2019) as the encoder of our GAB due to its powerful representation learning capability. Specifically, given feature matrix of nodes H E [xc" in which each row refers to the embedding of an emotion label and di corresponds to the dimensionality of node features in the 1-th GIN layer, the node features are able to update within a GIN layer with a message passing strategy by:
To achieve this, we first map the instance representation x² from the original feature space to a more powerful deep latent feature z with a fully-connected network having pa- rameters W Dxd= and b.: Rdd In order to uti- lize emotional semantics to guide the feature extraction for each label, we adopt an attention-like mechanism. Con- cretely, we attempt to obtain feature importance values ag for each emotion category by using a fully-connected net- work followed by a sigmoid function for eğ with parameters
Hig = f1+1((1 + + + A H 0.
(6)
4

---

EmoGrowth: Incremental Multi-label Emotion Decoding with Augmented Emotional Relation Graph
Tuned
Froeen
Arousll
similarity matrix (RSM) (Kriegeskorte et al., 2008) obtained from model feature for task b:
Anger
(active)
Joy
Turious
excited
Teacher]
Student
Teacher2
annoyed
delighizc
M
2,
2) T [zj 2) Z
(10)
W
WE
V-A
dissgusted
blissful Valence
2
M B-11
M
M
(negative))
disappointed
content (postive)
where 2 = E, [z.] denotes the mean model feature of all in- stances. Similarly, RSM obtained from affective dimension feature 7 can be formulated as:
Jepressee
surune
gacited
relaxed
Sadness
Pleasure
Cadmaate
Cichatr
(pseiee)
(T; - +) (T, - ITi Tj - -
M
(1I)
Figure 3. Diagram of clation-bbsed knowledge distillation with two teachers in the process of training task b. Teachers 1&2 were frozen during training. Fach discrete emotion category represents a point in the affective space formed by Arousal and Valence.
Then, we define the similarity loss Lind as:
Lkd -Ei -Eift (larctanh(M (M²)) - arctanh (1.)
We E Rd, and by E Rd= Then, we select pertinent features for each emotion category via the Hadamard prod- uct between the feature importance vector and the latent representation. Successively, we can obtain the semantic- specific feature for each emotion category from another fully-connected network. This procedure can be formulated as follows:
(12)
where Ly is the sample- based centered kernel alignment index. We leverage arctanh reparameterize the similarity values from the interval (-1,1) to (-30, approxi- mately obey a Gaussian distribution. Besides, to ensure training stability, we simultaneously pull together M and M using the same method to obtain Lidaaan- In this way, the model has access two teachers: the affective dimen- sion and the old model, as shown in Figure 3. This approach derives the overall knowledge distillation loss:
OK - <(Wo (o axk + bbJ,
where W Rd.xd and by Rd are shared learnable pa- rameters. @ refers to the Hadamard product, and denotes the activation function. At this point, we have defined the semantic-specific feature extractor Oμ- g& (xb, e Φ²),, in which ø = (W., We. W.. b,, be. }. Then, we can pre- dict the confidence score of the presence of emotion label & through the corresponding classifier:
Lud X Liddasta + X
(13)
3.7. Objective Function
8k ow k++ + bb) aw el; bg)-
As mentioned above, the prediction confidence scores for an instance x can be computed by Eq. 9, which denotes RIC*1 in task b. We have access to the ground truth from D', which is denoted as the multi- hot vector y = "".'""""""""" E RIC"), Additionally, we have computed the soft labels for the previous emotion categories using the model b 1, as described in Section 3.3, which are denoted as $ = [81, & E IR TRIC""), and Oh U C = C³. In summary, we train task b using the mixed ground truth ý isT y E RIC*) with a the binary cross entropy loss, formulated as follows:
(9)
e (1....,IC'}} |
[s , s|c=
3.6. Relation-based Knowledge Distillation
Although we have achieved the semantic-specific feature learning and overcome the past-missing partial label prob- lem by AEG-D, we have not yet addressed the issue of future-missing partial label problem in MLCII.. Previous studies (Schlosberg, 1954; Russell & Mchrabian, 1977) have shown that the affective dimension, as a complementary emotion model to emotion category, can represent infinitely many emotion categories within its constructed affective space, We propose that incorporating the domain knowledge of affective dimension space into the model can alleviate the problem of future-missing partial label problem. During the training process for each task, attempt to align the fea- ture space of our model with the predefined affective space constructed by some affective dimensions such as Arousal and Valence. Taking into account the heterogeneity of the two spaces, we adopt relation- celation-based knowledge distillation (RKD). Specifically, we firstly calculate the representation
|C*1
Lee =
> (slog(()) + (1- - T. log(1 - -s)]. (14)
Finally, our model is trained with the following objective function in an end-to-end manner:
L Lee Lx + dee + XyLLe.
(15)
After training in task b, given an unseen instance, its asso- ciated label set is predicted as {la|sk > 0.5, 1 < k S C*}. The algorithm of AESL. is written in Appendix A.
5

---

EmoGrowth: Incremental Multi-label Emotion Decoding with Augmented Emotional Relation Graph
Finetune
EWC
LwF
ER
RS
AGCN
PRS
OCDM
KRE-R
AESL
B0-19
B15-33
45-
0'
93
35-
35-
-5
20
25
30
30
20
15 21
24
13
21 Numbor classes
17
13
23
23
Number of classes
Number of classes
Number of clcsssss
(a) Brain27(1)
RB-13
R15-12
3.0
78-
02-
mAP
60-
13
50-
25-
15
15
20
34
15
19
31
25
Number of classes
Number of classes
Number of classes
Number of classes
(b) Video27
DO-17
D0-14
D16-13
B16-12
30
45
1AP
40
35
14
21
16
25
20
18
26
24
26
Number of classes
Number clases
Number of classes
Namber of classes
(c) Audio28
Figure 4. Comparison results (mAP) three datascts used in experiment under different protocols against compared CIL methods.
4. Experiments
three well-established MLCIL approaches AGCN (Du et al., 2022), PRS (Kim et al., 2020), OCDM (Liang & Li, 2022), and KRT-R (Dong et al., 2023) are also employed as com- paring approaches. Besides, we set the Upper-bound as the supervised training on the data of all tasks. More details about the experimental setups can be found Appendix B.
4.1. Experimental Setup
Datasets, For thoroughly evaluating the performance of AESL and comparing approaches, three datasets are lever- aged for experimental studies including Brain27 (Horikawa et al., 2020), Video27 (Cowen& & Keltner, 2017) and Audio28 (Cowen et al., 2020). We evaluate our method using two popular protocols in class incremental learning work (Dong et al., 2023), including (1) training all emotion classes in several splits and (2) first training a base model on a few classes while the remaining classes being divided into sev- eral tasks. For Brain27 and Video27, we split the datasets with BO-19 (base class is 0 and incremental class is 9), B0-13, B15-13 and B15-12. For Audio28, we split the dataset with B0-17, B0-14, B16-13 and B16-12.
4.2. Experimental Results
Comparative Studies. Tables 1, 2. and show the re- sults on subject of Brain27 (more results are shown in Appendix C), Video27 and Audio, respectively. We can ob- serve that AESL shows obvious superiority under different datasets and protocols, in terms of three widely used metrics mAP, maFl and miF1 (Zhang & Zhou, 2013). Especially for Brain27, AESL has a relative improvement of 9.6% in mAP (4 protocols averaged) and 9.7% in maF1 compared with the second place method. Figure 4 exhibits the comparison curves of AESL and comparing methods, which indicates that our method is consistently optimal at each task of incre- mental learning. Among the compared methods, EWC and LwF are traditional SLCIL methods. We can find that EWC is not suitable for direct application to MLCIL task due to its poor performance. In contrast, LwF achieves impressive per-
Baselines. The performance of AESL is compared with multiple essential and state-of-art class incremental meth- ods. Finetune baseline which means fine-tuning the model without any anti-forgetting constraints. We select four SLCIL methods including EWC (Kirkpatrick et al., 2017), LwF (Lee et al., 2019), ER (Rolnick et al., 2019) and RS (Vitter, 1985) for comparison. Furthermore, other
6

---

EmoGrowth: Incremental Multi-label Emotion Decoding with Augmented Emotional Relation Graph
Table Class incremental results on subject 1 of Brain27 dataset. AGCN, PRS, OCDM and KRT-R are MICIII algorithms among these
compared methods.
Brain27 B0-19
Brain27 B0-13
Brain27 BIS-13
Brain27 B15-12
Method
Avg. Ace
Last Ace
Avg. Ace mAP
Last Acc
Avg. Ace
Last Acc
Avg. Acc
Last Acc
mAP
maFi
miFl
mAP
mAP
maFi
miFi
mAP
MAP
maFi
miFl
mAP
Upper-bound
47.2
39.1
47.2
45.8
39.1
45.8
Finetone
34.8
9.2
19.3
26.0
30.8
5.0
13.8
21.2
26.0
5.0
13.9
21.1
23.7
18.7
EWC
33.2
17.1
30.9
5.0
13.9
5.3
14.3
22.1
19.5
LwF
37.3
12.2
29.1
29.1
37.0
23.4
40.0
27.0
31.9
14.2
28.6
24.9
29.1
15.2
31.8
20.4
ER
40.7
8.0
11.7
36.2
40.2
4,9
9.1
34.3
37.9
9.7
12.3
35.4
37,5
9.6
11.7
33.7
RS
42.1
9.4
12.6
39.8
41.2
4.5
7.4
33.3
38.3
8.0
11.1
33.9
37.9
5.1
8.7
32,5
AGCN
42.2
29.5
44.5
40.4
42.1
35.4
43.7
34,9
39.3
28.8
41.7
37.2
36.3
24.4
36,4
30,9
41.6
41.7
5.5
8.4
8.2
36.1
33.2
OCDM
41.6
9.7
15.9
37.7
40.6
5.3
7.6
3166
37.2
4.9
7.1
31.4
37.3
4.4
7.6
32.5
KRT-R
42.5
18.2
30.3
41.1
44.3
22.9
32.1
39.1
40.4
20.0
33.2
39.1
39.5
20.3
33.7
37.1
AESL
44.2
32.8
44.7
42.6
43.8
37.1
44.0
36.9
41.9
32,5
41.7
39.3
39.5
26.8
36.5
34,8
Table Class incremental results Video27 dataset. AGCN, PRS, OCDM, and KRT-R MLCIL algorithms among these compared
methods.
Video27 BO-19
Video27 B0-13
Video27 B1S-I3
Video27 B15-12
Method
Avg. Ace mAP
Last Ace
Avg. Ave mAP
Last Acc
Avg. Aue mAP
Last Acc
Avg. Ace
Last Ace
maF1
miFI
mAP
maFI
miFI
mAP
maFi
miFI
mAP
mAP
maFi
miFI
mAP
Upper-bound
36.8
46.3
45.4
36.8
46.3
45.4
36.8
46.3
45.4
36.8
46.3
45.4
Finetune
33.8
6.6
13.7
25.1
31.5
4.2
13.0
21.7
26.0
4.3
13.3
21.6
24.3
4,2
13.7
19.1
EWC
35.6
17.8
27.8
32.9
4.7
13.2
23.6
28.4
4.9
13.9
23.3
26.0
13.1
19.9
LWF
34.4
23.3
38.2
19.9
37.3
24.7
12.5
33.4
24.2
20.4
ER
42.0
5.1
7.0
39.0
43.0
4.5
4,8
35.4
39.4
8.0
8.3
37.6
37.2
10.1
12.6
34,0
RS
42.8
4.6
6.1
40.1
43.6
4.6
7.8
34.5
39.3
4.2
5.2
37.0
37.2
6.7
9.9
32.0
AGCN
42.1
22.4
39.4
39.2
44.5
34.2
44.5
36.1
39.5
22.7
38.4
37.0
38.0
23.8
36.2
34,0
42.6
15.0
40.4
43.0
5.8
9.6
33.4
37.8
8.9
13.4
36.0
37.7
13.3
33.7
OCDM
43.1
43.8
5.0
7.8
6.6
36.3
32.7
KRF-R
42.9
26.7
35.8
40.1
45.5
26.3
34.7
37.0
41.5
25.0
35.5
40.4
39.5
24.2
34.2
38.3
AESL
44.6
23.4
39.7
41.9
47.1
35.2
45.0
37.1
41.5
23.5
39.2
38.1
39.8
24.5
36.7
36.1
Table Class incremental results on Audio28 dataset, AGCN, PRS, OCDM and KRT-R are MLCIL algorithmm among these compared
methods.
Andio28 B0-17
Audio28
R16.13
Audiozs
Method
Avg. Are mAP
Last Acc
Avg. Ave mAP
Last Ace
Avg. Ave mAP
Last Ace
Avg. Ace mAP
Last Are
maFi
miF1
mAP
maFi
miFI
mAP
maFi
miF1
mAP
maFi
miFi
mAP
Upper-bound
51.4
61.1
57.1
51.4
61.1
57.1
51.4
61.1
57.1
51.4
61.1
57.1
Finetune
36.4
9.2
14.8
27.3
33.9
5.3
10.0
23.3
29.9
4,4
10.3
22.6
27.6
2.8
8.2
20.2
EWC
37.9
8.3
14.3
29.3
37.1
5.4
10.5
26.6
32.2
4.4
9.7
24.7
28.
2.8
8.7
22.5
LWF
46.6
37.9
51.7
45.8
39.9
49.8
37.6
45.0
32.3
45.2
40.0
42.3
28.8
41.4
36.5
FR
44.7
11.4
38.0
45.6
6.5
5.5
35.2
41.3
9.2
13.3
36.8
39.4
10.1
13.6
34,1
RS
43.7
8.1
12.3
36.5
43.6
5.9
9.3
32.0
38.7
7.5
11.7
32.9
38.2
5.8
11.6
31.8
AGCN
47,3
35.3
50.9
41.9
46.6
37.5
51.0
38.6
44.5
29.3
44.6
39.6
41.1
27.5
42.4
34.8
PRS
43.3
9.0
12.8
35.5
44.5
6.8
9.0
34.2
39.2
6.2
8.3
32.9
39.1
8.8
11.4
35.2
OCDM
44.5
12.0
43.8
7.5
8.8
31.5
38,2
5.5
9.7
30.1
36.3
3.7
7.9
30.2
KRI-R
46.3
47.3
18.1
33.0
27.9
38.9
38.8
AESL
49.0
38.4
518
42.7
48.7
41.1
51.7
39.8
47.8
32,3
48.0
42,3
45,3
30.8
45.1
39,3
formance on the Audio28 dataset. We infer that knowledge distillation, which essentially provides soft labels for old classes, contributes to overcoming catastrophic forgetting in MI.II.. Besides, is noticeable that rehearsal-basee meth- ods, including ER, RS, PRS, OCDM and KRT-R, are not satisfactory, especially for maF1 and miF1. This is because just saving the labels of current task aggravates the partial label problem in subsequent training, which illustrates that data replay is not suitable for MLCIL. Furthermore, AGCN serves as a strong baseline which can rank 2nd for most cases, and our method surpasses AGCN by incorporating ef- fective label semantics learning and introducing knowledge
distillation from the affective space.
Furthermore, we adopt the Friedman test (Demšar, 2006) for statistical testing in order to discuss the relative per- formance among the compared methods If there are algorithms and N datasets, we take use of the average ranks of algorithms R, N Σ; r! for Friedman test in which r! is the ranks of the j-th algorithm on the i-th dataset. If the null- mutt-hypothesis is that all the algorithms have the equivalent performance, the Friedman statistic Fr which will satisfy
We average the last mAP of four protocols regarding each dataset for further analysis.

---

EmoGrowth: Incremental Multi-label Emotion Decoding with Augmented Emotional Relation Graph
8
7
6
5
2
7
2
FWC Finetune
ALSL OCDM AGCN LwF PRS
ALSL AGCN
RS OCDM
ER
LwF
PRS
Finetune
ER
EWC
(a) maF1
(b) miF1
Finetune EWC
ALSL
AGCN
OCDM RS
PRS
(c) mAP
Figure 5. Pairwise comparisons with the Nemenyi test 7 datasets and 9 algorithms used in our experiments. Algorithms not connected with each other in the CD diagram are considered to have significantly different performance (CD 4.540 at 0.05 significance level),
the F-distribution with and (A: - () 1) degrees of freedom can be written as:
Table 4. Friedman statistics Fp in terms of cach metric and the critical value 0.05 significance level. (# compared algorithms
(N 1)x2
k = 9. # subjects N = 7.)
FF N(k 1) XF
(16)
maF1
miF1
mAP
critical value
37.025
64.000
70.696
2. 2.138
in which
12N XF (:Kk + 1)
2
Σ
R2
k(K + 1)
(17)
Table 5. The contribution of each component. Accuracy of these models is measured by mAP.
4
Model
ESL.
RKD
Arg. Ace
A
Last
Table shows the Friedman statistics Fr and the corre- sponding critical value in regard to each metric (# compar- ing algorithms k 9, # datasets N 7). With respect to each metric, the null hypothesis of equivalent performance among the compared methods can be rejected at the 0.05 significance level.
ESL&LD
47.6
-1.4
41.3
-1.4
+SE
45.3
-3.7
40,9
-1.8
HLE
46.8
-2.2
41.9
-0.8
48.1
RKD
48.3
+LR
47.0
-2.0
41.8
-0.9
AESL
49.0
0.0
42.7
0.0
Then, we perform the strict post-hoc Nemenyi test (Demšar, 2006) which is used account for pairwise comparisons for all compared approaches. The critical difference (CD) value of the rank difference between two algorithms is:
AESL and other MLCIL approaches (except OCDM) sig- nificantly outperform SLCIL methods. (2) terms of miF1 and maF1, rehearsal-free methods are significantly better than rehearsal-based algorithms.
(3) AESL is not signifi- cantly different from some MLCIL approaches. This is due to the factor that these approaches beat other comparing approaches and the Nemenyi test fails to detect that AESI. achieves a consistently better average ranks than other meth- ods on all evaluation metrics.
k<k 1)
CD = /a
(18)
6.N
in which fa = 3.102 at 0.05 significance level. There- fore, one algorithm can be considered as having signifi- cantly different performance than another method if their average ranks difference is larger than CD (CD 4.540 in experimental setting). Figure 5 reports the CD di- agrams on each metric, where the average rank of each compared method is marked along the axis (the smaller the better). The algorithms that are not connected by hori- zontal line are considered to have significant differences in performance. We can observe that: (1) In terms of mAP,
Ablation Studies. To evaluate the roles of AESL's three key components, we conduct ablation experiments on the Audio28 dataset under the B0-17 setting. We design seven baselines, as shown in Table 5: (1) w/o ESL&LD: Feed the feature vector z directly into the classifier without semantic- guided feature decoupling. (2) w/o ESL&LD, +SE: Extract sentence embeddings of emotion category descriptions via
8

---

EmoGrowth: Incremental Multi-label Emotion Decoding with Augmented Emotional Relation Graph
Augmennted ERG (AESL-woo RKD)
Augmensed
Orack ERG
(AESL)
Scrull
05
04
Tranor Colpusdion
0.75
(a)
(a) Brain27
(b) Audio28
42.5
Figure 6. t-SNE on the emotion embeddings learned by AESL..
42.0
42.0
41.5
41.5
41.0
+1.0
the advanced LLaMA 3.B (Dubey et al., 2024) and utilize them as emotion embeddings. (3) w/o ESL&LD, +LE:
40.5-
60.5
400
V18-27
40.0
140-27
Assign learnable embedding to each new emotion category. (4) w/o ESL&LD, +AD; Directly integrate the sample-wise affective dimension into category-wise emotion embeddings. (5) w/o LD: Use original confidence score for constructing adjacency matrix without label disambiguation. (6) w/o RKD: Remove the module of knowledge distillation from affective space. (7) w/o RKD, + LR: Replace RKD with a linear regressor with a non-linear activation to predict the target affective dimension, thereby constraining z and incorporating the affective dimension.
19.1
02
0.5
014
0.5
395
165
(b)
Figure 7. (a) Visualization of augmented emotional relation graph in Audio28 dataset. (b) The performance AESL measured by the last mAP changes as X2 and λ vary.
ERG. Using the proposed method, inter- and intra- label relationships are reconstructed well. Besides, by incorporat- ing RKD from the affective dimension space, the augmented ERG is closer to the oracle ERG (r 0.86
is noteworthy, but not surprising, that even when sentence embeddings are extracted using state-of-the-arr large lan- guage models, their usage emotion category labels fails to achieve comparable performance to semantic-guides fea- ture decoupling. This performance degradation suggests that the effectiveness of AESL relics not only on the quality of embeddings but also on the explicit structural alignment provided by semantic-guided decoupling, Results show that all the three components in AESL are critical for preventing forgetting and improving the model performance of MLCIL.
0.75).
Parameter Sensitivity. Figure 7(b) gives an illustrative example of how the performance of AESL changes as the regulation parameters A2 and A vary on the B0-19 and B0-17 protocols of three datasets (A1 is fixed to 1). Here, when the value of one parameter varies, the other is fixed to a reasonable value. We find that too large value of X2 dramatically degrades the model performance due to the noise in affective ratings, while too small value will not play the role of alleviating the future-missing partial label problem. As for Ag too large value will cause the model to focus too much on graph reconstruction, while too small value will not be able to learn label embeddings well, which both lead to a decline in model performance.
Emotional Semantics Visualization. Emotional semantics learning is an essential module for our approach. In Figure 6, we adopt the t-SNE (Van der Maaten Hinton, 2008) to visualize the emotion embeddings learned by the emo- tional semantics learning module. It is clear to see that, the learned embeddings maintain meaningful emotional seman- tic topology. Specifically, in Brain27, positive emotions are predominantly distributed in the bottom right, while neg- ative emotions are distributed in the top left. In Audio28, Bittersweer is just located between Sad and other positive emotions. This visualization further demonstrates the neces- sity of modeling label dependencies.
5. Conclusion
In this paper, we have proposed a novel AESL framework for multi-label class incremental emotion decoding. In de- tail, we developed an augmented ERG generation method with label disambiguation for handling the past-missing partial label problem. Then, knowledge distillation from affective dimension space was introduced for alleviating the future-missing partial label problem. Besides, we con- structed an emotional semantics learning module to learn indispensable label embeddings for subsequent semantic- specific feature extraction. Extensive experiments have il- lustrated the effectiveness of AESI..
Augmented ERG Visualization. In Figure 7(a), we provide the augmented ERG visulization on Audio28 dataset. We utilize the oracle ERG, which is constructed using ground truth label statistics of all tasks, as the upper bound. We also compute the Pearson's 's Correlation Coefficients (PCCs) to measure the similarity between the augmented and oracle

---

EmoGrowth: Incremental Multi-label Emotion Decoding with Augmented Emotional Relation Graph
Acknowledgments
Proceedings of the national academy of sciences, 114 (38):E7900-E790) 2017.
This work was supported part by the National Key R&D Program of China 2022ZD01 16500; in part by the Na- tional Natural Science Foundation of China under Grant 62206284; and in part by Beijing Natural Science Founda- tion under Grant 6; and in part by the Beijing Nova
Cowen, A. S., Fang. X., Sauter. D., and Keltner, D. What music makes us feel: At least 13 dimensions organize subjective experiences associated with music across dif- ferent cultures. Proceedings of the National Academy of Sciences, 117(4):124--133) 2020.
Program under Grant 20230484460.
De Lange, M., Aljundi, R., Masana, M., Parisot, S., Jia, X., Leonardis, A., Slabaugh, G., and Tuytelaars, T. continual learning survey: Defying forgetting in classifi- cation tasks. IEEE Transactions Pattern Analysis and Machine Intelligence, 44(7):3366-338 2021.
Impact Statement
This research is motivated by the growing role of affective computing in shaping human-machine interactions. Ad- vances in deep learning for emotion recognition and syn- thesis carry significant societal implications: Improved emotion-aware systems can enhance mental health support (e.g., Alaasisted therapy), education (e.g., adaptive learning systems), and human-computer collaboration (c.g., empa- thetic chatbots).
Demsar, J. Statistical comparisons of classifiers over multi- ple data sets. The Journal of Machine Learning Research, 7:1-30, 2006.
Dong. S., Luo, H., He, Y., Wei, X., Cheng, J., and Gong, Y. Knowledge restore and transfer for multi-label class- incremental learning. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 18711- 18720, 2023.
However, this technology presents inherent dual-use risks. While our models may help detect distress signals in clin- ical settings, the same techniques could be repurposed for unethical surveillance or manipulation (e.g., non-consensual emotional profiling). The accuracy-limitatee limitationns of emotion models also raise concerns about cultural bias or misinter- pretation of nuanced emotional states, potentially leading to harmful decisions in sensitive applications.
Du, K., Lyu, F., Hu, F., Li, L., Feng, W., Xu, F,, and Fu, Q. Agen: augmented graph convolutional network for lifelong multi-label image recognition. In 2022 IEEE In- ternational Conference on Multimedia and Expo (ICME), pp. 01-06. IEEE, 2022.
We argue that the benefits of responsible affective comput- ing pas--atcuullaly in healthcare and accessibility-outteeee the risks when accompanied by rigorous ethical safeguards. This work emphasizes the need for transparent model limi- tations, user consent protocols, and nomain-speciffe deploy- ment guidelines to mitigate misuse.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv: 2417.22407.2172 2024.
Eugenides, J. Middlesex. Anagrama, 2003.
References
Fu, K., Du, C., Wang. S., and He, H. Multi view multi- fine-grained emotion decoding from human brain activity. IEEE Transactions on Neural Networks and Learning Systems, 2022.
Chen, Z.-M., Wei, X.-S., Wang, P., and Guo, Y. Multi-label image recognition with graph convolutional networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5177-5186, 2019.
Glasser, M. F., Coalson, T. S., Robinson, E. C., Hacker, C. D., Harwell, J., Yacoub, E., Ugurbil, K., Andersson, J., Beckmann, C. F., Jenkinson, M., et al. A multi-modal parcellation of human cerebral cortex. Nature, 536(7615): 171-178, 2016
Chen, Z.-S., Wu, X., Chen, Q-GG,, Hu, Y., and Zhang, M.-L. Multi-view partial multi-label learning with graph-based disambiguation. In Proceedings of the AAAI Confer- ence on artificial intelligence, volume 34, pp. 3553-3560, 2020.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learn- ing for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Churamani, N. and Gunes, H. Clifer: Continual learning with imagination for facial expression recognition. In 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020), pp. 322-328. IEEE, 2020.
Horikawa, T., Cowen, A. S., Keltner, D., and Kamitani, Y. The neural representation of visually evoked emotion is nigh-dimensional, categorical, and distributed across transmodal brain regions. Iscience, 1315),001060, 2020.
Cowen, A. S. and Keltner, D. Self-report ceaf-reportcaptures 27 distinct categories of emotion bridged by continuous gradients.
10

---

EmoGrowth: Incremental Multi-label Emotion Decoding with Augmented Emotional Relation Graph
Timénez-Guamentt Magdiel, Alejo-Eleuteri, and Roberto. A class-incremental learning method based preserving the learned feature space for eeg-based emotion recogni- tion. Mathematics, 10(4):598, 2022.
IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 652-653, 2020.
Poria, S., Cambria, E., Bajpai, R., and Hussain, A. A review of affective computing: From unimodal analysis to multimodal fusion. Information fusion, 37:98 57:98-125, 2017.
Kim, C. D., Jeong. J., and Kim, G. Imbalanced continual learning with partitioning reservoir sampling. In Com- puter Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIII 16, pp. 41 411-428. Springer, 2020.
Rebufli, S.-A., Kolesnikov, A., Sperl, G., and Lampert, C. H. icarl: Incremental classifier and representation learning. Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 2001-2010, 2017.
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Des- jardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska-Barde Barwinska, A., et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, (14(1))332213355 2017.
Rolnick, D., Ahuja, A., Schwarz, J., Lillicrap. T., and Wayne, G. Experience replay for continual learning. Ad- vances in Neural Information Processing Systems, 32, 2019.
Kriegeskorte, N., Mur, M., and Bandettini, P. A. Repre- sentational similarity analysis-connecting the branches of systems neuroscience. Frontiers in systems neuroscience, pp. 4, 2008.
Russell, J. A. and Mehrabian, A. Evidence for three-factor theory of emotions. Journal of reseurch in Personality, 1 11(3):233-24, 1977.
Schlosberg, H. Three dimensions of emotion. Psychological review, (2):881 1954.
Le. N., Nguyen, K., Tran, Q.. Tjiputra, E., Le, B., and Nguyen, A. Uncertainty-aware label distribution learn- ing for facial expression recognition. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp.
Serra, J., Suris, D., Miron, M., and Karatzoglou, A. Over- coming catastrophic forgetting with hard attention to the task. International Conference on Machine Learning, pp. 4548 4557, 2018.
2023.
Lee, K., Lee, K. Shin, J., and Lee, H. Overcoming catas- trophic forgetting with unlabeled data in the wild. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 312-321,2011 2019.
Simonyan, K. and Zisserman, A. Very deep convolu- tional networks for large-scale image recognition. arXiv preprint 1rXiv:1409.1556, 2014.
Van der Maaten, L. and Hinton, G. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
Li, S. and Deng, W. Deep facial expression recognition: A survey. IEEE transactions affective computing, 13(3): 1195-1215, 2020.
Vitter, J. S. Random sampling with reservoir. ACM Transactions on Mathematical Software (TOMS), 11(1): 37-57, 1985.
Li, Z. and Hoiem, D. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2935-2447, 2017.
Wang, H., Zhang, L., Li, X., and Zhao, G. Adaptive emotion recognition through progressive knowledge distillation. IEEE Transactions on Affective Computing, 14(2):1012- 1024, 2023.
Liang, Y.-S. and Li, W.-J Optimizing class distribution in memory for multi-labe online continual learning. arXiv preprint arXiv:2209.1 |469, 2022.
Xu, K., Hu, W., Leskovec, J., and Jegelka, S. How powerful are graph neural networks? In Proceedings of the 7th International Conference on Learning Representations, 2019.
Liu, Y., Feng, W., Sun, J., and Zhang, X. Incremental multi- label learning with dual knowledge distillation. IEEE
Transactions on Neural Networks and Learning Systems, 34(5):2132-2146, 2022.
Yan, S., Xie, J.,, and He, X. Dynamically expandable repre- sentation for class incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3014-3023, 2021.
Ma, T.FF., Zheng, W.-L., and Lu, B.-L. Few-shot class- incremental learning for eeg-based emotion recognition. In International Conference on Neural Information Pro- cessing, pp. 445-455. Springer, 2022.
Zhang, K., Liu, F., Liu, M., and Yang. J. Meta- Meta-lcarning for adaptive emotion recognition in dynamic environments. In Proceedings of the 30th ACM International Conference on Multimedia, pp. 2167-2175, 2022.
Masana, M., Liu, X., Twardowski, B., Menta, M., Bagdanov, A. D., and van de Weijer, J. Class-incremennal learning: survey and performance evaluation, In Proceedings of the

---

EmoGrowth: Incremental Multi-label Emotion Decoding with Augmented Emotional Relation Graph
Zhang, M.-L. and Zhou, Z.-H. A review multi-label learning algorithms. IEEE transactions knowledge and data engineering, 26(8):1819-1877, 2013.
Zhang, Z., Wang. H., Luo, X., and Li, Q. Prototype-base based multi-label class incremental learning. In 2021 IEEE International Conference Image Processing (ICIP), pp. 2900-2907, 2021.
12

---

EmoGrowth: Incremental Multi-label Emotion Decoding with Augmented Emotional Relation Graph
A. The Algorithm of AESL
Algorithm 1 Training procedure of AESL.
Input: Training sequence {D).... . Hyperparameters o, B, X1 22, A3- Affective dimension features
{r) .r").
for b = 1 B do
while not converged do for (x", y²) ~ Do do
if then
Compute A directy with label matrix Y using Eq.1.
else
Compute soft label matrix S with (xx", A A 10-1)) and set intial label confidence matrix Fo S. Compute the normalizing weight matrix p PD- P is the similarity between two instances in D>. Implement label propagating to obatin refined soft label matrix S with Eq.3. Compute BA directy with label matrix Y using Eq.
A 6-1
Rd
Compute RR and Q' with S and Y using Eq.4 and 5, then obtain A
Q
B¹
{Get augmented ERG shown in Section 3.3.)
end if
Construct (augmented) ERG go with initial node features H and adjacency matrix A Implement message passing strategy using Eq.6 to obtain label semantic embeddings E
[Implement label semantics learning shown Section 3.4.}
Compute importance vectors using fully- connected network followed by sigmoid function. Compute semantic-specific features o with deep latent feature and importance vectors a with Eq.8. Compute the label confidence scores s using Eq.9 to obtain the prediction for emotion classes 1,... Co.
[Implement semantic-gun guided feature decoupling to obtain semantic speccie features shown Section 3.5.}
Compute the representation similarity matrix M M and M with Eq.10 10 and 11.
[Implement relation-based knowledge distillation with affective dimension features shonw Section 3.6.)
Compute the final loss L with Eq.7, 12 and 13. Update AESL model by minimizing L. end for end while end for
B. Details of Experiments
B.1. Details of Datasets
Brain27 is visually evoked emotional brain activity dataset (Horikawa et al., 2020), which contains the blood-oxygen-level dependent (BOLD) responses of five subjects, who were shown 2196 video clips while functional Magnetic Resonance Imaging (fMRI) data were recorded. These data were collected using a 3T Siemens scanner with multiband gradient Echo-Planar Imaging (EPI) sequence (TR, 2000ms; TE, 43ms; flip angle, 80 deg; FOV, 192x 192 mm; voxel size, x im; number of slices, 76; multiband factor, 4). The fMRI data was preprocessed and averaged with each video stimulus, which means the brain activity of one voxel a scalar for a video stimulus.
Video27 is an emotionally evocative visual dataset (Cowen Keltner, 2017), which had been used to collect the brain activity in Brain27. This dataset contains 2196 videos whose durations ranged from 0.15s to 90s. Some video screenshots of Video27 have been shown in Figure 1.
Audio28 is an emotionally evocative auditory dataset (Cowen et al., 2020), which consists of 1841 music samples without lyrics. these music clips, 1572 were selected from YouTube, 88 came from Howard Shore's s Lord of the Rings soundtrack, and 181 came from Wagner's ' Ring cycle. These segments of music can convey strong feelings, whose durations ranged from 0.73s 7.89s.
13

---

EmoGrowth: Incremental Multi-label Emotion Decoding with Augmented Emotional Relation Graph
In terms of emotion category and affective dimension ratings, in Brain27 and Video27, each instance was voted by multiple raters across 27 emotion categories and 14 affective dimensions. In Audio28, each music clip was judged by multiple rates across 28 emotion categories and 11 affective dimensions. Emotion category ratings range from 0 to I and we set threshold 0.1 for Brain27 and Video27 and 0.15 for Audio28 to construct emotion label matrix. The average number of emotion labels for the former two datasets and Audio28 is 4.64 and 5.27, respectively. Affective dimension ratings were rated by a 9-scale Likert scale, which are standardized before RKD in our experiments.
In the process of splitting emotion labels for incremental learning, we just follow the order of the alphabet without other interfere. The order of Brain27 and Video27 is Admiration, Adoration, Aesthetic appreciation, Amusement, Anger, Anxiety, Awe, Awkwardness, Boredom, Calmness, Confusion, Craving, Horror, Interest, Joy, Nostalgia, Relief. Romance, Sadness, Satisfaction, Sexual desire and Surprise. The order of Audio28 is Amusing, Angry, Annoying, Anxious, Amazing. Beautiful. Bittersweet, Calm, Compassionate, Dreamy, Eerie, Energizing, Entrancing, Erotic, Euphoric, Exciting, Goose bumps, Indignant, Joyful, Nauseating, Painful, Proud, Romantic, Sad, Scary, Tender, Transcendent and Triumphant. Definitions of the 27 and 28 emotion categories are detailed in Table 6.
pain, EntrancementExxcitemenn,, Fear,
Affective dimensions used in Brain27 and Video27 are Approach, Arousal, Attention. Certainty, Commitment, Control, Dominance, Effort, Fairness, Identity, Obstruction, Safety, Upswing and Valence. Affective dimensions used in Audio28 are Arousal, Attention, Certainty, Commitment, Dominance, Enjoyment, Familiarity, Identity, Obstruction, Safery and Valence.
Table 7 shows the characteristics of the three datasets used in our experiments. Properties of each dataset are characterized by several statistics, including the number of training instances [Dtel. the number of test instances (Drel, the number of features Dim(D), the threshold for constructing label matrix (DD). the number of possible class labels L(D), the number of affective dimensions (D)) the label cardinality (average number of labels per instance) I.Card(D). the label density (label cardinality over L(D)) LDen(D), and the modality. For Brain27, we also exhibit the number of voxels VDD) before RO1-pooling used in our experiments.
B.2. Comparing Approaches
Details of these compared methods are as follows.
EWC (Kirkpatrick et al., 2017): A Single- Class Incremental Learning algorithm that reduces catastrophic forgetting by constraining important parameters uses the Fisher information matrix to compute the importance of parameters.
LWF (Lee et al., 2019): The first algorithm to apply knowledge distillation to the Single-Label Class Incremental Learning task uses the old model as a teacher and minimizes the KL. divergence between the probability distributions of the outputs of the new and old models.
ER (Lee al., 2019): A Single-Label Class Incremental Learning algorithm based on data replay, where the construction of data buffer employs a random sampling strategy.
RS (Vitter, 1985): A Single-Label Class Incremental Learning algorithm based on data replay, where the construction of a data buffer utilizes a reservoir sampling strategy.
AGCN (Vitter, 1985): A Multi-Label Class Incremental Learning algorithm based on graph convolutional neural networks, where the graph adjacency matrix continuously expands as the tasks progress.
PRS (Kim et al., 2020): A Multi-Label Class Incremental Learning algorithm based on data replay, which improves upon the reservoir sampling strategy to ensure that the number of samples for each class in the data buffer is as balanced as possible.
OCDM (Liang& & Li, 2022): A Multi-Label Class Incremental Learning algorithm based on data replay that defines the construction and updating of the data buffer as an optimization problem to be solved.
KRT-R (Dong et al., 2023): A Multi-Label Class Incremental Learning algorithm based on Knowledge Restore and Transfer (KRT) framework.
B.3. Feature Extraction
In Brain27 dataset, we extract a 2880-dimensional feature vector with ROI-pooling (see Figure 8). In Video27, visual object features have been extracted with a pre-trained VGG19 model (Simonyan & Zisserman, 2014) for one frame and averaged across all frames to construct 1000-dimensional features. In Audio28 dataset, we compute Mel- Hel-frequency cepstral
14

---

EmoGrowth: Incremental Multi-label Emotion Decoding with Augmented Emotional Relation Graph
Table 6. Definitions of emotion categories for Brain27, Video28, and Audio28.
Name
Definition
Admiration
A feeling deep respect and uppreciation for someone's qualities or achievements.
Adoration
A profound sense of love, devotion, reverence for someone or something cherished.
Aesthetic appreciation
The pleasure and admiration felt when encountering beauty or artistic excellence.
Amusement
A lighthearted and joyful response something funny entertaining
Anger
A strong emotional reaction to perceived harm, injustice, or frustration. tense, uncasy foeling often associated with fear worry about uncertain outcomes. A profound emotional response to something vast. grand, or beyond ordinary comprchension.
Anxiety
Awe
Awkwardness
sense of discomfort embarrassmet in socially clumsy or uncertain situations.
Boredom
A state of weariness or dissatisfaction caused by a lack of interest or engagement.
Calmness
A screne, peaceful state of mind free from stress or agitation.
27 emotton categories
Confusion
A feeling of bewilderment uncertainty when faced with something unclear unexpected.
Craving
An intense desire or longing for something specific, often food or experiences.
Disgust
A strong foeling of avension revulsion, often triggered something offensive unpleasant.
Fimpathc pain
An emotional resonance with another's suffering, leading to shared feelings of distress. captivating. hypootic feeling that draws one into deeply absorbing experience.
Entrancement
Excitement
A heightened state of anticipation or enthusiasm ahout something exhilarating or enjoyahle. powerful emotion response to perceived danger or threat, prompting light flight reaction.
Horror
An intense fear mixed with shock revulsion, often caused by something terifying gruesome.
Interest
A sense of curiosity and attention sparked by something engaging or mouughtprroooing
Joy
A profound and upliftigg feeling of happiness or delight.
Nostalgia
A hittersweet longing for past experiences, often accompanied by fond memories.
Relief
A lightened and eased feeling after the alleviation of pain, worry.
Romance
A tender and affectionate emotion linked to love and intimate connection. heavy. sorrowful foeling typically associated with loss, disappointment, empathy.
Sadness
Satisfaction
A contented sense of fulfillment after achieving goal or desire.
Sexual desire
deep. physical and emotional yearning intimacy and connection.
Surprise
A sudden and unexpected emotion elicited by an unforeseen event realization.
Amusing
This emotion brings a light-hearted, fun feeling, often sparking smiles and laughter.
Angry
A powerful emotion, usually triggered by frustration injustice, that drives strong sense displeasure.
Annoying
A mild irritation or frustration caused by something persistently bothersome or inconvenient.
Anxious
feeling worry, nervousness, unease about an uncertain outcome or event.
Amazing
A sense of wonder and admiration often evoked hy something remarkable or extraordinary.
Beautiful
An emotion tied to the appreciation of visual, auditory, or conceptual harmony and appeal.
Bittersweet
A mixed emotion of happiness and sadness, usually arising from nostalgia or a cherished memory.
Calm
A soothing, peaceful feeling of tranquility and lack of disturbance.
Compassionate
A warm, emputhetic response involves caring deeply for
else's suffering.
Dreamy
A relaxed, whimsical feeling, often associated with sense of escapism or fantasy. feeling mystery tinged with uncase, often evoked by something strange or uncanny.
Eerie
28 emotion categories
Energizing
A rush of motivation and vitality that makes one feel alert and ready for action. mesmerizing, absorbing
Entrancing
captivates one's attention.
Frotic
A deeply intimate, sensual feeling characterized by physical and emotional desire.
Euphoric
An intense, exhilaruting joy that feels almost trunscendent or surreal.
Exciting
A feeling of lively anticipation and enthusiasm about something anticipoted unfolding.
Goose bumps
A physical reaction to intense emotions, often linked to awe, fear, or admiration.
Indignant
A righteous anger resentment, typically provoked by perceived unfair treatment.
Joyful
A pure, light-hearted happiness that uplifts and brightens one's 's mood.
Nauseating
A strong feeling of physical discomfort, often coupled with disgust or revulsion.
Painful
An intense, often distressing sensation caused by physical or emotional suffering.
Proud
positive feeling satisfaction and fulfillment, often in recognition of achievement.
Romantic
A tender, affectionute feeling centered around love and connection.
Sad
A heavy, sorrowful feeling often caused by loss, disappointment, or emputhy.
Scary
A strong. unsettling sense of fear triggered by perceived threat danger.
Tender
A geatle, warm-hearted feeling of affection and care.
Transcendent
An emotion that goes beyond ordinary experience, often bringing of enlightenment.
Triumphant
A victorious, celebratory emotion after overcoming a challenge or achieving success.
15

---

EmoGrowth: Incremental Multi-label Emotion Decoding with Augmented Emotional Relation Graph
Table 7. The characteristics of the experimental datasets.
Dataset
[Dr+]
[Dee
V(D)
Dim((D
Th(D)
L(D)
Aff(D)
LCard(D)
LDen(D)
Modality
Bruje27(Subject))
1800
396
120930
2880
0.10
27
14
4.64
0.17
IMRI
Brain27 Brain27(Suhject2)
1800
396
116260
2880
0.10
27
14
4.64
0.17
IMRI
Brain27(Suhject))
1800
396
102941
2880
0.10
14
4.64
0.17
fMRI
BSubn27(Subject4)
1800
396
118533
2880
0.10
14
4.64
0.17
IMRI
(Sbbe27(S
1800
396
116699
2880
0.10
27
14
4.64
0.17
IMRI
Video27
1800
396
1000
0.10
27
11
4.64
0.17
Video
Audio28
1500
341
512
0.15
28
11
5.27
0.19
Audio
ROI
Figure schematic diagram ROI pooling. Each orange small cube represents voxel. Using the brain voxels signal directly for voxel-wise decoding will introduce lots of noise and easily cause overfitting Therefore, first use the HCP360 template (Glasser al., 2016) to divide the whole brain into multiple brain regions (ROIs) that include cortical regions defined by a parcellation provided from the Human Connectome Project. In order further extract the features of each ROI, place the voxels of each ROI in 3-D volume according to its coordinates, then split the volume evenly into sub-volumes and calculate the average brain activity of voxels in each sub-volume the feature of this sub-volume illustrated in Figure 8. In other words, for each ROI get 8-dimensional features. Then we concatenate the features of 360 ROIs in each hemisphere to get 2880-dimensional features.
coefficients (MFCC) for each audio fragment. All MFCC fragments from the same audio are then input into pre-trained ResNet-18 (He et al., 2016) model and averaged across all fragments to obtain 512-dimensional features.
B.4. Hyperparameters Settings
In our experiments, the balancing parameter 3 is set to 0.95 in Eq.3. We set X to in Eg. Besides, X2 is searched in {0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8} and À3 is searched in {0.001, 0.01, 0.1, 1, 5, 10). The dimensionality of deep latent representations z is set to 64 in three datasets. We train the model using the Adam optimizer with (51, }2} = {0.9, 0.9999}. set weight decay to 0.005 and learning rate to -4 for Brain27 and Video27, and weight decay and learning rate to 10-3 for Audio28. We conducted all the experiments on one NVIDIA TITAN GPU.
C. More Comparative Results
Tables 8, 9, 10 and ll show the results on subject 2, subject 3, subject4 and subject 5 in Brain27 dataset. Figure 9 exhibits the comparison curves of AESL and comparing methods for these subjects in Brain27 dataset. We observe that similar conclusions can be drawn as mentioned Section 4.2.
D. Limitations
At the application level, in the affective HCI tasks in real-life scenarios, in addition learning emotion categories, also need to adapt to new subjects. Therefore, it is necessary to further consider the continuous learning of emotions from different subjects, which can be regarded as domain incremental learning. At the experimental level, the impact of the order of learning emotion categories and the number of emotion categories in each task on the experimental results needs to be further explored.
16

---

EmoGrowth: Incremental Multi-label Emotion Decoding with Augmented Emotional Relation Graph
Table Class incremental results on subject 2 Brain27 dataset. AGCN, PRS and OCDM are MLCIL algorithms among these compared
methods.
Brain27 BO-19
Brain27 B0-13
Brain27 B15-13
Brain27 B15-12
Method
Avg. Acc
Last Ace
Avg. Acc
Last Acc
Avg. Acc
Last Acc
Avg. Ace
Last Ace
mAP
maF1
miF1
mAP
mAP
maF1
miFI
mAP
mAP
maF1
miFI
mAP
mAP
maF1
miFI
mAP
Upper-bound
34.0
44.3
42.3
34.0
44.3
42.3
34.0
44.3
42.3
34.0
44.3
42.3
Finetune
24.8
21.0
20.0
18.7
EWC
34.3
18.1
25.2
31.2
13.7
21.7
26.4
13.8
20.1
19.4
LwF
38.9
11.4
28.6
29.6
37.0
19.4
37.0
25.9
31.8
15.7
32.5
24.4
29.4
15.4
32.6
21.9
ER
40.4
8.5
13.1
34.9
39.3
3.8
4.8
20.8
37.3
6.9
11.6
34.1
36.3
7.4
11.0
32.7
RS
41.0
7.6
10.8
35.9
40.4
3.7
5,9
28.9
36.4
6.2
10.3
31.4
35.8
4.3
7.8
31.0
AGCN
43.4
28.8
43.1
39.8
43.0
32.4
43.6
33.7
39.0
24.8
39.5
35.2
37.3
24.0
35.5
32.1
40.4
34.4
20.2
37.8
12.0
34.1
37.2
11.8
32.5
OCDM
12.9
33.4
8.1
28.2
36.1
30.9
34.7
29.4
AESL
45.4
29.8
43.2
41.4
45.5
32.5
44.2
35.3
40.2
25.5
40.1
36.4
39.3
24.8
36.4
35.0
Table 9. Class incremental results on subject 3 of Brain27 dataset. AGCN, PRS and OCDM are MLCIL algorithms among these compared
methods.
Brain27 BO-19
Brain27
B0-13
Brain27 B15-13
Brain27 B15-12
Method
Avg. Acc mAP
Last
Avg.
Last
Avg.
Last
Avg.
Last Ace
maFl
miF1
mAP
mAP
maFi
miF1
mAP
mAP
maFl
miFl
mAP
mAP
maFl
miFl
mAP
Upper-bound
33.6
44,2
42.0
33.6
44.2
42.0
33.6
44.2
42.0
33.6
44.2
42.0
Finetune
33.0
7.7
17.3
25.8
29.4
5.0
14.0
20.5
25.0
5.3
14.0
20.0
23.2
4.1
13.8
19.2
EWC
32.4
7.7
16.7
25.3
29.6
5.2
13.8
19.8
25.5
5.3
14.0
20,0
24.0
4.0
13.6
19.6
LWF
36.4
28.7
30.3
34.7
21.7
38.3
23.5
30.3
15.4
31.5
23.6
28.2
15.6
31.8
21.3
ER
37.1
33.5
36.7
30.5
34.6
31.9
35.4
10.9
32.9
35.8
29.0
30.3
33.8
30.0
AGCN
40.2
27.6
42.0
38.2
40.1
32.0
42.9
32.7
37.0
27.0
37.1
33.6
34.9
24.1
36.6
30.2
PRS
38.0
7.6
12.0
34.3
38.8
3.6
6.0
29.5
35.5
5.5
7.7
32.7
36.0
4.5
7.3
33.1
OCDM
37.9
8.4
11.7
35.0
38.0
5.4
8.2
30.1
34.4
4.0
6.0
30.3
33.2
3.1
5.3
28.5
AESL
42.6
28.5
44,0
40.8
42.3
33,4
43.2
33.4
38.8
27.2
38,9
35.4
37.9
25.5
37.7
34.1
Table 10. Class incremental results on subject Brain27 dataset. AGCN, PRS and OCDM are MLCIL algorithms among these
compared methods.
Brain27 BO-19
Brain27 B0-13
Brain27 B15-13
Brain27 B15-12
Method
Avg. Acc
Last
Avg.
Last
Avg.
Last
Avg.
Last
mAP
maFl
miFl
mAP
maFi
miFl
mAP
maFi
miFl
mAP
mAP
maFl
miF]
mAP
Upper-bound
38.3
48.6
45.1
38.3
48.6
45.1
38.3
48.6
45.1
38.3
48.6
45.1
Finetune
36.0
7.9
19.0
25.2
30.9
4.9
13.7
21.1
26.4
4.6
13.4
20.3
23.9
3.7
13.2
18.1
EWC
35.0
7.9
18.5
25.3
31.7
5.1
13.8
22.0
27.2
4.8
13.7
20.4
25.1
3.7
13.2
20.1
LWF
39.6
33.9
30.8
37.5
20.5
37.5
23.1
33.3
17.2
32.8
25.2
30.9
15.8
33.5
22.7
37.7
34.8
37.0
36.4
RS
42.6
14.2
38.1
7.1
34.4
38.8
10.8
34.6
37.8
33.2
AGCN
43.7
31.8
46.4
40.9
44.1
36.1
44,5
36.3
41.0
30.3
43.7
38.6
39.3
26.6
37.1
35.0
PRS
43.5
10.7
15.9
39.9
44.2
4.0
5.6
32.8
40.2
8.5
11.1
36.1
39.3
9.1
12.2
35.6
OCDM
43.4
11.4
15.1
39.8
43.0
4.9
7.8
31.7
38.2
6.0
5.9
33.4
38.4
5.2
9.4
33.4
AESL
45.7
36.4
47.3
44,0
47.1
37.2
46.4
38.7
43.4
31.3
44.4
39.9
42.2
29.0
40.4
37.7
17

---

EmoGrowth: Incremental Multi-label Emotion Decoding with Augmented Emotional Relation Graph
Table 11. Class incremental results on subject of Brain27 dataset. AGCN, PRS, OCDM MI.CII. algorithm in these compared
methods.
Brain27 B0-19
Brain27 B0-13
Brain27 B15-13
Brain27 B15-12
Method
Avg. Acc mAP
Last Acc
Avg. Acc
Last Acc
Avg. Ace
Last Ace
Avg. Ace
Last Ace
maFl
miF1
mAP
mAP
mali
miF1
mAP
mAP
mal']
miF1
mAP
mAP
maFi
miFi
mAP
Upper-hound
36.6
46.8
45.0
36.6
46.8
45.0
36.6
46.8
45.0
36.6
46.8
45.0
Finetune
34.7
17.8
25.4
21.7
25.4
5.2
18.9
23.4
13.2
19.2
EWC
34.1
15.6
25.1
31.0
21.9
26.3
13.7
20.3
24.3
20.2
LWF
37.6
12.2
30.3
28.7
35.5
21.0
37.9
25.8
31.2
15.8
31.7
24.6
29.4
14.3
29.0
22.4
ER
40.7
8.8
13.9
35.8
41.7
4.5
7.6
34.0
38.1
6.9
8.9
36.4
36,3
6.9
9.8
32.9
RS
41.4
7.6
11.3
36.9
41.5
4.4
7.6
31.7
37.1
5.7
8.6
33.4
37.3
5.4
9.0
33.0
AGCN
43.8
29.5
44,0
41.4
42.9
32.5
42.5
34.1
38.4
28.7
41.5
35.0
37.3
24.6
37.6
33.1
PRS
41.2
8.9
16.4
37.0
42.4
5.3
8.3
32.0
37.8
34.1
38.4
6.5
34,9
OCDM
41.1
14.0
35.5
41.3
29.7
36.8
7.9
31.5
34.8
4.6
30.1
AESL
46.0
32,9
45.8
44.1
45.5
35.3
45.6
35.4
40.4
29.5
44.5
37.5
39.9
29.3
40.8
36.2
Finetune
BWC
LwP
BR
RS
AGCN
PRS
OCDM
AESL
B0-19
B0-13
B15-11
R15-32
40-
42
20
32
25
21 Number classes
Number classes
Number
Number
(a) Brain27(2)
RO-1
45-
mAP
-5
25
IF
27
12 15
1
34
27
21 Numbver of classos
19
25
27
Number of classes
Number classses
Namber of classes
(b) Brain27(3)
B0-19
R0-13
B15-1
B15-12
4.5
40
mAP
51-
15
23
32
Number of classos
Number classss
Number classes
Number of classes
Brain27(4)
B0.19
B0-13
B15-13
1315-12
45
32
21
classes
classos
Number
(d) Brain27(5)
Figure 9. Comparison results (mAP) on three datasets used in our experiment under different protocols against compared CIL methods.
18


Reviews:
[{"review_id": "8b33w62dfD", "reviewer": "Reviewer_mBeK", "summary": "The paper proposes **EmoGrowth**, a framework addressing **multi-label fine-grained class incremental emotion decoding**. This paradigm enables models to learn **new emotion categories incrementally** while preserving the ability to recognize **multiple concurrent emotions** in dynamic real-world environments.  The **Augmented Emotional Semantics Learning (AESL)** framework introduces three innovations:  \n- **Emotional Relation Graph (ERG)** for label disambiguation and capturing inter-class dependencies.  \n- **Affective dimension-based knowledge distillation** to align features with continuous emotion representations.  \n- **Semantic-specific feature decoupling** guided by emotion embeddings.  \n\nExperiments on **brain activity (Brain27)**, **video (Video27)**, and **audio (Audio28)** datasets demonstrate superior performance in decoding **28 fine-grained emotions**, outperforming existing methods while mitigating catastrophic forgetting.  \n\n---\n\n### **Key Contributions**  \n1. **Technical Innovation**:  \n   - **Augmented ERG**: Dynamically integrates historical and new task emotional relationships using graph-based label disambiguation. This resolves the *past-missing label problem* by generating reliable soft labels and preserving label correlations across tasks.  \n   - **Relation-based Knowledge Distillation (RKD)**: Aligns model features with **affective dimension spaces** (e.g., valence-arousal), leveraging domain knowledge to address the *future-missing label problem*.  \n   - **Emotional Semantics Learning**: A graph autoencoder learns emotion embeddings for semantic-guided feature decoupling, enhancing multi-label recognition.  \n\n2. **Empirical Validation**:  \n   - Evaluated under multiple incremental protocols (e.g., B0-I7, B16-I3), AESL achieves **state-of-the-art results**, surpassing SLCIL methods (e.g., LwF, ER) and MLCIL baselines (e.g., AGCN, OCDM) by large margins (e.g., **15–20% higher mAP** on Audio28).  \n   - Visualization (t-SNE, ERG adjacency matrices) validates emotion embeddings’ semantic topology and label relationship reconstruction.  \n\n---", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1.Conduct ablation studies on emotion order and task size variability.**\n\nFor the emotion order, we conducted 10 randomized shuffles of the category sequence. The experimental results are presented below:\n|Method | Brain27 B0-I9 | Brain27 B0-I3 | Brain27 B15-I3| Brain27 B15-I2 |\n|-----|-------------------|------------------|---------------------|--------------------|\n|AESL|44.8 $\\pm$ 0.3|43.9 $\\pm$ 0.5|41.7 $\\pm$ 0.5|39.9 $\\pm$ 0.5 |\n\n|Method | Video27 B0-I9 | Video27 B0-I3 | Video27 B15-I3| Video27 B15-I2 |\n|-----|-------------------|------------------|---------------------|--------------------|\n|AESL|45.1 $\\pm$ 0.5|47.3 $\\pm$ 0.4|41.6 $\\pm$ 0.3|39.5 $\\pm$ 0.7|\n\n|Method | Audio28 B0-I7 | Audio28 B0-I4 | Audio26 B16-I3| Audio28 B16-I2 |\n|-----|-------------------|------------------|---------------------|--------------------|\n|AESL|49.4 $\\pm$ 0.6|48.5 $\\pm$ 0.3|47.9 $\\pm$ 0.3|45.2 $\\pm$ 0.2   |\n\nFor the task size variability, we have conducted experiments for different emotion categories per task. The experimental results are presented below:\n\n|Method | Brain27 B0-I9 | Brain27 B0-I7 | Brain27 B0-I5| Brain27 B0-I3 |Brain27 B0-I1|\n|-----|-------------------|------------------|---------------------|--------------------|-----|\n|AESL|44.8 $\\pm$ 0.3|44.6 $\\pm$ 0.4|43.9 $\\pm$ 0.3|43.9 $\\pm$ 0.5    |43.0$\\pm$ 0.9|\n\n|Method | Video27 B0-I9 | Video27 B0-I7 | Video27 B0-I5| Video27 B0-I3 |Video27 B0-I1|\n|-----|-------------------|------------------|---------------------|--------------------|-----|\n|AESL|45.1 $\\pm$ 0.5|46.8 $\\pm$ 0.2|46.4 $\\pm$ 0.2|47.3 $\\pm$ 0.4  |44.8 $\\pm$ 0.5|\n\n|Method | Audio28 B0-I7 | Audio28 B0-I5 | Audio28 B0-I4| Audio28 B0-I2 |Audio28 B0-I1|\n|-----|-------------------|------------------|---------------------|--------------------|------|\n|AESL|49.4 $\\pm$ 0.6|49.0 $\\pm$ 0.2|48.5 $\\pm$ 0.3|47.4 $\\pm$ 0.7  |46.1$\\pm$ 0.9|\n\nIn most cases, as the task size decreases and the number of learned tasks increases, the model performance exhibits an overall declining trend.\n\n**2.Explore curriculum learning strategies to optimize task sequences.**\n\nWe appreciate the valuable suggestion regarding curriculum learning for task sequencing. However, in practical deployment scenarios, the emotion categories appearing at each incremental stage are typically unforeseen in advance. This inherent uncertainty makes our current approach of randomized task shuffling with repeated evaluations (as shown above) a methodologically sound solution, ensuring robustness to arbitrary category arrival orders. \n\n**3.Test cross-domain incremental learning (e.g., Brain27 → Audio28).**\n\nWe have conducted cross-domain incremental learning experiments from Brain27 to Audio28. The results on the transferred Audio28 dataset are compared with those obtained by training directly on Audio28 from scratch as follows:\n\n|Method | Audio28 B0-I7 | Audio28 B0-I4 | Audio26 B16-I3| Audio28 B16-I2 |\n|-----|-------------------|------------------|---------------------|--------------------|\n|Brain→Audio|49.6 $\\pm$ 0.1|48.4 $\\pm$ 0.2|47.4 $\\pm$ 0.5|45.5 $\\pm$ 0.3 |\n|Audio|49.4 $\\pm$ 0.6|48.5 $\\pm$ 0.3|47.9 $\\pm$ 0.3|45.2 $\\pm$ 0.2|\n\nThe experimental results demonstrate that transfer learning can enhance model performance compared to cold-start training in certain scenarios. However, due to the significant feature disparity, the cross-domain transfer may conversely degrade performance in some cases. \n\n**4.Investigate unsupervised/semi-supervised affective dimension estimation for new tasks.**\n\nIn our work, we assume the affective dimension labels of samples are manually annotated ratings. Considering the prohibitive cost of manually rating each new sample in practical applications, we explored pre-training an affective dimension scoring model with additional held-out training set. This allows us to automatically extract affective dimension features for incoming samples using the pre-trained model. Comparative experimental results between knowledge distillation using model-predicted affective dimension labels versus human-annotated labels are presented below:\n\n|Method | Brain27 B0-I9 | Brain27 B0-I3 | Brain27 B15-I3| Brain27 B15-I2 |\n|-----|-------------------|------------------|---------------------|--------------------|\n|Model-predicted|44.5 $\\pm$ 0.2|43.8 $\\pm$ 0.5|41.7 $\\pm$ 0.4|39.7 $\\pm$ 0.3  |\n|Human-annotated |44.8 $\\pm$ 0.3|43.9 $\\pm$ 0.5|41.7 $\\pm$ 0.5|39.9 $\\pm$ 0.5 |\n\nThe results demonstrate that utilizing model-predicted affective dimension labels can achieve comparable performance to using human-annotated affective dimension labels.\n\n**5.Formalize guarantees for knowledge transfer between affective spaces and emotion categories.**\n\nWe sincerely appreciate this foundational question. In our upcoming work, we will establish error bounds for cross-space knowledge transfer between emotion category manifolds and dimensional affective spaces by deriving Lipschitz continuity conditions and optimal transport-based projection guarantees.", "claims_and_evidence": "### **Weaknesses**  \n1. **Limited Exploration of Task Dynamics**:  \n   - The impact of **emotion category ordering** (e.g., learning \"Adoration\" before \"Awe\") on performance remains unexamined, which could affect real-world deployment.  \n   - **Task granularity**: The effect of varying emotion categories per task (e.g., 2 vs. 7 categories per incremental task) is not analyzed.  \n\n2. **Practical Constraints**:  \n   - While affective dimensions (valence/arousal) are utilized, their annotation process for new tasks is not discussed—raising questions about scalability.  \n   - No evaluation on **cross-subject** or **cross-domain adaptation** (e.g., transferring from brain signals to audio), limiting applicability to heterogeneous data scenarios.  \n\n3. **Theoretical Gaps**:  \n   - The interplay between emotion categories and affective dimensions is underexplored. For instance, how explicit alignment (via RKD) improves incremental learning beyond empirical results lacks theoretical justification.  \n\n---\n\n### **Suggestions**  \n1. **Expand Task Dynamics Analysis**:  \n   - Conduct ablation studies on emotion order and task size variability.  \n   - Explore **curriculum learning** strategies to optimize task sequences.  \n\n2. **Enhance Generalizability**:  \n   - Test cross-domain incremental learning (e.g., Brain27 → Audio28).  \n   - Investigate unsupervised/semi-supervised affective dimension estimation for new tasks.  \n\n3. **Deepen Theoretical Foundation**:  \n   - Formalize guarantees for knowledge transfer between affective spaces and emotion categories.", "methods_and_evaluation_criteria": "SEE Claims And Evidence", "theoretical_claims": "SEE Claims And Evidence", "experimental_designs_or_analyses": "SEE Claims And Evidence", "relation_to_broader_scientific_literature": "SEE Claims And Evidence", "essential_references_not_discussed": "NO", "other_strengths_and_weaknesses": "SEE Claims And Evidence", "other_comments_or_suggestions": "SEE Claims And Evidence", "questions_for_authors": "SEE Claims And Evidence", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oza47jNVy4", "reviewer": "Reviewer_DLjo", "summary": "The paper introduces multi-label, fine-grained class incremental emotion decoding AESL to adapt to the scenarios where novel emotion categories continuously emerge. To solve the critical past-missing partial label problem, AESL introduces an augmented Emotional Relation Graph (ERG) module, using graph-based label disambiguation to generate reliable soft labels. AESL enhances ERG by integrating historical ERG with new data, preserving emotional label correlations. Moreover, a relation-based knowledge distillation framework is proposed to align model features with the affective dimension space. The emotional semantics learning module utilizes ERG to design a graph autoencoder and learns emotion embeddings to facilitate semantic-specific feature decoupling. Comprehensive evaluations across three datasets (Brain27, Video27, and Audio28) prove the effectiveness of the proposed AESL.\n\n## update after rebuttal\nThe rebuttal addresses several concerns, including category order on the original tasks, teacher models, and CSC comparison. However, it falls short in providing a discussion/experiments on varying granularity level of tasks (2-cls sentiment vs 7-cls basic emotion vs 28-cls emotion), a motivation with evidence to apply class incremental learning for emotion analysis, and a literature review & SOTA comparison with recent works. The aforementioned lack limits the robustness of their claims. The explanation of motivation also lacks depth, especially regarding its significance and characteristics.\n\nGiven these unresolved issues, I recommend a decision leaning towards a weak reject, as the core contributions are promising but not fully substantiated.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1.The motivation of the task.**\n\nThe practical significance of multi-label incremental emotion decoding can be summarized as two folds.\n\n(1)\tIn real world Brain-Computer Interfaces, practical applications often require dynamic adaptation to new emotion categories over time. For example, in clinical BCIs for depression monitoring, initial models may focus on basic emotions (happy/sad), but later need to incorporate nuanced states (anhedonia, emotional blunting) as therapy progresses.\n\n(2)\tThe rapid advancement of psychology has led to increasingly fine-grained discoveries in emotion categories. Recent study [1] has empirically identified up to 80 distinct emotional states, far exceeding traditional coarse taxonomies. Besides, emotion categories are not static but evolve with interdisciplinary findings. For example, \"Bittersweetness\" and \"awe\" were later additions to emotion frameworks.\n\n**2.Discussion on category order.**\n\nIf the model initially learns exclusively positive emotions followed by negative emotions in subsequent tasks, the excessive inter-task categorical divergence may exacerbate catastrophic forgetting.\nWe conducted 10 randomized shuffles of the category sequence and computed the mean and standard deviation. The experimental results are presented below:\n|Method | Brain27 B0-I9 | Brain27 B0-I3 | Brain27 B15-I3| Brain27 B15-I2 |\n|-----|-------------------|------------------|---------------------|--------------------|\n|AESL|44.8 $\\pm$ 0.3|43.9 $\\pm$ 0.5|41.7 $\\pm$ 0.5|39.9 $\\pm$ 0.5|\n\n|Method | Video27 B0-I9 | Video27 B0-I3 | Video27 B15-I3| Video27 B15-I2 |\n|-----|-------------------|------------------|---------------------|--------------------|\n|AESL|45.1 $\\pm$ 0.5|47.3 $\\pm$ 0.4|41.6 $\\pm$ 0.3|39.5 $\\pm$ 0.7|\n\n|Method | Audio28 B0-I7 | Audio28 B0-I4 | Audio26 B16-I3| Audio28 B16-I2 |\n|-----|-------------------|------------------|---------------------|--------------------|\n|AESL|49.4 $\\pm$ 0.6|48.5 $\\pm$ 0.3|47.9 $\\pm$ 0.3|45.2 $\\pm$ 0.2|\n\n**3.The relationship between the two teacher models.**\n\nThe two teacher models exhibit complementary characteristics. The old-model teacher serves to mitigate the forgetting of previously learned emotion categories, while the affective-dimension teacher addresses catastrophic forgetting caused by future-missing partial label problem. Besides, relying solely on distillation from the affective dimension space would lead to suboptimal model performance due to the heterogeneity of feature representations, resulting in training difficulties.\n\n**4.Distillation with old knowledge can easily reduce the plasticity of the model and lacks the ablation of λ1.**\n\nKnowledge distillation from the old model is a widely used method to mitigate catastrophic forgetting, where the balance between plasticity and stability is primarily controlled by the regularization parameter—specifically, λ₁ in Eq(15). In the Brain27 dataset with B0-I9 setting, we fixed λ₂ = 0.5 and λ₃ = 2, and observed the impact of varying λ₁ on model performance as follows:\n\n| $\\lambda_1$ | 0.2   | 0.3   | 0.4  | 0.5  | 0.6  |  0.7  |  0.8  |\n|-----------------|---------|---------|--------|-------|--------|----------|---------|\n| **mAP (%)** | 41.4 | 41.9  |42.7 | 42.9  | 43.1 |  42.6 |  42.0  |\n\nWe can observe that an excessively small λ₁ leads the model to distill more information from the affective dimension space. Considering the heterogeneity of the feature space, this makes the model harder to train and results in lower accuracy. On the other hand, an overly large λ₁ makes the model more susceptible to the impact of future-missing partial label problem, thereby degrading its performance.\n\n**5.Some of the SOTA MLCIL methods in 2024 have not been compared, such as CSC.**\n\nWe have conducted comparative experiments with the CSC method, and the results will be incorporated into the final version of the paper. The experimental results (Avg. Acc) are as follows:\n|Method | Brain27 B0-I9 | Brain27 B0-I3 | Brain27 B15-I3| Brain27 B15-I2 |\n|-----|-------------------|------------------|---------------------|--------------------|\n|CSC|42.1 $\\pm$ 0.2|43.0 $\\pm$ 0.3|41.1 $\\pm$ 0.6|39.4 $\\pm$ 0.4|\n|AESL|44.8 $\\pm$ 0.3|43.9 $\\pm$ 0.5|41.7 $\\pm$ 0.5|39.9 $\\pm$ 0.5|\n\n|Method | Video27 B0-I9 | Video27 B0-I3 | Video27 B15-I3| Video27 B15-I2|\n|-----|-------------------|------------------|---------------------|--------------------|\n|CSC|44.1 $\\pm$ 0.7|46.8 $\\pm$ 0.2|41.6 $\\pm$ 0.2|38.4 $\\pm$ 0.6|\n|AESL|45.1 $\\pm$ 0.5|47.3 $\\pm$ 0.4|41.6 $\\pm$ 0.3|39.5 $\\pm$ 0.7|\n\n|Method | Audio28 B0-I7 | Audio28 B0-I4 | Audio26 B16-I3| Audio28 B16-I2|\n|-----|-------------------|------------------|---------------------|--------------------|\n|CSC|47.8 $\\pm$ 0.3|48.0 $\\pm$ 0.7|46.9 $\\pm$ 0.5|45.2 $\\pm$ 0.3|\n|AESL|49.4 $\\pm$ 0.6|48.5 $\\pm$ 0.3|47.9 $\\pm$ 0.3|45.2 $\\pm$ 0.2|\n\n**References:**\n\n[1] Koide-Majima N et al. Distinct dimensions of emotion in the human brain and their representation on the cortical surface. NeuroImage, 2020", "claims_and_evidence": "The motivation of the task is not fully explained. The authors illustrate the necessity of multiple emotions using quotes and references. However, the argument that “novel emotion categories continuously emerge” has not been proven. Unlike the subject classification task, which has a large number of categories, the categories of emotions seem to be relatively few and fixed, such as the 27 and 28 categories in the experimental dataset of the paper. The class increment task will appear unnecessary.", "methods_and_evaluation_criteria": "yes", "theoretical_claims": "yes", "experimental_designs_or_analyses": "Comparative experiments are insufficient. On the one hand, among the compared methods, the latest SLCIL method is only published in 2019. More importantly, some of the SOTA MLCIL methods in 2024 have not been compared, such as CSC[1].\n\n[1] Du K, Zhou Y, Lyu F, et al. Confidence self-calibration for multi-label class-incremental learning[C]//European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024: 234-252.", "relation_to_broader_scientific_literature": "no", "essential_references_not_discussed": "no", "other_strengths_and_weaknesses": "Pro\n\n-\tThe paper is almost well-written and well-organized.\n-\tThe proposed method can be applied to datasets of different modalities. \n\nCon\n\n- The motivation of the task is not fully explained. The authors illustrate the necessity of multiple emotions using quotes and references. However, the argument that “novel emotion categories continuously emerge” has not been proven. Unlike the subject classification tas,k which has a large number of categories, the categories of emotions seem to be relatively few and fixed, such as the 27 and 28 categories in the experimental dataset of the paper. The class increment task will appear unnecessary.\n- Discussion on category order. Unlike general classification tasks, the relationship between different emotion categories is very different. Therefore, different category orders in the incremental process will have an impact on performance. For example, if the old task is positive, the ability to learn negative and positive in the new task is different. Please discuss. In addition, a more reasonable performance report should be the average and standard deviation of the results under different orders.\n- Stability on relation-based knowledge distillation. This module distills two teacher models at the same time. What is the relationship between the two teacher models and whether they are complementary? Related discussions are recommended. In addition, distillation with old knowledge can easily reduce the plasticity of the model and lacks the ablation of λ1.\n- Comparative experiments are insufficient. On the one hand, among the compared methods, the latest SLCIL method is only published in 2019. More importantly, some of the SOTA MLCIL methods in 2024 have not been compared, such as CSC[1].", "other_comments_or_suggestions": "no", "questions_for_authors": "see weakness", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "u6PRn2Yz0S", "reviewer": "Reviewer_dimC", "summary": "This paper introduces multi-label fine-grained class incremental emotion decoding, which aims to develop models capable of incrementally learning new emotion categories while maintaining the ability to recognize multiple concurrent emotions. It proposes an Augmented Emotional Semantics Learning (AESL) framework to address two critical challenges: past- and future-missing partial label problems. AESL incorporates an augmented Emotional Relation Graph (ERG) for reliable soft label generation and affective dimension-based knowledge distillation for future-aware feature learning. Experiments demonstrate the effectiveness of the proposed method. The main contributions of this paper are:\n\n- It introduces the multi-label class incremental emotion decoding.\n- It develops an innovative augmented emotional semantics learning framework.\n\n## update after rebuttal\nThe authors' rebuttal addresses most of my concerns. I would like to keep my rating and weakly support the acceptance of the paper.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1.What are the unique challenges of multi-label class incremental emotion decoding compared to similar problems?**\n\n(1) Unlike traditional single-label class incremental learning, multi-label class incremental faces unique challenges in addressing catastrophic forgetting from past- and future-missing partial label problems. Specifically, while a single sample may correspond to multiple emotion categories, only a specific subset is available during each task phase. For past labels, if the model fails to reconstruct them, all previous emotion categories would be incorrectly treated as negative samples, significantly degrading model performance. For future labels, the model requires the incorporation of domain knowledge to develop certain reasoning capabilities about unknown categories.\n(2) Multi-label class-incremental emotion decoding deals with highly fine-grained emotional categories. In typical scenarios, human emotional variations are remarkably subtle, causing samples from different categories to be distributed in closer within the feature space. Consequently, decoding such fine-grained emotional categories presents significant challenges.\n\n**2.From Table 4, one can observe that the performance of \"w/o ESL&LD+AD\" is very close to AESL and better than other cases. The authors should analyze the reason for this phenomenon.**\n\n“w/o ESL&LD+AD” means that directly integrate the sample-wise affective dimension into category-wise emotion embeddings. In this setup, we aim to explore the rationale of incorporating domain knowledge on affective dimension to mitigate the issue of future missing partial label problem. In fact, the affect dimension space can represent an arbitrary number of emotion categories, so directly using affective dimension features for class-incremental emotion decoding would yield excellent results. However, in practical applications, this approach requires pre-establishing a mapping between emotion categories and affective dimensions, rather than constituting an end-to-end model.\n\n**3.How do the hyperparameter λ1 affect the final performance?**\n\nIn fact, in the loss function shown in Equation (15), the relative magnitudes of λ₁, λ₂, and λ₃ are quite important. In the Brain27 dataset with B0-I9 setting, we fixed λ₂ = 0.5 and λ₃ = 2, and observed the impact of varying λ₁ on model performance as follows:\n\n| $\\lambda_1$ | 0.2   | 0.3   | 0.4  | 0.5  | 0.6  |  0.7  |  0.8  |\n|-----------------|---------|---------|--------|-------|--------|----------|---------|\n| **mAP (%)** | 41.4 | 41.9  |42.7 | 42.9  | 43.1 |  42.6 |  42.0  |\n\nWe can observe that an excessively small λ₁ leads the model to distill more information from the affective dimension space. Considering the heterogeneity of the feature space, this makes the model harder to train and results in lower accuracy. On the other hand, an overly large λ₁ makes the model more susceptible to the impact of future-missing partial label problem, thereby degrading its performance.", "claims_and_evidence": "The claims are supported by evidence. The proposed method's superiority is evident in its consistent outperformance of existing methods.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria make sense for the studied problem. The proposed AESL is designed to handle the specific challenges of multi-label class incremental emotion decoding, and the evaluation metrics are appropriate for assessing the performance of the proposed method.", "theoretical_claims": "Theoretical statements are not involved in this paper.", "experimental_designs_or_analyses": "The experimental designs and analyses are overall reasonable. However, the parameter sensitivity analyses are missing. It is not clear how the regularization term $\\lambda_1$ affect the final performance.", "relation_to_broader_scientific_literature": "The paper's contributions are relevant to the broader scientific literature in affective computing and machine learning. It builds on existing research in class incremental learning and multi-label classification, advancing the field by addressing specific challenges in emotion decoding.", "essential_references_not_discussed": "There are no essential references that are missing from the paper.", "other_strengths_and_weaknesses": "### Strengths\n\n- The problem studied in this paper is interesting.\n\n- This paper is well written and in good sharp, which is easy to follow.\n\n- The experimental results are somehow promising.\n\n  \n\n### Weaknesses\n\n- This paper introduces a novel research problem, multi-label class incremental emotion decoding. However, the unique challenges of this problem compared to similar problems are not explicitly pointed out, and the significance of this problem also needs to be further elaborated.\n- From Table 4, one can observe that the performance of \"w/o ESL&LD+AD\" is very close to AESL and better than other cases. The authors should analyze the reason for this phenomenon.\n- It is not clear how the hyperparameter $\\lambda_1$ affects the final performance.", "other_comments_or_suggestions": "Providing more details on computational requirements and efficiency can further strengthen the paper. Moreover, discussing the limitations of the proposed method and potential future research directions can better facilitate the reader's understanding.", "questions_for_authors": "1. What are the unique challenges of multi-label class incremental emotion decoding compared to similar problems?\n2. From Table 4, one can observe that the performance of \"w/o ESL&LD+AD\" is very close to AESL and better than other cases. The authors should analyze the reason for this phenomenon.\n3. How do the hyperparameter $\\lambda_1$ affect the final performance?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fU13JdFCGk", "reviewer": "Reviewer_W2Hb", "summary": "The author proposes an Augmented Emotional Semantic Learning (AESL) framework, which incorporates an Emotion Relation Graph (ERG) to enhance emotion classification. To address the issue of missing partial labels in past data, a reliable soft label generation method is introduced. Additionally, a Relation-based Knowledge Distillation (RKD) approach is proposed to mitigate the impact of missing labels in future data. Furthermore, this work pioneers the application of Multi-Label Class-Incremental Learning (MLCIL) to real-world emotion classification tasks. The effectiveness of the proposed framework is demonstrated through various incremental learning protocols on three different datasets.\n\n## update after rebuttal\nI would like to keep my rating.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1.I would like to understand the practical significance of the Multi-Label Class Incremental Emotion Decoding field. Many existing emotion-related methods are capable of integrating different tasks into a single input for large models or other frameworks.**\n\n (1) In real world Brain-Computer Interfaces, practical applications often require dynamic adaptation to new emotion categories over time. For example:\n\n•  In clinical BCIs for depression monitoring, initial models may focus on basic emotions (e.g., happy/sad), but later need to incorporate nuanced states (e.g., anhedonia, emotional blunting) as therapy progresses.\n\n•  Consumer-grade BCIs (e.g., gaming or education) may start with coarse emotion labels but require incremental addition of context-specific labels (e.g., \"frustration during learning\" or \"flow state\").\n\n(2) Although large language models (LLMs) demonstrate strong performance in multi-task integration, many studies have shown that their continual learning capabilities remain limited and often suffer from catastrophic forgetting[1][2]. Therefore, investigating incremental learning remains crucial, particularly for specialized models in affective computing.\n\n**2.The explanations of the associated figures are somewhat vague, and there is insufficient discussion of their relevance. Is the ERG specifically constructed for this task accurate?**\n\nWe sincerely apologize for any confusion caused. To clarify:\n\n•  Figure 2(a) presents the overall model architecture\n\n•  Figure 2(b) provides detailed illustration of the AEG-D component marked in Figure 2(a)\n\n•  Figure 3 specifically shows the knowledge distillation scheme introduced to mitigate catastrophic forgetting caused by future label absence, which contributes additional loss functions for better training of the Figure 2(a) framework.\n\nWe will enhance the explanation of these relationships in the final version.\n\nFor ERG, as evidenced in Figure 5, the learned emotion embeddings maintain meaningful topological structures in the semantic space, which illustrates the accuracy of the ERG.\n\n**3.The overall content presentation and sequence of the paper lack clarity. For example, the definition of ERG should be introduced prior to Section 3.3.**\n\nWe appreciate the reviewer's suggestion and will adjust the manuscript to present ERG prior to Section 3.3 for better logical flow in the final version.\n\n**4.The compared works have not publicly conducted experiments on the three datasets used in this study. Therefore, it is unclear whether the experimental results align with their optimal configurations.**\n\nAll compared methods maintain their original model architectures as reported in their respective papers. For feature extraction, a critical component in emotion decoding, we adopted the most widely-used approaches for these three datasets, as established in prior literature[3][4]. For parameter optimization, we reserved a separate validation set and carefully selected optimal configurations for fair comparison.\n\n**References:**\n\n[1] Chen W, Zhou Y, Du N, et al. Lifelong language pretraining with distribution-specialized experts[C]//International Conference on Machine Learning. PMLR, 2023: 5383-5395.\n\n[2] Hu H, Sener O, Sha F, et al. Drinking from a firehose: Continual learning with web-scale natural language[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 45(5): 5684-5696.\n\n[3] Fu K, Du C, Wang S, et al. Multi-view multi-label fine-grained emotion decoding from human brain activity[J]. IEEE Transactions on Neural Networks and Learning Systems, 2022.\n\n[4] Horikawa T, Cowen A S, Keltner D, et al. The neural representation of visually evoked emotion is high-dimensional, categorical, and distributed across transmodal brain regions[J]. Iscience, 2020, 23(5).\n\n[5] Cowen A S, Fang X, Sauter D, et al. What music makes us feel: At least 13 dimensions organize subjective experiences associated with music across different cultures[J]. Proceedings of the National Academy of Sciences, 2020, 117(4): 1924-1934.", "claims_and_evidence": "The related work and ablation experiments provide theoretical support for the various components of this complex model and demonstrate their effectiveness in practice. However, the overall presentation lacks clarity and rigor.", "methods_and_evaluation_criteria": "Yes. The proposed method is relatively novel, and the experiments are fairly comprehensive.", "theoretical_claims": "The content description and organization of the paper are somewhat disordered. For example, the definition of ERG in Section 3.4 should be introduced before Section 3.3.", "experimental_designs_or_analyses": "The compared works have not publicly reported experiments on these three datasets. Therefore, it is unclear whether the experimental results under the customized protocol adhere to the optimal settings of the respective baseline methods.", "relation_to_broader_scientific_literature": "Research in the field of Multi-Label Class Incremental Emotion Decoding is relatively scarce, and this paper appears to be among the early studies to focus on this area.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths:\n1. The paper introduces a novel and dynamic Emotion Relation Graph (ERG) that utilizes cross-attention to capture correlations between new and old classes. A label propagation algorithm is used to iteratively refine soft labels, while a Graph Autoencoder (GAE) learns semantic embeddings of emotion labels. Furthermore, high-dimensional alignment is performed within the Valence-Arousal feature space. These combined methods effectively mitigate the problem of catastrophic forgetting.\n2. The proposed framework is thoroughly evaluated on three datasets, with extensive ablation experiments and visualization provided.\n\nWeaknesses:\n1. The framework presented is relatively complex, with significant interdependence between its modules. However, the explanations of the associated figures are somewhat vague, and there is insufficient discussion of their relevance. Is the ERG specifically constructed for this task accurate?\n2. The overall content presentation and sequence of the paper lack clarity. For example, the definition of ERG should be introduced prior to Section 3.3.\n3. The compared works have not publicly conducted experiments on the three datasets used in this study. Therefore, it is unclear whether the experimental results align with their optimal configurations.", "other_comments_or_suggestions": "No", "questions_for_authors": "1. I would like to understand the practical significance of the Multi-Label Class Incremental Emotion Decoding field. Many existing emotion-related methods are capable of integrating different tasks into a single input for large models or other frameworks.\n2. Please refer to the weaknesses above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper received one weak reject and three weak accepts. Some weaknesses include limited exploration of task dynamics, motivation needs more details, and the significance of the problem needs better detailed. Strengths include nice technical innovation, paper is well written, and experiments are well designed, the problem is interesting, and the proposed approach has good novelty. Considering this, the strengths outweigh the weaknesses.
Decision: Accept (poster)