Gaussian-Based Instance-Adaptive Intensity Modeling for Point-Supervised Facial Expression Spotting
Authors: Yicheng Deng
Hideaki Hayashi
Hajime Nagahara
Venue: ICLR 2025 Poster (2025)
Keywords: micro-expression spotting
semi-supervised learning
soft pseudo-labeling
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: An instance-adaptive Gaussian-based soft pseudo-labeling method for point-supervised facial expression spotting, which models the expression intensity distribution at the instance level.

Abstract: Point-supervised facial expression spotting (P-FES) aims to localize facial expression instances in untrimmed videos, requiring only a single timestamp label for each instance during training. To address label sparsity, hard pseudo-labeling is often employed to propagate point labels to unlabeled frames; however, this approach can lead to confusion when distinguishing between neutral and expression frames with various intensities, which can negatively impact model performance. In this paper, we propose a two-branch framework for P-FES that incorporates a Gaussian-based instance-adaptive Intensity Modeling (GIM) module for soft pseudo-labeling. GIM models the expression intensity distribution for each instance. Specifically, we detect the pseudo-apex frame around each point label, estimate the duration, and construct a Gaussian distribution for each expression instance. We then assign soft pseudo-labels to pseudo-expression frames as intensity values based on the Gaussian distribution. Additionally, we introduce an Intensity-Aware Contrastive (IAC) loss to enhance discriminative feature learning and suppress neutral noise by contrasting neutral frames with expression frames of various intensities. Extensive experiments on the SAMM-LV and CAS(ME)$^2$ datasets demonstrate the effectiveness of our proposed framework. Code is available at https://github.com/KinopioIsAllIn/GIM.

Published as a conference paper at ICLR 2025
GAUSSIAN-BASED INSTANCE-ADAPTIVE INTENSITY MODELING FOR ROINT-SUPERVISER FACIAL EXPRES- SION SPOTTING
Yicheng Deng, Hideaki Hayashi, Hajime Nagahara
Osaka University yichengeis ids www.aoskkaa..... uuaaauuuc.... ac. ip, (hayashi, nagahara] (ashara)@ids @ids. osaka-u. ac.
ABSTRACT
Point-supervised facial expression spotting (P-FES) aims to localize facial expres- sion instances in untrimmed videos, requiring only single timestamp label for each instance during training. To address label sparsity, hard pseudo-labeling is often employed to propagate point labels to unlabeled frames; however, this ap- proach can lead to confusion when distinguishing between neutral and expression frames with various intensities, which negatively impact model performance. In this paper, we propose a two-branch framework for P-FES that incorporates Gaussian-based instance-adaptive Intensity Modeling (GIM) module for soft pseudo-labeling. GIM models the expression intensity distribution for each in- stance. Specifically, we detect the pseudo-apex frame around each point label, estimate the duration, and construct a Gaussian distribution for each expression instance. We then assign soft pseudo-labels to pseudo-expression frames as in- tensity values based on the Gaussian distribution. Additionally, we introduce an Intensity-Awar -waare Contrastive (IAC) loss to enhance discriminative feature learning and suppress neutral noise by contrasting neutral frames with expression frames various intensities. Extensive experiments on the SAMM-LV and CAS(ME)² datasets demonstrate the effectiveness of our proposed framework. Code is avail- able at http::/://ggttuub.oo///i/////
INTRODUCTION
Facial expressions play an important role in conveying human emotions as a typical form of nonverbal communication. Facial expres- sions can be divided into macro-expressions (MaEs) and micro-expressioe expressions are of high intensity, and they usu- ally last between 0.5 and 4.0 seconds (Ekman)
Intensity
(MEs). Macro-
Fully
Frame
Intensity
2003a). Macro-expression analysis impor- tant in various applications such as social robots (Rawal & Stock-Homburg 2022), virtual reality (Ortmann al. (023), and so on. In contrast, micro-expressions are subtle and rapid (shorter than 0.5 seconds) (Ben et al., 2021). They are also used in many cenotion-related applications, such as lie detection (Ekman Friesen 1969)
Point
Frame
A nnotated frame
Figure 1: Different methods for annotation. The fully-supervised method requires annotating the onset, apex, and offset frames of each instance, whereas the point-supervised method requires an- notating only a single frame for each instance.
and psychological counseling (Ekman (003b) since they are spontaneous and represent real emotions. Therefore, both macro- and micro- expression analysis are significant human life.
Facial expression spotting (FES) is important task in facial expression analysis. As a preliminary step to recognizing the specific emotional types facial expressions, FES aims to localize facial expression instances in untrimmed videos, determining the onset and offset frames and classifying

---

Published as a conference paper at ICLR 2025
the expression type (i.e., MaE or ME) for each instance. FES is crucial for accurately identifying various expressions in videos, enabling more precise emotion recogmition and enhancing applications in numan-compiin interaction. Previous works (Yin al. 2023 Yu et al. [023] Deng et al., 2024a) have mainly focused on fully-supervised FES (F-HES). They usually extract optical flow features and develop deep learning models analyze the extracted features, achieving good performance. The remarkable progress of F-FES can be attributed to the use of frame-level annotations.
To incorporate the findings of these F-FES studies into more practical problems under limited annotation cost, this paper investigates point-supervised FES (P-FES). As illustrated in Figure [ in contrast F-FES, which requires annotating the onset and offset frames with low expression intensity, P-FES requires only a single timestamp annotation at any intensity for each instance. This approach can significantly reduce the annotation burden and time required for training models, making it more feasible to deploy in real-world applications.
The potential challenge in P-FES lies in the diffi- culty of detecting complete expression instances in the absence of boundary labels while also sup- pressing neutral noise, which refers to specific neutral frames that do not convey any signifi- cant emotions but can interfere with the spotting process. Even though there is a notable lack of research on P-FES, many efforts have been de- voted to point-supervised temporal action local- ization (P-TAL), which shares the same problem setting as our task, with the only difference be- ing target domains. Previous P-TAL methods (Ma et al. 2020,[Lee & Byun 2021] Zhang et al., 2024) typically employ a two-branch framework for class-agnostic score estimation and action classification. Subsequently, they mine reliable pseudo-action frames based on class-agnostic scores and feature similarity, then assign hard pseudo-labels to them. However, such hard sseudo-labeling strategy may not be directly ap- plicable to the P-FES task, it fails to help the model distinguish between neutral and expression frames with various intensities. As illustrated in Figure [] although low-intensity expression frames are similar to neutral frames in intensity, they should be assigned label of when using hard beeudo-labeling-jie just like the high-intensio frames near the apex frames. In this case, hard pscudo- labels cannot precisely describe the characteristics of expression intensity, resulting in inaccurate class-agnostic output scores.
Intensily
Label: 1 Apea
Hard labe: or 1
Soft label: 0.2
Label: 0 (
Onset
Offset
Frame
Figure 2: Motivation illustration. Due to the fact that expression frames have various intensities, is difficult to describe this characteristic by hard pseudo-labeling. We use soft pseudo-labeling to learn the intensity distribution of each instance, reducing ambiguity in distinguishing between neu- tral and expression frames with various intensities.
To solve the above-mentioned problem, we propose two-branch framework that converts the binary classification-bass class-agnostic branch into a regression- -baed expression intensity branch. In this paper, we assume that the expression intensity in each instance follows an individual smooth Gaussian distribution instead of Bernoulli distribution. Based on this assumption, we propose Baussian-basce instance-adaptive Intensity Modeling (GIM) for P-FES. Specifically, we first employ two-branch framework to estimate expression intensity scores and action scores. We then detect the pseudo-apex frame around each labeled frame and estimate the rough duration for each expression instance. Subsequently, we build Gaussian distribution for each expression instance individually. The mean of the Gaussian distribution is determined by the feature of the pseudo-apex frame, and the variance is calculated by measuring the distance between the mean and the features of other pseudo- expression frames in the duration. Finally, we assign soft pseudo-labels as the expression intensity values for supervision and optimize the expression intensity branch. In addition, we introduce an Aitensity-Awar Contrastive (IAC) loss on reliable pseudo-labeled frames from different classes, enhancing the model's ability to distinguish between neutral frames and expression frames with various intensities to suppress the influence of neutral noise and highlight expression frames.
Our contributions are as follows:
We analyze the limitations of directly applying current P-TAL frameworks to P-FES and find that hard pseudo-labeling makes distinguishing between neutral and expression frames
2

---

Published as a conference paper at ICLR 2025
with various intensities ambiguous. Thus, we propose two-branch framework consisting of a regression branch to model facial expression intensity distribution using a soft pseudo- labeling strategy to reduce this ambiguity.
We propose a Gaussian-basse based instance-adaptive adaptive intensity modeling module to model the expression intensity distribution of each expression proposal and assign soft pseudo-labels to pseudo-expression frames for supervision. GIM, not only is soft pseudo-labeling crucial for P-FES, but apex frame detection is also significant for further expression recognition.
We introduce intensity-awan contrastive learning on pseudo-labeled frames from different classes with various intensities enhance the discriminative feature learning and suppress neutral noise.
2 RELATED WORKS
2.1 FACIAL EXPRESSION SPOTTING
Previous FES methods can be grouped into traditional methods and deep-learning methods. Tra- ditional methods generally extract optical flow and analyze the pattern of cach region of interest. Yuhong (2021 used the optical flow for facial alignment climinate the influence of head movement. Zhao et al. (2022) refined feature extraction and employed a Bayesian optimization algorithm for analyzing optical flow patterns. Wang al. (2024) proposed skip-k-frame block-wiss main directional mean optical flow (Liu et al. 2015) features and analyzed the M- M-pattern of these features.
Recently, many researchers have developed deep learning- based frameworks solve the F-FES task. [Lng et al. (2022) extended BSN (Lin et al., 2018). which was originally designed for TAL, and adapted it for FES. Yin al. (2023) refined (Leng et al. 2022) approach by introducing graph convolutional networks and action unit (AU) label information. Yu et al. (2021 2023) designed two-branch framework based on A2Net (Yang al. 2020) and introduced additional attention modules for facial expression spotting. Deng et al. (2024a) proposed an SW-MRO feature and introduced SpoT--GNN to improve the classification of individual frames. They then enhanced the framework by introducing SpotFormer (Deng et al. 2024b) and explored various model architectures.
2.2 POINT-SUPERVISED TEMPORAL ACTION LOCALIZATION
Many researchers have devoted their efforts to P-TAL to mitigate the intensive labor required for frame-level labels in fully-supervised TAL. Ma et al. (2200) proposed SF-Net to mine neighboring pseudo-action frames around each labeled frame to train the classifiers. Lee & Byun (2021) proposed to search for the optimal sequence for completeness learning using point labels. Fu et al. (2022)
measured the confidence of each frame based the feature similarity and rectified the output scores to assign reliable pseudo-labels. Zhang et al.) (024) proposed a two-stage framework to propagate high-confidence cues in point annotations both snippet and instance levels. Xia et al. (2024) claimed that the most salient frame tends to appear in the central region of each instance and presented a proposal-level plug-in framework relearn the aligned confidence of proposals to refine them.
Although P-FES, the focus of this study, has many similarities with P-TAL, research specifically dedicated to P-FES extremely limited. To our knowledge, (Yu et al. 2024) is the only paper that explored P-FES, employing framework similar to general P-TAL methods. In this paper, instead of employing hard labulig,, which may increase the ambiguity in distinguishing between neutral and expression frames with various intensities, we propose a Gaussian- based soft pseudo-labeling strategy to model the expression intensity distribution for each instance.
2.3 SOFT PSEUDO-LABELING
Soft pseudo-labeling is an advanced semi-supervised learning technique generally investigated in classification tasks. Unlike hard pseudo-labeling, which assigns a single class label to unlabeled data based on the model's 's most confident prediction, soft pseudo-labeling generates soft labels representing the full distribution of class probabilities, considering uncertainty in predictions. Nassar ct al. (2023) proposed PROTOCON, which refines soft pseudo- by knowledge of neighbors in prototypical embedding space for semi-supervised image classification. Lukov al. 2022
3

---

Published as a conference paper at ICLR 2025
Branch
action
Action Score 5
F
LaCC
Mal
Neutral
SpotFormer
Mal
Intensity- ware Contrastive Learning (IAC)
Class A gnostic Expression Intensity
X
Branch
Expcession Intansity Soore a Soft Pseudb-Labels
GIM
Feature Vectors
1-D CNN Layer
MB
Mall
ME
Mall
Data Flow
Intermediate Calculation
Frame
Pseudo-A Apex Frame
Neutral
Neutra
Mall 42
Point Label
Gaussian based Instance A daptive Intensity Modeling (GIM)
Figure 3: Overview of the proposed framework. The framework initially calculates the optical flow and extracts snippet features. These features are fed into a two-branch framework to obtain action and expression intensity scores. A GIM module is employed build the Gaussian distribution for each expression instance and assign soft pseudo-labels to model the intensity distribution. An IAC module employed to build contrasts among pseudo-labeled frames with various intensities enhance feature learning and suppress neutral noise.
proposed smooth multiple high-confidence classes in the logits based on their confidence while assigning a fixed low probability to low-confidence classes to handle noisy labels in in-the-wild facial expression recognition. Recently, several methods (Liang al. 2022: Wu et al.! 2023 Shen et al. (202)) built Gaussian Mixture Models to model class-wise feature distribution for semantic segmentation. Inspired by these works, we propose construct an individual Gaussian distribution for each expression instance to assign soft pseudo-labels as direct intensity supervision signals and train a regression model learn the expression intensity distribution, rather than the distribution of class probabilities. To the best of our knowledge, are the first to investigate the application of soft pseudo-labeling for P-FES, providing novel perspective on modeling expression intensity.
METHODOLOGY
3.1
PROBLEM FORMULATION
Given untrimmed facial video V (v:) we only have single timestamp annotation for each facial expression instance, i.e., Y (p., " where p; represents the annotated frame of the i-th expression instance, N denotes the total number of ground-truth expression instances, and denotes the multi-hot vector representing the action class (i.e., MaE and ME), respectively. Our objective is to detect many expression instances as possible, localizing the boundary frames and determining the expression type for each instance.
3.2 BASELINE FRAMEWORK
The input video V is first divided into T overlapping snippets, where each snippet represents short sequence of consecutive video frames that contain the temporal context of a single frame, following (Deng al. 2024b). Then, extract and embed optical flow features into feature vectors and concatenate them along the channel dimension, resulting in F TxD, where D denotes the dimension of each snippet feature. Then,
employ SpotFormer (Deng al.) (202bb)
the feature extractor to
4

---

Published as a conference paper at ICLR 2025
similar to general P-TAL works, we input the embedded feature vectors into two-branch framework to estimate the class-agnostic expression intensity scores a E R' and the action scores S e RT where C' represents the number of expression classes (i.e., MaE and ME). Note that the FES task does not involve emotion recognition; therefore, the estimated intensity scores and action scores are independent of emotional categories.
3.3 MOTIVATION
Due to the sparsity of point labels, propagating labels from labeled to unlabeled frames is key to enhancing model training and performance. Current P-TAL methods (Lee & Byun 202 Zhang et al.) 2024) usually assign hard pseudo-labels to neighboring frames, frames with high class-agnostic scores, or frames that have high feature similarity with labeled frames. Then, the hard pscudo-labels are used to train binary classification model for the class-agnostic branch and a multiclass classification model for the action classification branch. However, we observe that the hard pseudo-labeling strategy will cause ambiguity in FES when distinguishing between neutral and expression frames with various intensities. For example, expression frames near boundary frames have low expression intensity, which makes them have higher feature similarity with neutral frames than apex frames. Therefore, it is difficult to assign hard pseudo-labels to these low-intensity expression frames, resulting in the binary alass-ggnostic branch and hard pseudo-labeling this issue, we convert the binary classification-base based class-agnostic branch into a regression based expression intensity branch and propose GIM to assign soft pseudo-labels to frames with various intensities, modeling the expression intensity distribution of each instance.
being unsuitable for P-FES. To overcome
3.4 GAUSSIAN- BASED INSTANCE-ADAPTIVE INTENSITY MODELING (GIM)
Our solution is based the assumption that the expression intensity within each expression instance follows a smooth Gaussian distribution, with the apex frame corresponding to the peak intensity, which decreases symmetrically on both sides. Neutral frames are assumed to have an intensity of O.
Figure shoows the proposed framework. Our framework consists of an expression intensity branch and an action classification branch. To model the intensity distribution for each expression instance, we build the instance-adaptive Gaussian distributions based on the intermediate feature representations X E IR 7'xx / from the expression intensity branch and the output intensity scores a.
The algorithm for constructing the instance-adaptive Gaussian distributions and assigning soft pseudo- labels is described as follows.
Step 1. Given the output intensity scores a and a point label p; with expression class c, we detect the pseudo-apex frame peee with the highest intensity score in pre-defined range I;:
apex arg max ajz JEL.
where 7, (n E Z P Sn ). and k. denotes the general duration of the e-th class expression instance (i.e., MaE or ME). The intermediate feature of the pseudo-apex frame, a: ar.pex is selected as the μa, for the i-th Gaussian distribution g
μ4 x apex
This selection strategy ensures that the pseudo-apex frame is the center of the Gaussian distribution and has the highest soft pseudo-label.
Step 2. Centered at the pseudo-apex frame v. vepex, we estimate the rough duration L, for the i-th expression instance by filtering out the neighboring expression frames whose intensity score is larger than a threshold 0:
L, ki E J a; 0}}.
where J, = (n E Z papex ke S S ui apex + k ). and ay corresponds to the intensity score of Uz- Then, expand the rough duration by coefficient á to consider unreliable low-intensity expression frames to complete the expression proposal. In each expression proposal, we measure the feature distance between each frame and the pseudo-apex frame; then, we calculate the variance o, for the Gaussian distribution. The formulation can be described as:
T
11
(4)
JEK;
5

---

Published as a conference paper at ICLR 2025
where ||
ll2 denotes the Euclidean distance, and K; {n Z
U, uppex 81, S S v/
apex
8hg }.
Step 3. Finally, we build an unnormalized Gaussian distribution 9i for the i-th expression instance: F. p4:
gi ( o,) = exp
jE K₁
(5)
20
Given the Gaussian distribution Ii. we can assign a soft pseudo-label to each pseudo-expression frame in the range of (0, 1].
3.5 INTENSITY-WWAR CONTRASTIVE LEARNING ON PSEUDO-LAABLLA FRAMES
To further suppress neutral noise, highlight expression frames, and learn inter-class differences for action classification, introduce contrastive learning (Khosla et al. 2020) to pscudo-labeled frames. Additionally, we consider the impact of intensity differences and propose an Intensity- Aware Contrastive (IAC) loss. The intuition is that the intensity differences between frames independent of the class, and we should consider these intensity differences when building contrasts on pseudo-labeled frames. Specifically, for two samples with the same pseudo-class label, we focus less on pulling them together when their intensity difference is large; for two samples with different pseudo-class labels, we focus less on pushing them apart when their intensity difference small.
To introduce contrastive learning, identifying the neutral frames is necessary since neutral labels provided. In the previous section, focused on assigning soft pseudo-labels pseudo- expression frames. Suppose we assign pseudo-expression labels to Nexp frames. We employ the top-k strategy to select pseudo-neutral frames from those not given pseudo-labels with the top-k lowest expression intensity scores. The number of pseudo-neutral frames Nneus is determined by:
Nneut min (Nexp T - Nexp).
Then, we select reliable pseudo-expression frames with pseudo-intensity labels larger than 0.5 and build intensity-awar contrasts among reliable pseudo-neutral and pseudo-expression frames. Let Z represent the set of reliable expressix and pseudo- neutral frames, and the loss function is formulated as:
Wigg exp( S//T) EneF()) Wiee exp(f, fe/T)
CIAC Σ
log
[((il
gEQ(()
wij {
[ū; àzln
if 1/4 - イ
(8)
la;
ا if U$ y
where E(1) = IVi, and Q(1) : {q E E(1) Vg 7 represents the set samples in the video who has the same pseudo-class label with the i-th sample, is the embedded feature of the i-th sample (the i-th snippet feature of F E DR pseudo-intensity label of the i-th sample, respectively.
)), E I is a scalar temperature parameter, à represents the
3.6
TRAINING AND INFERENCE
3.6.1 LOSs FUNCTION
General P-TAL methods (Lee & Byun| 2021 Zhang et al. 2024) treat the class-agnostic branch as binary-classificatio branch and employ the binary cross-entropy (BCE) loss function to optimize the branch. In this paper, since assume the expression intensity follows smooth Gaussian distribution instead a Bernoulli distribution, we treat it as a regression task and employ the mean squared error (MSE) loss optimize the expression intensity branch:
in each expression instance
LGIM
Σ(a (a: a;)²²
Nucut + N exp NET
where T represents the set of all pseudo-labeled frames, and a; and à; represent the output intensity score and the corresponding soft pseudo-label of the frame v, respectively.
Following the previous work (Hong et al. 202I), due to the sparsity of expressions in the video, we add an L.1 normalization loss on the intensity scores:
Luorm
|| |1
(10)

---

Published as a conference paper at ICLR 2025
where || 1111 is L1-norm function.
We also employ video smooth loss to encourage temporal consistency in video output by ensuring that consecutive frames have similar predictions, stabling the training process:
T-1 | -0111.
Samooth
T - Σ
(11)
For the action classification branch, we employ general cross-entropy loss. Due to the normalization loss and the tendency for most frames to have lower scores (neutral and low-intensity expression frames), the model tends to produce low expression intensity scores. Therefore, we first refine action scores by combining them with the intensity scores:
3tc Si,e 0i: i [I,...,T,,ce [I,...,c}.
(12)
where Si,e denotes the original output probability that the i-th sample belongs to the e-th class expression. Then we calculate the action classification loss encourage reliable expression frames to generate higher scores and reliable neutral frames to generate lower scores:
1 CII-1
Laction
Ti.e log 85
log(1 - Ki,e),
(13)
I++
IEI+
c=l
iEI
c
where I+ and I represent the set of reliable pseudo-expression frames and pseudo-neutral frames, and Fi.e represents the pseudo-class label the i-th sample, respectively.
Finally, the total loss function can be summarized as:
L CGIM + Laction + X Camooth A2Cnorm + A3CIAC,
(14)
where \1, λ2, and As are hyyer-parametern for balancing the losses, which are determined empirically.
3.6.2 TRAINING PIPELINE
In the early training epochs, the output expression intensity scores are not sure to represent the expression intensity, and the highest score does not definitely represent the apex frame. Therefore, we employ an easy-to-hard learning paradigm and set several warm-up training epochs to make sure that the output intensity score can represent expression intensity. Specifically, 1) in the first stage, we assign hard pseudo-labed to adjacent frames around each labeled frame p; in range [p, - Kei
kell. 2) In the second stage, build the Gaussian distribution centered at the labeled frame and assign soft pseudo-labele in pre-defined small range in range [p, ki,2, P; + k22). 3) In the third stage, we employ our proposed GIM module for soft pseudo-labeling and model training. The pseudo- apex frame and the range for soft pseudo-labeling are updated at each training epoch to enhance intensity-related feature learning.
3.6.3 INFERENCE
In the inference phase, we first obtain the expression intensity scores a and suppressed action scores S. We then generate candidate expression proposals by using multiple thresholds for a, where each proposal includes consecutive frames with intensity scores higher than a given threshold. Each proposal is represented as (s;, Cs c, PIIC), where frame, expression type, and the outer-inner-contrastiie (OIC) score (Shou et al. 2018), respectively. Specifically, is determined by applying threshold of 0.5 to the action score of the pseudo-apex frame since is the most representative frame for each expression instance. The OIC score can be calculated as follows:
and p. represent the onset frame, offset
est
OIC pt
2
Σ
+ Σ
(15)
L [=
L
1= 4
1=e+11
L = 6, - 3, +
(16)
Finally, we apply class-wise NMS (Bodla et al. 2017
Yin al. 2023)
remove redundant ones.

---

Published as a conference paper at ICLR 2025
Table Comparison with the state-of-the-an methods on SAMM-LV and CAS(ME)² in terms of F1 score. The dagger idenotes the method originally for P-TAL and reproduced to P-FES.
Methods
SAMM-LV
CAS(ME)
MaE
ME
Overall
MaE
ME
Overall
F-FES
SOFTNet (Liong et al. 2021)
0.2169
0.1520
0.1881
0.2410
0.1173
0.2022
Concat-CNN Yang al. 2021
0.3553
0.1155
0.2736
0.2505
0.0153
0.2019
LSSNet Yu et al. 2021)
0.2810
0.1310
0.2380
0.3770
0.0420
0.3250
3D-CNN (ap et
2022
0.1595
0.0466
0.1084
0.2145
0.0714
0.1675
MTSN Liong et al. 2022)
0.3459
0.0878
0.2867
0,4104
0.0808
0.3620
ABPN (eng et al. 2022)
0.3349
0.1689
0.2908
0.3357
0.1590
0.3117
AUW-GCN Yin et al. Z0233
0.4293
0.1984
0.3728
0.4235
0.1538
0.3834
SpoT-GCN (Deng al. 2024a
0.4631
0.4035
0.4454
0.4340
0.2637
0,4154
SpotFormer (Deng et al. 2024b)
0.4447
0.4281
0.4401
0.5061
0.2817
0.4841
P-FES
LACiLee & Byun DF
0.3714
0.1983
0.3223
0.3889
0.0833
0.3598
HR-Pro(Zhang al. 2024)1
0.3395
0.1667
0.2895
0.3515
0.1345
0.3261
JSP-Net(XIa- et al.
2024
0.3152
0.1567
0.2703
0.3781
0.0571
0.3358
Ours
0.4189
0.2033
0.3587
0.4395
0.0588
0.4000
Table 2: Ablation study on loss functions.
Loss functions
SAMM-LV
CAS(ME) 2
(MIM(MSE)
(BIM(BCE)
LICC
Others
MaF
ME
Overall
MaR
ME
Overall
0.3994
0.2189
0.3477
0.4151
0.0556
0.3785
0.2892
0.1095
0.2375
0.3246
0.0282
0.2877
0.4050
0.1519
0.3360
0.4207
0.0870
0.3872
0.4189
0.2033
0.3587
0.4395
0.0588
0.4000
4 EXPERIMENTS
4.1 EXPERIMENTAL SETTINGS
Datasets. We follow the protocol of MEGC2021 and validate our method on two datasets: SAMM- LV (Yap al. 2020) and 2AS(ME) Qu et al. 2017). The SAMM-LV dataset has 147 annotated videos from 32 subjects with 200 fps, including 343 MaE clips and 159 ME clips. The CAS(ME)² dataset has 98 annotated videos from 22 subjects with 300 MaE clips and 57 ME clips, and the frame rate is 30 fps. Since the frame rates of both datasets are different, we downsample the frame rate of SAMM-LV seven times to align the frame rates.
We employ leave-one cetvee-onesubplee mue-subeeettout cross- validaaaii strategy in the experiments. An expression proposal is considered true positive (TP) if the Intersection over Union (IoU) between the expression proposal and ground-truth expression instance satisfies:
Evaluation metrics.
W Proposal O
WGronndTruth
> 0IoU,
(17)
W Proposal U GroundTruth
where Giou is the IoU threshold, set to 0.5. We calculate the Fl score to evaluate the performance of our model and compare with other methods.
Training details. We generate single-frame annotations using a Gaussian distribution centered on the ground-truth apex frame for each instance. The model trained by the Adam optimizer (Kingma
2015) on both datasets for 100 epochs with learning rate of 2.0 x 10-5 and weight decay of O.T. The coefficient for duration estimation is set to 1.2. For the multi-stage training, the epochs for each stage are 1, 4, and 95, respectively. k. is set to 16 for ME and 32 for MaE, respectively. K. is set to for MEs and 5 for MaEs, k22 is set to for MES and 4 for MaEs. We set the loss weight X, to 0.1, 0.3, and 2.0 x 1 for SAMM-LV, and to 0.1, 2.5. and 1.4 10 for CAS(ME)2. respectively. The threshold for estimating the rough duration of each expression proposal decreases linearly from 0.8 0.5 over 30 epochs and then remains 0.5 until the end.
8

---

Published as a conference paper at ICLR 2025
Table 3: Ablation study on pseudo-labeling strategies.
SAMM-LV
CAS(ME)²
Strategy
MaïE
ME
Overall
MaE
ME
Overall
Hard
0.3994
0.2189
0.3477
0.4151
0.0556
0.3785
Soft
0.1496
0.1036
0.1335
0.2705
0.0286
0.2380
Class-wisc
0.2551
0.1140
0,2049
0.3072
0.0000
0.2598
Ours
0.4189
0.2033
0.3587
0.4395
0.0588
0.4000
Ground-Truth Boundary Frame
Label
Ground- Truth Apex Frame
Plaaado Label
S S
Parado Label
Pasado Label
Prsudo Label
12 &
8 &
Frume
Frume
Frame
Frame
(a) ME #1
(b) ME
[c) McE
(d) MaE 42
Figure 4: Pseudo-label results of four expression instances. The line graph with blue dots represents soft pseudo-labels assigned by our model; the leftmost and rightmost blue dots indicate the estimated expression duration for pseudo-labeling, while the peak dot indicates the pseudo-apg apex frame
4.2 COMPARISON WITH STATE-OF-THE-ARE METHODS
We first compare the performance with state-of-the-art (SOTA) deep learning methods, and the results shown in Table Since there is prior P-FES method, reproduce and apply several SOTA P-TAL methods to the P-FES task. Note that for a fair comparison, we use the same feature extractor in method when reproducing these SOTA P-TAL methods, which significantly improve the performance of P-FES. It can be seen that our method outperforms SOTA point-supervised methods by 11.3% SAMM-LV and 11.2% CAS(ME)² Note that the separate Fl scores for MaE and ME spotting in Table represent their performance when we achieved the optimal overall performance, rather than their individual best performances. This partially explains why ME spotting performance on CAS(ME)² is lower than that of other SOTA methods. We also show the results of several F-FES methods for comparison, and the results demonstrate that our method achieves competitive performance in MaE spotting but relatively low ME spotring performance. The reason that our method focuses on significantly suppressing neutral noise, which may overshadow extremely subtle micro-expressions without the help of precise frame-level annotations.
4.3
ABLATION STUDY
4.3.1 LOSS FUNCTION
We conduct ablation studies loss functions verify the effectiveness of our proposed modules. The results are shown in Table By comparing the results of using the MSE loss and the BCE loss, we can see that treating the expression intensity branch as regression task performs better than treating it as binary classification task. This is because when our GIM generates soft labels for low-intensity expression frames, and these labels accurately describe the intensity, using the MSE loss allows them to be treated as direct intensity supervision without causing large loss value. However, we use the BCE loss, even when the soft pseudo-labels are accurate enough, the loss value could still be large, severely affecting model training. The first row of Table shows that when we only use the proposed GIM to estimate the duration of each expression instance and assign hard pseudo-labels. the performance is lower. The results demonstrate that proposed GIM module describe the intensity of each expression frame and improve the performance. In addition, the third row and the fourth in Table also demonstrate the effectiveness of our proposed IAC loss.
accurately

---

Published as a conference paper at ICLR 2025
Ground-Trud ME
Ground-Tnun MaE
Intensity Score
0.8
0.6
0.4
nnd inmurren ww
Am
0.2
Frame
Figure 5: Expression intensity results an entire video.
4.3.2 PSEUDO-LABELING STRATEGY
To verify the effectiveness of our proposed GIM, we conduct ablation studies various pseudo- labeling strategies. In Table Hard denotes our implemented hard pseudo-labeling method, which is the same the first row in Table Soft refers to using cosine similarity in the feature space the soft pscudo-label instead of employing the proposed GIM, and Class-wise indicates that we use the class-wise average feature of point labels the fı of the Gaussian distribution instead of detecting the pseudo-apex frame. The comparison with Hard and Soft demonstrates the effectiveness of our proposed GIM for soft pseudo-labeling. Additionally, the comparison with Class-wise highlights the effectiveness of our instance-adaptiv approach and the choice of the μ for the Gaussian distribution. This is because when we select the class-wise average feature corresponding to point labels as the of the Gaussian distribution, even expressions within the same class can vary greatly in intensity. Therefore, considering the whole class instead of individual expression instances may cause some expressions to be assigned low soft labels, leading the model to ignore certain expressions, which negatively affects the performance. In summary, the results demonstrate the effectiveness of our instance-adaptive Gaussian distribution, built based on pscudo-apex frames and feature distances, for describing the expression intensity distribution.
4.4 QUALITATIVE EVALUATION
Pseudo-label results. For intuitive illustration, present some qualitative results of pseudo- labels, which are shown in Figure 4 According to the results, we can see that our GIM can detect apex frames more accurately than just using point labels the pseudo-apex frames. In addition, our GIM can precisely estimate the duration of the expression instance, thus assigning reliable soft pseudo-labels.
Intensity score results of an entire video. We present the expression intensity score results for an entire video, shown in F The results demonstrate that our method significantly suppresses neutral noise while maintaining the intensity of expressions, highlighting the effectiveness of our proposed method.
CONCLUSION
In this paper, we investigated point-supervised facial expression spotting (P-FES). For this purpose, we proposed a two-branch framework by converting the general binary classification-basee class- agnostic branch to regression- based expression intensity branch to model the expression intensity distribution of each expression instance. In the expression intensity branch, we introduced a Gaussian- based -staaiceaddativve Intensity Modeling (GIM) module for soft pseudo-labeling During training, we detected the pseudo-apex frame around each labeled frame and estimated the rough duration of each expression instance. Then, we built the Gaussian distribution centered at the pseudo-apex frame and assigned soft pseudo-labels all potential expression frames in the estimated duration. In addition, we introduced an Intensity-Awart Contrastive (IAC) loss on pseudo-neutral frames and pseudo-expret expression frames with various intensities to enhance feature learning and further suppress neutral noise. Extensive guantitative and qualitative experiments on the SAMM-LV and CAS(ME)2 datasets demonstrated the effectiveness of proposed method.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was partially supported by Innovation Platform for Society 5.0 from Japan Ministry of Ed- ucation, Culture, Sports, Science and Technology. and JSPS KAKENHI Grant Number JP24K03010. This work was also supported by JST SPRING, Grant Number JPMJSP2138.
REFERENCES
Xianye Ben, Yi Ren, Junping Zhang. Su-Jing Wang. Kidiyo Kpalma, Weixiao Meng, and Yong-Jin Liu. Video-based facial micro-eppression analysis: A survey of datasets, features and algorithms. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(9):5826- 2021.
Navaneeth Bodla, Bharat Singh, Rama Chellappa, and Larry S Davis. Soft-nms Soft-mms-improving object detection with one line of code. In Proceedings of the International Conference Computer Vision, pp. 5561-5569. IEEE, 2017.
Yicheng Deng, Hideaki Hayashi, and Hajime Nagahara. Multi-scale spatio-temporal graph convolu- tional network for facial expression spotting. In Proceedings of the International Conference on Automatic Face and Gesture Recognition. IEEE, 2024a.
Yicheng Deng, Hideaki Hayashi, and Hajime Nagahara. Spotformer: Multi-scale spatio-temporal transformer for facial expression spotting. arXiv preprint arXiv:2407.20799, 2024b.
Paul Ekman. Darwin, deception, and facial expression. Annals of the new York Academy of sciences, 1000(1):205-22)) 2003a.
Paul Ekman. Emotions revealed: Recognizing faces and feelings improve communication and emotional life. Holt Paperback, 120-8))140-140, 2003b.
Paul Ekman and Wallace V Friesen. Nonverbal leakage and clues to deception. Psychiatry, 32(1): 88-106, 1969.
Jie Fu, Junyu Gao, and Changsheng Xu. Compact representation and reliable classification learning for point-level reppri-sueereieed action localization. IEEE Transactions on Image Processing, 31: 7363-7377, 2022.
Fa-Tine Ta-Ting Hong, Jia- lia-Chang Feng, Dan Xu, Ying Shan, and Wei-Shi Zheng. Cross-modal consensus network for weakly supervised temporal action localization. In Proceedings of the International Conference Multimedia, pp. 1591-1599, ACM, 2021.
Prannay Khosla, Piotr Teterwak, Chen Wang. Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. In Advances in Neural Information Processing Systems, volume 33, pp. 18661-18673 2020.
Diederik P Kingma. Adam: A method for stochastic optimization. In Proceedings of the International Conference on Learning Representations, 2015.
Pilhyeon Lee and Hyeran Byun. Learning action completeness from points for seakry-superrised temporal action localization. Proceedings of the International Conference on Computer Vision, pp. 13648-13657. IEEE/CVF, 2021.
Wenhao Leng, Sirui Zhao, Yiming Zhang, Shiifeng Liu, Xinglong Mao, Hao Wang, Tong Xu, and Enhong Chen. Abpn: Apex and boundary perception network for micro-and macro-expression spotting. In Proceedings of the International Conference Multimedia, pp. 7160-7164. ACM, 2022.
Chen Liang, Wenguan Wang, Jiaxu Miao, and Yi Yang. Gmmsseg Gaussian mixture based generative semantic segmentation models. Advances in Neural Information Processing Systems, 35:31360- 31375, 2022.
Tianwei Lin, Xu Zhao, Haisheng Su, Chongjing Wang, and Ming Yang. Bsn: Boundary sensitive network for temporal action proposal generation. In Proceedings of the European conference on computer vision, pp. 3-19, 2018.
11

---

Published as a conference paper at ICLR 2025
Gen-Bing Liong, John See, and Lai- Kuan Wong, Shallow optical flow three- stream emn for macro-ane and micro-expression spotting from long videos. In Proceedings of the International Conference on Image Processing, pp. 2643-2647. IEEE, 2021.
Gen Bing Liong, Sze-Teng Liong, John See, and Chee-Seng Chan. MTSN: A multi-temppral stream network for spotting facial macro-and micro-expression with hard and soft pseudo-labels. In Advanced Techniques for Multi-Modal
Proceedings of the Workshop on Facial Facial Expression Analysis, pp. 3-10. ACM, 2022.
Jon Liu, Jin Zhang, Wen-Jing Yan, Su-Jing Wang, Guoying Zhao, and Xiaolan Fu. A main directional mean optical flow feature for spontancous micro-expression recognition. JEEE Transactions Affective Computing, 7(4):299-310 2015.
Tohar Lukov, Na Zhao, Gim Hee Lee, and Ser-Nam Lim. Teaching with soft label smoothing for mitigating noisy labels in facial expressions. In Proceedings of the European Conference on Computer Vision. pp. 648-665 688-665.66 Springer, 2022.
Fan Ma, Linchao Zhu, Yi Yang. Shengxin Zha, Gourab Kundu, Matt Feiszli, and Zheng Shou. Sf-net: Single-frame supervision for temporal action localization. In Proceedings of the European Conference on Computer Vision, pp. 420-437. Springer, 2020.
Islam Nassar, Munawar Hayat, Ehsan Abbasnejad, Hamid Rezatofighi, and Gholamreza Haffari. Protocon: Pseudo-label refinement via online clustering and prototypical consistency for efficient semi-supervised learning. In Proceedings of the Conference on Computer Vision and Pattern Recognition, pp. 11641-11650, IEEE/CVF, 2023.
Thorben Ortmann, Qi Wang, and Larissa Putzar. Facial emotion recognition in immersive virtual reality: A systematic literature review. In Proceedings of the International Conference on PErvasive Technologies Related to Assistive Environments, pp. 77-82, 2023.
Fangbing Qu, Su-Jing Wang, Wen-Jing Yan, He Li, Shuhang Wu, and Xiaolan Fu. CAS(ME)?: A database for spontaneous macro-expression and micro-expression spotting and recognition. IEEE Transactions on Affective Computing, 94):4244436. 2017.
Niyati Rawal and Ruth Maria Stock StoburHom Facial emotion expressions in human robot interaction: survey. International Journal of Social Roboties, 14(7):1583 1604, 2022.
Junao Shen, Kun Kuang, Jiaheng Wang, Xinyu Wang, Tian Feng, and Wei Zhang. Cgmgm: A cross- gross-gaussian mixture generative model for few-shot semantic segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp.
2024.
Zheng Shou, Hang Gao, Lei Zhang, Kazuyuki Miyazawa, and Shih-Fu Chang. Autoloc: Weakly- supervised temporal action localization in untrimmed videos. In Proceedings the European Conference on Computer Vision, pp. 154-171. Springer, 2018.
Jinxuan Wang, Shiting Xu, and Tong Zhang. A unique M-pattern for micro-expression spotting in long videos. In Proceedings of the International Conference on Learning Representations, 2024.
Linshan Wu, Zhun Zhong, Leyuan Fang, Xingxin Hc, Qiang Liu, Jiayi Ma, and Hao Chen. Sparsely annotated semantic segmentation with adaptive gaussian mixtures. In Proceedings of the Confer- ence on Computer Vision and Pattern Recognition, pp. 15454-15464. IEEE/CVF, 2023.
Ziying Xia, Jian Cheng, Siyu Liu, Yongxiang Hu, Shiguang Wang, Yijie Zhang, and Liwan Dang. Realigning confidence with temporal saliency information for point-level weakly-supervise temporal action localization. In Proceedings of the Conference on Computer Vision and Pattern Recognition, pp. 18440-18450. IEEE/CVE, 2024.
Bo Yang, Jianming Wu, Zhiguang Zhou, Megumi Komiya, Koki Kishimoto, Jianfeng Xu, Keisuke Nonaka, Toshiharu Horiuchi, Satoshi Komorita, Gen Hattori, al. Facial action unit-based deep learning framework for spotting macro-and miero-expressions in long video sequences. In Proceedings of the International Conference Multimedia, pp.
ACM, 2021.
12

---

Published as a conference paper at ICLR 2025
Le Yang, Houwen Peng, Dingwen Zhang, Jianlong Fu, and Junwei Han. Revisiting anchor mecha- nisms for temporal action localization. IEEE Transactions on Image Processing, 29:8535-8855 2020.
Chuin Hong Yap, Connah Kendrick, and Moi Hoon Yap. Samm long videos: A spontaneous facial micro-and macro-expressions dataset. In Proceedings of the International Conference on Automatic Face and Gesture Recognition, pp. 771-776. IEEE, 2020.
Chuin Hong Yap, Moi Hoon Yap, Adrian Davison, Connah Kendrick, Jingting Li, Su-Jing Wang. and Ryan Cunningham. 3D-CNN for facial micro-and macro-expression spotting on long video sequences using temporal oriented reference frame. Proceedings of the International Conference Multimedia, pp. 7016-7020. ACM, 2022.
Shukang Yin, Shiwei Wu, Tong Xu, Shifeng Liu, Sirui Zhao, and Enhong Chen. AU-aware graph convolutional network for macroand micro-expression spotting. In Proceedings of the International Conference Multimedia and Expo, pp. 228-233. IEEE, 2023.
Wang-Wam Wang Yu, Jingwen Jiang, and Yong-Jie Li, LSSNet: two-stream convolutional neural network for spotting macro-am and micro-expression in long videos. In Proceedings of the International Conference on Multimedia, pp. 4745-4749. ACM, 2021.
Wang-Wayd Wang Yu, Jingwen Jiang, Kai-Fu Yang, Hong-Mei Yan, and Yong-Jie Li. LGSNet: A two- stream network for miero- and macro-expression spotting with background modeling. IEEE Transactions on Affective Computing, pp. 1-18, 2023.
Wang-Woy Wang Yu, Xian-Shi Zhang, Fu- Ya Luo, Yijun Cao, Kai-Fu Yang, Hong-Mei Yan, and Yong-Jie Li. Weak supervision with arbitrary single frame for micro-and macro-expression spotting. arXiv preprint arXiv:2403. 142444 2024.
He Yuhong. Research on micro-expression spotting method based on optical flow features. In Proceedings of the International Conference on Multimedia, pp. 4803-4807 ACM, 2021.
Huaxin Zhang, Xiang Wang, Xiaohao Xu, Zhiwu Qing. Changxin Gao, and Nong Sang. Hr-pro: Point- supervised temporal action localization via hierarchical reliability propagation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 7115-7123, 2024.
Yuan Zhao, Xin Tong, Zichong Zhu, Jianda Sheng, Lei Dai. Lingling Xu, Xuehai Xia, Yu Jiang, and Jiao Li. Rethinking optical flow methods for micro-expression spotting. Proceedings of the International Conference Multimedia, pp- 7175-7179. ACM, 2022.
13

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 DISCUSSION ON CONSECUTIVE FACIAL EXPRESSION SPOTTING
In typical scenarios, facial expression spotting often focuses individual expressions. However, two or more facial expressions may also occur consecutively within a short time, each with distinct apexes. This raises challenges in learning accurate intensity outputs and spotting consecutive expression instances.
To address this challenge, categorize the issue into two cases: 1) consecutive expression instances of the same class (either MaEs or MEs); 2) consecutive expression instances of different classes.
Soft pseudo-labeling. For the first case, frame might have multiple soft pseudo-labels when multiple close point-labels are used to build partially overlapping Gaussian distributions for soft pseudo-labeling. When assigning a soft pseudo-label to single frame that already has soft pseudo-label (not 0), we randomly retain one label and discard the other. This is done to prevent our model from being biased toward either high- high-intensity or low-intensity output, which could affect the performance of our model.
We further validate the effectiveness of our solution by comparing it with two other optional solutions: 1) keeping the higher soft label when two soft labels are generated for single frame, denoted as 'Higher'; 2) keeping the lower soft label when two soft labels are generated for a single frame, denoted as 'Lower'. The results are shown in Table 4 which demonstrate that the 'random selection' strategy achieves the best overall performance.
Table 4: Ablation study on pseudo-labeling strategies for consecutive expression instances.
SAMM-LV
CAS(ME)
Strategy
MaE
ME
Overall
MaE
ME
Overall
Higher
0.4104
0.1885
0.3473
0.4338
0.0882
0.3988
Lower
0.4304
0.1579
0.3570
0.4276
0.0579
0.3891
Random (Ours)
0.4189
0.2033
0.3587
0.4395
0.0588
0.4000
For the second case, is practical for us use two arrays to store the soft pseudo-labels for the two different classes separately, even if the two classes share the same intensity output. This means that one frame may have two soft pseudo-labels for intensity supervision. When calculating the loss LCIM between the output intensity and pscudo-labels, both soft pseudo-labels are used, and the two loss values are averaged. This strategy enables our model to learn various intensity information of possible composite expressions that occur in different facial areas (e.g.. person is performing MaE with the eyebrows, and an ME occurs at the mouth corner later).
SAMM 011 3
SAMM 037
CASIMEJ 32 0503
CASIMEY 32 0505
LAC
111111
Ous
Figure 6: Expression intensity results for several examples of consecutive facial expressions. Each subfigure includes the expression intensity outputs of LAC (Lee & Byun 2021 (first row) and our model (second row). Orange lines and blue lines represent the ground-truth boundaries of two consecutive expression instances, respectively.
Inference phase. During the processoc,s,,ng, a trough will appear in the intensity output between two consecutive expressions, allowing to the multi-threshold strategy detect and separate
14

---

Published as a conference paper at ICLR 2025
them by the trough. The expression class is determined by each proposal's ' pseudo-apex frame, which has the highest intensity score.
Figure O presents more intuitive qualitative results. Combining with the multi-threshold strategy, method can effectively detect and separate consecutive expression instances.
A.2 HYPER-PARAMETER k. FOR MACRO-EXPRESSION
As the hyper-paraaee syper-parameter of the range for soft pseudo-labeling, Ke can affect the model performance significantl.. In our method, ke is set based on the understanding that general MEs last less than 0.5 seconds, while MaEs last longer than 0.5 seconds. Since we pre-process the datasets standardize the frame rate at 30 fps, 0.5 seconds corresponds to 15 frames. In practice, we set &. to 16 for MEs. However, due to the varying length of MaEs, the choice of ke for MaEs can affect the accuracy of assigned soft pseudo-labels. Small number is not helpful for assigning enough reliable soft pseudo labels while a large number could result assigning soft labels to noisy neutral frames or to expression frames of other expression instances. Therefore, evaluated several choices of kg for MaEs, and the results are shown in Table
Table 5: Ablation study on choices of ke for macro-expressions.
SAMM-LV
CAS(ME)
ke for MaEs
MaE
ME
Overall
MaE
ME
Overall
24
0.4240
0.1700
0.3541
0.4272
0.0317
0.3898
0,4189
0.2033
0.3587
0,4395
0.0588
0.4000
48
0.4144
0.1735
0.3510
0.4172
0.0822
0.3811
64
0.3833
0.1826
0.3271
0,4046
0.0563
0.3682
The results show that optimal performance is achieved when k. is set to 32 for MaEs, which is suitable for assigning enough reliable pscudo-MaE frames and enhancing feature learning.
A.3 EVALUATION PSEUDO-APER APEX FRAME DETECTION
Although the facial expression spotting task only requires detecting the onset and offset frames, the apex frame is crucial for further emotion recognition. Therefore, we evaluate the perfor- mance of our model in detecting pseudo-apex frames. Figure shows that as training proceeds, the average frame distance between pseudo-apex and ground-truth apex frames decreases and sta- bilizes a low level, which demonstrates the effectiveness of our method in apex frame detec- tion.
61
CAS(ME)
5.9
SAMMLV
Averge Apec Enor
5.5
5.1
49
23
47 59 Epoch
Figure Changes in the average frame distance be- tween pseudo-apex and ground-truth apex frames during training, evaluated starting from the 5th epoch.
15


Reviews:
[{"review_id": "WnyynZNixo", "reviewer": "Reviewer_nGC4", "summary": "This paper treats facial expression spotting as a point-supervised temporal action localization (P-TAL) problem. Drawing from existing P-TAL techniques, the authors propose a two-branch framework for localizing facial expressions. One branch, known as the action-classification branch, predicts the category scores of the facial expression $S$, while the other branch predicts its intensity score $a$.\n\nThe framework utilizes a proposed Gaussian-based Instance-Adaptive Intensity Modeling approach to establish soft pseudo-labels for each frame based on the predicted intensity and the given point label. The expression intensity branch is optimized by minimizing the difference between the predicted intensity and the soft pseudo-label, while the action-classification branch employs cross-entropy loss, with class labels also derived from the soft pseudo-label. To reduce neutral noise further, the authors introduce an intensity-aware contrastive loss.\n\nThe proposed method is evaluated on two datasets containing both macro and micro facial expressions. Experimental results indicate that this approach surpasses state-of-the-art P-TAL methods in the task of facial expression spotting.", "questions": "Please refer the the questions.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The setting of point-supervised facial expression spotting is more practical than fully-supervised facial expression spotting.\n\n2. The proposed intensity-aware contrastive learning is interesting. Rather than simply pulling together the same-class instances and pushing apart the different-class instances, the authors consider the influences of intensity, i.e., less attention should be paid to the same-class instances with large intensity differences, and to the different-class instances with small intensity differences.", "weaknesses": "1. In the proposed method, the intensity score is shared across all facial expression classes. However, two or more facial expressions may occur within a short clip, each with distinct apexes. This raises a concern: what happens if the method is applied to an untrimmed video containing two closely timed apexes of different facial expressions? It would be beneficial to discuss how the model handles this scenario.\n\n2. The presentation of the inference lacks sufficient detail. Specifically, it would be helpful to clarify what the outer-inner-contrastive score is and how these scores contribute to determining the temporal range and position of the facial expressions.\n\n3.  The soft pseudo-label is established based on the predicted intensity scores. However, these predicted intensity scores can fluctuate throughout the training process. It would be helpful to clarify how the soft pseudo-label is updated in response to these changing intensity scores.\n\n4. The legend of Fig 4 is unclear. The marks of ground-truth boundary frame are missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FiDmHo8ng2", "reviewer": "Reviewer_qxKN", "summary": "In this paper, the author introduced a two-branch framework for Point-supervised facial expression spotting (P-FES) task. \nThe proposed framework consists of two main branches: (1) Expression Intensity branch with a proposed Gaussian-based instance-adaptive Intensity Modeling (GIM) to model the intensity distribution of expression proposals; and (2) Action-Classification branch to classify the expression class from the proposals.\nAn Intensity-Aware Contrastive Learning on pseudo-labeled frame is also presented to enhance the discriminative feature learning process.\nExperiments on SAMM-LV and CAS(ME)2 have shown the advantages of the proposed approach.", "questions": "Please address the concerns in the Weaknesses section.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper addresses interesting issue of facial expression spotting.\n- Ablation study has shown some contributions of the proposed terms.", "weaknesses": "Although the soft-label is well-motivated, it has been widely used as a label smoothing technique in literature.\nMoreover, the training process is complicated and ad-hoc with 3 training stages. However, the improvement is minor and cannot out-perform F-FES techniques.\n\n\n1. During the training process, warm-up epochs are very important as both features and scores are not reliable for mining steps (i.e. extract pseudo-apex frame, gaussian modeling and reliable pseudo-neutral frames). Then, hard label technique is required. As a result, the proposed framework can only give a minor improvement on top of hard label technique. \n\n2. In the proposed framework, one of the mining step is to extract pseudo-neutral frames. This will be challenging for ME classes as the micro-expression frames will be very similar to neutral frames and their feature difference is minor. How can the proposed framework extract reliable pseudo-neutral frames and achieve better performance on ME classes ?\n\n3. How can the duration of each expression class be chosen?\n\n4. More analysis on the pseudo-apex frame is recommended. \nFor example, How are the pseudo-apex frames changing during different training epochs?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ExaGBDmMgX", "reviewer": "Reviewer_2FHD", "summary": "The article proposes a two-branch framework for Point-supervised Facial Expression Spotting (P-FES), which includes a Gaussian-based Instance-Adaptive Intensity Modeling (GIM) module for soft pseudo-labeling. This framework aims to address the problem of locating facial expression instances in untrimmed videos, requiring only a timestamp label for each instance during training.", "questions": "1、Not all facial expression instances conform to a Gaussian distribution in every situation, such as when someone is suddenly startled. The authors do not seem to address how to handle this special case.\n\n2、I hope that Figure 4 can provide a clearer explanation.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1、The motivation of this article is well-founded, and the proposed method effectively addresses the problem of point-supervised facial expression recognition.\n\n2、The experiments are quite comprehensive, and the comparisons with other methods are reasonable.", "weaknesses": "Some details are unclear; see the \"Questions\" section below for more information.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)