Gaussian-Based Instance-Adaptive Intensity Modeling for Point-Supervised Facial Expression Spotting

Authors: Yicheng Deng, Hideaki Hayashi, Hajime Nagahara
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: micro-expression spotting, semi-supervised learning, soft pseudo-labeling

Abstract:
Point-supervised facial expression spotting (P-FES) aims to localize facial expression instances in untrimmed videos, requiring only a single timestamp label for each instance during training. To address label sparsity, hard pseudo-labeling is often employed to propagate point labels to unlabeled frames; however, this approach can lead to confusion when distinguishing between neutral and expression frames with various intensities, which can negatively impact model performance. In this paper, we propose a two-branch framework for P-FES that incorporates a Gaussian-based instance-adaptive Intensity Modeling (GIM) module for soft pseudo-labeling. GIM models the expression intensity distribution for each instance. Specifically, we detect the pseudo-apex frame around each point label, estimate the duration, and construct a Gaussian distribution for each expression instance. We then assign soft pseudo-labels to pseudo-expression frames as intensity values based on the Gaussian distribution. Additionally, we introduce an Intensity-Aware Contrastive (IAC) loss to enhance discriminative feature learning and suppress neutral noise by contrasting neutral frames with expression frames of various intensities. Extensive experiments on the SAMM-LV and CAS(ME)$^2$ datasets demonstrate the effectiveness of our proposed framework. Code is available at https://github.com/KinopioIsAllIn/GIM.

Decision: accept (poster)
Reviewer scores: 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730530887958

thread_type:
official_review

<thread_object>
creator: reviewer_2fhd

<content>
summary:
The article proposes a two-branch framework for Point-supervised Facial Expression Spotting (P-FES), which includes a Gaussian-based Instance-Adaptive Intensity Modeling (GIM) module for soft pseudo-labeling. This framework aims to address the problem of locating facial expression instances in untrimmed videos, requiring only a timestamp label for each instance during training.

strengths:
1、The motivation of this article is well-founded, and the proposed method effectively addresses the problem of point-supervised facial expression recognition.

2、The experiments are quite comprehensive, and the comparisons with other methods are reasonable.

weaknesses:
Some details are unclear; see the "Questions" section below for more information.

questions:
1、Not all facial expression instances conform to a Gaussian distribution in every situation, such as when someone is suddenly startled. The authors do not seem to address how to handle this special case.

2、I hope that Figure 4 can provide a clearer explanation.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730667218761

thread_type:
official_review

<thread_object>
creator: reviewer_qxkn

<content>
summary:
In this paper, the author introduced a two-branch framework for Point-supervised facial expression spotting (P-FES) task. 
The proposed framework consists of two main branches: (1) Expression Intensity branch with a proposed Gaussian-based instance-adaptive Intensity Modeling (GIM) to model the intensity distribution of expression proposals; and (2) Action-Classification branch to classify the expression class from the proposals.
An Intensity-Aware Contrastive Learning on pseudo-labeled frame is also presented to enhance the discriminative feature learning process.
Experiments on SAMM-LV and CAS(ME)2 have shown the advantages of the proposed approach.

strengths:
- The paper addresses interesting issue of facial expression spotting.
- Ablation study has shown some contributions of the proposed terms.

weaknesses:
Although the soft-label is well-motivated, it has been widely used as a label smoothing technique in literature.
Moreover, the training process is complicated and ad-hoc with 3 training stages. However, the improvement is minor and cannot out-perform F-FES techniques.


1. During the training process, warm-up epochs are very important as both features and scores are not reliable for mining steps (i.e. extract pseudo-apex frame, gaussian modeling and reliable pseudo-neutral frames). Then, hard label technique is required. As a result, the proposed framework can only give a minor improvement on top of hard label technique. 

2. In the proposed framework, one of the mining step is to extract pseudo-neutral frames. This will be challenging for ME classes as the micro-expression frames will be very similar to neutral frames and their feature difference is minor. How can the proposed framework extract reliable pseudo-neutral frames and achieve better performance on ME classes ?

3. How can the duration of each expression class be chosen?

4. More analysis on the pseudo-apex frame is recommended. 
For example, How are the pseudo-apex frames changing during different training epochs?

questions:
Please address the concerns in the Weaknesses section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730744263750

thread_type:
official_review

<thread_object>
creator: reviewer_ngc4

<content>
summary:
This paper treats facial expression spotting as a point-supervised temporal action localization (P-TAL) problem. Drawing from existing P-TAL techniques, the authors propose a two-branch framework for localizing facial expressions. One branch, known as the action-classification branch, predicts the category scores of the facial expression $S$, while the other branch predicts its intensity score $a$.

The framework utilizes a proposed Gaussian-based Instance-Adaptive Intensity Modeling approach to establish soft pseudo-labels for each frame based on the predicted intensity and the given point label. The expression intensity branch is optimized by minimizing the difference between the predicted intensity and the soft pseudo-label, while the action-classification branch employs cross-entropy loss, with class labels also derived from the soft pseudo-label. To reduce neutral noise further, the authors introduce an intensity-aware contrastive loss.

The proposed method is evaluated on two datasets containing both macro and micro facial expressions. Experimental results indicate that this approach surpasses state-of-the-art P-TAL methods in the task of facial expression spotting.

strengths:
1. The setting of point-supervised facial expression spotting is more practical than fully-supervised facial expression spotting.

2. The proposed intensity-aware contrastive learning is interesting. Rather than simply pulling together the same-class instances and pushing apart the different-class instances, the authors consider the influences of intensity, i.e., less attention should be paid to the same-class instances with large intensity differences, and to the different-class instances with small intensity differences.

weaknesses:
1. In the proposed method, the intensity score is shared across all facial expression classes. However, two or more facial expressions may occur within a short clip, each with distinct apexes. This raises a concern: what happens if the method is applied to an untrimmed video containing two closely timed apexes of different facial expressions? It would be beneficial to discuss how the model handles this scenario.

2. The presentation of the inference lacks sufficient detail. Specifically, it would be helpful to clarify what the outer-inner-contrastive score is and how these scores contribute to determining the temporal range and position of the facial expressions.

3.  The soft pseudo-label is established based on the predicted intensity scores. However, these predicted intensity scores can fluctuate throughout the training process. It would be helpful to clarify how the soft pseudo-label is updated in response to these changing intensity scores.

4. The legend of Fig 4 is unclear. The marks of ground-truth boundary frame are missing.

questions:
Please refer the the questions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732002100422

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We sincerely thank all the reviewers for their constructive comments and for kindly appreciating our motivation and insights. We would like to address all the concerns raised by the reviewers. We separately respond to individual reviewers to address their concerns. Additionally, we revised the manuscript in response to their suggestions, with changes highlighted in red.

The main changes in the revised manuscript regarding the reviewers’ concerns can be summarized as:

1.	We revised Fig. 4 and included additional details to enhance its clarity;

2.	We added more detailed descriptions in Section 3 and 4 regarding the inference process, hyper-parameter setting, and specific ablation studies.

3.	We added an appendix for the further discussion about: 1) disccusion on consecutive facial expression spotting; 2) the choice of hyper-parameter $k_c$ for MaE; and 3) evaluation of pseudo-apex frame detection.

Other changes:

We corrected the range for the rough duration estimation in Eq. (3), which was a typographical error and does not affect any of the original methodology, experimental results, or conclusions. We are sorry for the typo.
</content>
</thread_object>

</thread>


Metareview:
metareview:
Weak Accepted:
The rebuttal provided clarifications about the proposed method and its analysis that were useful for assessing the paper's contribution and responded adequately to most reviewer concerns. All reviewers recommend acceptance after discussion (with marginally above the acceptance threshold), and the ACs concur. The final version should include all reviewer comments, suggestions, and additional clarifications from the rebuttal.