SeFAR: Semi-supervised Fine-grained Action Recognition with Temporal
  Perturbation and Learning Stabilization
Authors: name: Yongle Huang; name: Haodong Chen; name: Zhenbang Xu; name: Zihan Jia; name: Haozhou Sun; name: Dian Shao
Published: 2025-01-02 13:12:12

Abstract: Human action understanding is crucial for the advancement of multimodal
systems. While recent developments, driven by powerful large language models
(LLMs), aim to be general enough to cover a wide range of categories, they
often overlook the need for more specific capabilities. In this work, we
address the more challenging task of Fine-grained Action Recognition (FAR),
which focuses on detailed semantic labels within shorter temporal duration
(e.g., "salto backward tucked with 1 turn"). Given the high costs of annotating
fine-grained labels and the substantial data needed for fine-tuning LLMs, we
propose to adopt semi-supervised learning (SSL). Our framework, SeFAR,
incorporates several innovative designs to tackle these challenges.
Specifically, to capture sufficient visual details, we construct Dual-level
temporal elements as more effective representations, based on which we design a
new strong augmentation strategy for the Teacher-Student learning paradigm
through involving moderate temporal perturbation. Furthermore, to handle the
high uncertainty within the teacher model's predictions for FAR, we propose the
Adaptive Regulation to stabilize the learning process. Experiments show that
SeFAR achieves state-of-the-art performance on two FAR datasets, FineGym and
FineDiving, across various data scopes. It also outperforms other
semi-supervised methods on two classical coarse-grained datasets, UCF101 and
HMDB51. Further analysis and ablation studies validate the effectiveness of our
designs. Additionally, we show that the features extracted by our SeFAR could
largely promote the ability of multimodal foundation models to understand
fine-grained and domain-specific semantics.