# Unsupervised Temporal Adaptation in Skeleton-Based Human Action Recognition
## 1 Introduction  
Skeleton-based human action recognition has been widely exploited for privacy-sensitive computer vision applications, such as human activity analysis [1], interactive gaming [2], and video surveillance [3]. Human actions can be represented by skeleton model videos [4] that encode the trajectories of skeleton joints captured by specialized acquisition systems, offering the merits of being agnostic to the environment and providing compact data representation. Graph Convolution Networks (GCNs) have dominated the recent research [5, 6, 7] on vision-based human action recognition by demonstrating their pivotal competence in aggregating action dynamics from spatial-temporal skeleton topologies and realizing end-to-end sequence-wise action classification.  
Even though training a powerful GCN model for action recognition from large-scale public datasets [8, 9] is promising, the transposition of the resulting model to applications in real-world environments remains challenging. For instance, the data distribution in a target deployment environment can deviate partially from that of the source training domain due to (_i_) the particular image acquisition configuration adopted or (_ii_) variability between subjects performing the same action. This consideration refers to the data domain covariate issue [10]. A common strategy consists of conducting fine-tuning rounds with full supervision using data collected in the target image acquisition configuration to reduce data discrepancy between the source domain and the target domain. However, the collection and annotation of a large volume of skeleton data for fine-tuning is extremely time consuming.  
This work proposes an unsupervised domain adaptation (UDA) approach for skeleton-based human action recognition. It aims to utilize copious data from a controlled source domain, e.g., public datasets or laboratory environments, to pre-train an initial model. The approach then refines a target-adaptive action recognition model to deploy in a real-world (uncontrolled) domain with a fine-tuning phase using unlabeled data from the latter domain. UDA is achieved in two phases: Contrastive Prototype Learning (**CPL**) and Temporal Adaptation Modeling (**TAM**). Specifically, **CPL** utilizes supervised contrastive learning [11] to model discriminative action prototypes on trimmed data from the source domain. **TAM** formulates a self-training layer on the top of action prototypes. Since data in the target domain are unlabeled, TAM first leverages the pre-learned action prototypes to generate pseudo-labels on the target domain. Second, using both pseudo-labeled target data and labeled source data, it trains an action classification model that is adaptive to both domains. The contributions of this work consist of the following three elements:  
* An original domain adaptation framework allows the action recognition network to initially learn human action knowledge from a labeled data domain, e.g., a public dataset or a controlled laboratory environment, to adapt to a target application domain;
* The proposed method requires only labeled data from the source domain and unlabeled data from the target domain for model adaptation, significantly reducing the effort invested in data annotation in the target domain;
* The proposed method is generalizable to state-of-the-art network architectures for unsupervised domain adaptation in action recognition without additional constraints.