# Unsupervised Temporal Adaptation in Skeleton-Based Human Action Recognition
## 3 Method
### Contrastive Prototype Learning (CPL)  
In conventional skeleton-based human action recognition, a classification network receives a skeleton sequence \(X\) where its feature subnetwork (encoder), \(\mathcal{M}:\mathbb{R}^{T\times V\times 3}\to\mathbb{R}^{T\times V\times C_{f \alpha}}\), is responsible for parsing a spatiotemporal skeleton representation of \(X\) into a high-dimensional feature space where hidden states are closely clustered into _action prototypes_ such that the task subnetwork (classifier), \(\mathcal{C}\), is able to determine action-wise classification boundaries. Based on these observations, we propose to use Contrastive Prototype Learning (**CLP**) to learn _action prototypes_ by optimizing \(\mathcal{M}\) using skeleton data from the source domain \(\mathcal{D}_{S}\).  
Specifically, **CPL** utilizes a projection layer, \(\mathcal{P}:\mathbb{R}^{T\times V\times C_{f\alpha}}\to\mathbb{R}^{C_{p\beta}}\), on top of the feature subnetwork \(\mathcal{M}\) to implement contrastive learning. Recent research has utilized a variety of data augmentation schemes [7; 14] to operate skeleton-data-based contrastive learning. In contrast, **CPL** opts out of data augmentation and leverages action labels in contrastive learning. Since data labels carry strong supervisory signals, **CPL** helps learn explicit and discriminative action prototypes.  
Let \(\mathcal{B}_{S}=\left\{\left(X_{s}^{(i)},y_{s}^{(i)}\right)\right\}_{i=1}^{N}\) be a data batch from the source domain \(\mathcal{D}_{S}\), where \(N\ll|\mathcal{D}_{S}|\). Let \(Q=\{1,\ldots,N\}\) be the set of instance indices of \(\mathcal{B}_{S}\). The data batch considers a semantically identical set \(P(i)\) with respect to the _anchor_\(X_{s}^{(i)}\), in which each element \(p\in P(i)\) indices a sequence \(X_{s}^{(p)}\) which has the same sequence label as the _anchor_, i.e., \(P(i)=\left\{p\in A(i):y^{(p)}=y^{(i)}\right\}\), where \(A(i)\in Q\setminus\{i\}\) indicates all elements in \(Q\) except for \(i\). We call \(X_{s}^{(p)}\) a _positive_ counterpart of the _anchor_. In this way, the representation pair \((z_{s}^{(i)},\ z_{s}^{(p)})\) consists of a dynamic skeleton dictionary from which **CLP** can learn underlying feature invariances by \(z^{i}=(\mathcal{P}\circ\mathcal{M})\Big{(}X_{s}^{(i)}\Big{)}\) and \(z^{(p)}=(\mathcal{P}\circ\mathcal{M})\Big{(}X_{s}^{(p)}\Big{)}\). In order to map semantically similar sequences to action prototypes, **CLP** uses a SupCon loss [11] to minimize the distance between feature embeddings of _anchor_ and _positives_:  
\[L_{\text{SupCon}}=-\frac{1}{|\mathcal{B}|}\sum\
olimits_{i\in\mathcal{B}}\frac {1}{|P(i)|}\sum\
olimits_{p\in P(i)}log\frac{\exp\left(z^{i}\cdot z^{(p)}/ \tau\right)}{\sum\
olimits_{n\in\mathcal{N}}\exp\left(z^{i}\cdot n/\tau\right)} \tag{2}\]  
where \(\tau\) acts as the temperature parameter and \(n\) represents a negative counterpart sampled from a memory bank \(\mathcal{N}\) (we follow the same implementation as in [14] on the construction of the memory bank).  
Figure 1: Unsupervised domain adaptation (UDA) framework combining the proposed CPL and TAM strategies. In CPL, the training data (labeled) from the source domain supports supervised representation learning. The learned backbone is reused in TAM for refining over data samples (initially unlabeled) from the target domain. Network embedding is denoted with dotted lines if it is updated during training and with solid lines otherwise.