STARS: Self-supervised Tuning for 3D Action Recognition in Skeleton Sequences

## Abstract

Abstract Self-supervised pretraining methods with masked prediction demonstrate remarkable within-dataset performance in skeleton-based action recognition. However, we show that, unlike contrastive learning approaches, they do not produce well-separated clusters. Additionally, these methods struggle with generalization in few-shot settings. To address these issues, we propose Self-supervised Tuning for 3D Action Recognition in Skeleton sequences (STARS). Specifically, STARS first uses a masked prediction stage using an encoder-decoder architecture. It then employs nearest-neighbor contrastive learning to partially tune the weights of the encoder, enhancing the formation of semantic clusters for different actions. By tuning the encoder for a few epochs, and without using hand-crafted data augmentations, STARS achieves state-of-the-art self-supervised results in various benchmarks, including NTU-60, NTU-120, and PKU-MMD. In addition, STARS exhibits significantly better results than masked prediction models in few-shot settings, where the model has not seen the actions throughout pretraining. Project page: https://soroushmehraban.github.io/stars/

## 1 Introduction

Human action recognition is receiving growing attention in computer vision due to its wide applications in the real world, such as security, human-machine interaction, medical assistance, and virtual reality [, , , ]. While some previous works have focused on recognizing actions based on appearance information, other approaches have highlighted the benefits of using pose information. In comparison to RGB videos, Representing videos of human activities with 3D skeleton sequences offers advantages in privacy preservation, data efficiency, and excluding extraneous details such as background, lighting variations, or diverse clothing types. Recent models for 3D action recognition based on skeleton sequences have demonstrated impressive results [, , , ]. However, these models heavily depend on annotations, which are labor-intensive and time-consuming to acquire. Motivated by this, in this study, we investigate the self-supervised representation learning of 3D actions.

Figure: Figure 1: Comparison between training time and test-time accuracy on linear evaluation protocol. Training time is evaluated on a single NVIDIA GeForce RTX 3090 GPU.
Refer to caption: extracted/5729889/Figures/train-accuracy_tradeoff.png

Prior studies in self-supervised learning have employed diverse pretext tasks, such as predicting motion and recognizing jigsaw puzzles [, , ]. More recently, current research has shifted its focus towards contrastive learning [, , ] or Mask Autoencoders (MAE) [, ].

Contrastive learning approaches, although effective in learning representations, rely heavily on data augmentations to avoid focusing on spurious features. Without using data augmentations, they are prone to the problem of shortcut learning [], leading to potential overfitting on extraneous features, such as a person’s height or the camera angle, which do not provide a valid cue to discriminate between different actions. As a result, some knowledge expert [] is needed to design different augmentations of the same sequence; and methods that incorporate extreme augmentations in their pretraining pipeline [, ] have shown significant improvements.

MAE-based methods mask out a proportion of the input sequence and use an encoder-decoder architecture to reconstruct the missing information from the input based on the latent representation of unmasked regions. These approaches [, ] have recently outperformed their contrastive counterparts in within-dataset metrics. However, we show that representations learned by these models cannot separate different actions as effectively as contrastive learning-based methods in few-shot settings.

Despite the significant efforts that have been made, how to learn a more discriminative representation of skeletons is still an issue for skeleton-based action recognition. We believe that integrating MAE-based approaches with contrastive-learning methods can enhance the generalizability of representations, while preserving the strong performance of MAE models in within-dataset evaluations. To this end, we propose Self-supervised Tuning for 3D Action Recognition in Skeleton Sequences (STARS), a simple and efficient self-supervised framework for 3D action representation learning. It is a sequential approach that initially uses MAE as the pretext task. In the subsequent stage, it trains a contrastive head in addition to partially tuning the encoder for a few epochs, motivating the representation to form distinct and well-separated clusters. Fig. 1 shows that STARS requires significantly less resources during pretraining compared to contrastive learning approaches. In addition, STARS outperforms both MAE and contrastive learning approaches. In summary, our main contributions are as follows:

- •
We propose the STARS framework, a sequential approach that improves the MAE encoder output representation to create well-separated clusters, without any extra data augmentations, and with only a few epochs of contrastive tuning.
- •
We show that, although MAE approaches excel in within-dataset evaluations, they exhibit a lack of generalizability in few-shot settings. Subsequently, we significantly enhance their few-shot capabilities while maintaining their strong within-dataset performance by employing our method.
- •
We validate the efficacy of our approach through extensive experiments and ablations on three large datasets for 3D skeleton-based action recognition, attaining state-of-the-art performance in most cases.

## 2 Related Work

### 2.1 Self-supervised Skeleton-Based Action Recognition

The objective of self-supervised action recognition is to train an encoder to discriminate sequences with different actions without providing any labels throughout the training. Methods such as LongTGAN [] pretrain their model with 3D skeleton reconstruction using an encoder-decoder architecture; and P&C [] improves the performance by employing a weak decoder. Colorization [] represents the sequence as 3D point clouds and colorizes it based on the temporal and spatial orders in the original sequence.

Several studies explored various contrastive learning approaches, showing promising results [, , , , ]. CrosSCLR [] applies the MoCo [] framework and introduces cross-view contrastive learning. This approach aims to compel the model to maintain consistent decision-making across different views. AimCLR [] improves the representation by proposing extreme augmentations. CMD [] trains three encoders simultaneously and distills knowledge from one to another by introducing a new loss function. I^2MD [] extends the CMD framework by introducing intra-modal mutual distillation, aiming to elevate its performance through incorporating local cluster-level contrasting. ActCLR [] employs an unsupervised approach to identify actionlets, which are specific body areas involved in performing actions. The method distinguishes between actionlet and non-actionlet regions and applies more severe augmentations to non-actionlet regions.

Recently, MAE-based approaches showed significant improvements. SkeletonMAE [] reconstructs the spatial positions of masked tokens. MAMP [] uses temporal motion as its reconstruction target and proposes a motion-aware masking strategy. However, we show that MAE-based methods exhibit limited generalization in few-shot settings when compared to contrastive-learning based approaches.

### 2.2 Combining Masked Autoencoders with Instance Discrimination

Some recent works in the image domain investigated the effect of combining MAE and Instance Discrimination (ID) methods [, , , , ]. iBOT [] combines DINO [] and BEiT [] for the pretext task. RePre [] extends the contrastive learning framework by adding pixel-level reconstruction loss. CAN [] adds gaussian noise to the unmasked patches and it reconstructs the noise and masked patches, and adds a contrastive loss to the encoder output. MSN [] aligns an image view featuring randomly masked patches with the corresponding unmasked image. SiameseIM [] predicts dense representations from masked images in different views. MAE-CT [] proposes a sequential training by adding contrastive loss after MAE training.

Our work is a sequential self-supervised approach for pretraining of skeleton sequences. It initially employs an MAE approach using an encoder-decoder architecture and further enhances the output representation of the encoder by tuning its weights using contrastive learning.

Figure: Figure 2: The overall pipeline of our proposed STARS framework. The first stage uses MAMP [] to reconstruct the motion of masked tokens. The second stage trains parameters of the projector and predictor using a contrastive learning approach in addition to partially tuning the encoder weights.
Refer to caption: x1.png

## 3 Method

### 3.1 Framework Overview

The overall framework of STARS is illustrated in Fig. 2. It is a sequential self-supervised approach consisting of two main stages. The first stage relies on an MAE-like framework to pretrain the weights of the encoder. We use MAMP [] because it shows promising result in 3D action representation learning; however, any alternative MAE-based approach is also applicable. The next stage is designed to tune the parameters of the encoder using an instance discrimination method. Specifically, the second stage replaces the decoder with a projector and predictor []. It trains them in addition to the encoder using Nearest-Neighbor Contrastive Learning (NNCLR) [] to converge to a representation capable of discriminating different sequences. This approach helps the encoder learn to output distinct clusters for different actions, improving its ability to discriminate between various sequences.

### 3.2 MAMP Pre-training (Stage 1)

MAMP [] uses a transformer encoder-decoder architecture to reconstruct motions from the 3D skeleton sequence. It receives the input skeleton sequence $S\in\mathbb{R}^{T_{s}\times V\times C_{s}}$, where $T_{s}$, $V$, and $C_{s}$ are the temporal length, number of joints, and coordinate channels, respectively. Next, the sequence is divided into non-overlapping segments $S^{\prime}\in\mathbb{R}^{T_{e}\times V\times l\cdot C_{s}}$, where $T_{e}=T_{s}/l$ and $l$ is the segment length. This division results in having $T_{e}\times V$ tokens and reduces the temporal resolution by a factor of $l$. Subsequently, the input joints are linearly projected into joint embedding $E\in\mathbb{R}^{T_{e}\times V\times C_{e}}$
where $C_{e}$ is the dimension of embedding features.

As for the pretraining objective and the masking strategy, MAMP leverages the motion information. Given an original sequence $S$, the motion $M\in\mathbb{R}^{T_{s}\times V\times C_{s}}$ is derived by employing temporal difference on joint coordinates:

$$ $M_{i,:,:}=S_{i,:,:}-S_{i-m,:,:},\quad i\in m,m+1,...,T_{s}-1$ (1) $$

where the step size of the motion is controlled by the hyperparameter $m$. Specifically, MAMP uses a stride of $m=l$ to capture motion among different segments of the sequence.

For masking the input sequence based on the motion, the obtained motion $M$ should have the same dimension as the segmented sequence $S^{\prime}$. Hence, the motion $M$ is padded by replicating the sequence and further reshaped into $M^{\prime}\in\mathbb{R}^{T_{e}\times V\times l\times C_{s}}$. Subsequently, to signify the importance of motion in each spatio-temporal segment, the motion intensity $I$ is calculated as follows:

$$ $\displaystyle I=\sum_{i=0}^{l}\sum_{j=0}^{C_{i}}|M^{\prime}_{:,:,i,j}|\in \mathbb{R}^{T_{e}\times V},$ (2) $\displaystyle\pi=\texttt{Softmax}(I/\tau_{1}),$ $$

where $\pi$ indicates the probability of masking each embedding feature, and $\tau_{1}$ is a temperature hyperparameter. Finally, to increase the diversity in mask selection, the Gumbel-Max trick is used:

$$ $\displaystyle g$ $\displaystyle=-\log(-\log\epsilon),\,\epsilon\in U[0,1]^{T_{e}\times V},$ (3) $\displaystyle idx^{\text{mask}}$ $\displaystyle=\texttt{Index-of-Top-K}(\log\pi+g),$ $$

where $U[0,1]$ represents a uniform distribution ranging from 0 to 1, and $idx^{\text{mask}}$ denotes the masked indices.

On the joint embedding $E$, spatio-temporal positional embedding is added and unmasked tokens are passed to the encoder. Following the computation of the encoder’s latent representations, learnable mask tokens are inserted to them according to the mask indices $idx^{\text{mask}}$. The decoder then predicts the motion $M^{pred}$ and the reconstruction loss is computed by applying mean squared error (MSE) between the predicted motion $M^{pred}$ and the reconstruction target $M^{target}$, as follows:

$$ $\displaystyle\mathcal{L}=\frac{1}{|idx^{\text{mask}}|}\sum_{(i,j)\in idx^{ \text{mask}}}\|(M_{i,j,:}^{\text{pred}}-M_{i,j,:}^{\text{target}})\|_{2}^{2}.$ (4) $$

### 3.3 Contrastive tuning (Stage 2)

In the second stage, we replace the decoder with projection and prediction modules. The projection module aligns the encoder representation with a space targeted for contrastive loss. The prediction module takes one positive sample from a pair and generates a representation vector resembling the other sample in the positive pair to minimize the contrastive loss. More specifically, The encoder $f_{\theta}$ receives segmented sequence tokens $S^{\prime}$ and outputs representation tokens $Y_{\theta}=f_{\theta}(S^{\prime})$. After applying average pooling of the output tokens, the projector $g_{\theta}$ aligns the result to the final representation vector $z_{\theta}=g_{\theta}(\bar{Y_{\theta}})$. Following the NNCLR approach, vector $z_{\theta}$ is inserted into the queue $Q$ and is compared to sequence representations from previous iterations. From these representations, the top nearest neighbor is sampled as a positive sample in contrastive loss:

$$ $\text{NN}(z,Q)=\arg\min_{q\in Q}||z-q||_{2}$ (5) $$

Concurrently, the feature vector $z_{\theta}$ is given to predictor module to output the feature $z_{\theta}^{+}$. Next, given positive pairs ($\text{NN}(z,Q)$, $z^{+}$), we have:

$$ $\mathcal{L}_{i}^{\text{NNCLR}}=-\log{\frac{\exp{\left(\text{NN}(z_{i},Q)\cdot z _{i}^{+}/\tau_{2}\right)}}{\sum_{k=1}^{n}\exp{\left(\text{NN}(z_{i},Q)\cdot z_ {k}^{+}/\tau_{2}\right)}}}$ (6) $$

where $\tau_{2}$ is a fixed temperature hyperparameter, $i$ is the sample index in batch of data, and $n$ is the batch size. Notably, in contrast to other contrastive learning approaches, our method operates more effectively with a single, unaltered view of the sequence, without relying on two different augmented views. Additionally, we show (later, in Fig. 4) that after training these two modules, the predictor output representation forms better cluster separation compared to the encoder trained with MAMP framework in previous stage.

In addition to projector and predictor modules, we partially tune the encoder parameters to produce well-separated clusters. Specifically, we use layer-wise learning rate decay [] to tune the second-half of the encoder parameters. This is formulated as:

$$ $LR_{i}=BaseLR*\alpha^{(N-i)}$ (7) $$

where $LR_{i}$ denotes learning rate of the $i$^th layer, $\alpha$ is the learning rate decay, and $N$ is the total number of layers.

## 4 Experiments

### 4.1 Datasets

NTU-RGB+D 60 [] is a large-scale dataset containing 56,880 3D skeleton sequences of 40 subjects performing 60 actions.
In this study, we use the recommended cross-subject (X-sub) and cross-view (X-view) evaluation protocols. In the cross-subject scenario, half of the subjects are selected for the training set, and the remaining subjects are used for testing. For the cross-view evaluation, sequences captured by cameras 2 and 3 are employed for training, while camera 1 sequences are used for testing.

NTU-RGB+D 120 [] is the extended version of NTU-60, in which 106 subjects perform 120 actions in 114,480 skeleton sequences. The authors also substitute the cross-view evaluation protocol with cross-setup (X-set), where sequences are divided into 32 setups based on camera distance and background. Samples from half of these setups are selected for training and the rest for testing.

PKU-MMD [] contains around 20,000 skeleton sequences of 52 actions. We follow the cross-subject protocol, where the training and testing sets are split based on subject ID. The dataset contains two phases: PKU-I and PKU-II. The latter is more challenging because of more noise introduced by larger view variations, with 5,332 sequences for training and 1,613 for testing.

### 4.2 Experimental Setup

Data Preprocessing:
From an initial skeleton sequence, a consecutive segment is randomly trimmed with a proportion $p$, where $p$ is sampled from the range [0.5, 1] during training and, similar to [], remains fixed at 0.9 during testing. Subsequently, the segment is resized to a consistent length $T_{s}$ using bilinear interpolation. By default, $T_{s}$ is set to 120.

Network Architecture: We adpoted the same network architecture as MAMP []. It uses a vanilla vision transformer (ViT) [] as the backbone with $L_{e}=8$ transformer blocks and temporal patch size of 4. In each block, the embedding dimension is 256, number of multi-head attentions is 8, and hidden dimension of the feed-forward network is 1024. It also incorporates two spatial and temporal positional embeddings into the embedded inputs. The decoder used in first stage is similar to the transformer encoder except that it has $L_{d}=5$ layers. In the contrastive tuning modules used in the second stage, the projector module is solely a Batch Normalization [], given the relatively small size of the 256-dimensional embedding space. The predictor module consists of a feed-forward network with a single hidden layer sized at 4096.

Pre-training: The first stage follows the same setting as MAMP []. For the second stage, we use the AdamW optimizer with weight decay 0.01, betas (0.9, 0.95), and learning rate 0.001. In the second stage, we train the projection and prediction modules in addition to finetuning the encoder for 20 epochs. We employ layer-wise learning rate decay with a decay rate of 0.20. All the pretraining experiments are conducted using PyTorch on four NVIDIA A40 GPUs with a batch size of 32 per GPU.

### 4.3 Evaluation and Comparison

**Table 1: Performance comparison on NTU-60, NTU-120, and PKU-MMD in the linear evaluation protocol. Single-stream: Joint. Three-stream: Joint+Bone+Motion. The best and second-best accuracies are in bold and underlined, respectively. * indicates that result is reproduced using our GPUs.**
| Method | Input | NTU-60 | NTU-120 | PKU-II |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| XSub(%) | XView(%) | XSub(%) | XSet(%) | XSub(%) |  |  |
| Other pretext tasks: |  |  |  |  |  |  |
| LongTGAN [] | Single-stream | 39.1 | 48.1 | - | - | 26.0 |
| Contrastive Learning: |  |  |  |  |  |  |
| ISC [] | Single-stream | 76.3 | 85.2 | 67.1 | 67.9 | 36.0 |
| CrosSCLR [] | Three-stream | 77.8 | 83.4 | 67.9 | 66.7 | 21.2 |
| AimCLR [] | Three-stream | 78.9 | 83.8 | 68.2 | 68.8 | 39.5 |
| CPM [] | Single-stream | 78.7 | 84.9 | 68.7 | 69.6 | - |
| PSTL [] | Three-stream | 79.1 | 83.8 | 69.2 | 70.3 | 52.3 |
| CMD [] | Single-stream | 79.4 | 86.9 | 70.3 | 71.5 | 43.0 |
| HaLP [] | Single-stream | 79.7 | 86.8 | 71.1 | 72.2 | 43.5 |
| HiCLR [] | Three-stream | 80.4 | 85.5 | 70.0 | 70.4 | - |
| HiCo-Transformer [] | Single-stream | 81.1 | 88.6 | 72.8 | 74.1 | 49.4 |
| SkeAttnCLR [] | Three-stream | 82.0 | 86.5 | 77.1 | 80.0 | 55.5 |
| I^2MD [] | Three-stream | 83.4 | 88.0 | 73.1 | 74.1 | 49.0 |
| ActCLR [] | Three-stream | 84.3 | 88.8 | 74.3 | 75.7 | - |
| Masked Prediction: |  |  |  |  |  |  |
| SkeletonMAE [] | Single-stream | 74.8 | 77.7 | 72.5 | 73.5 | 36.1 |
| MAMP [] | Single-stream | 84.9 | 89.1 | 78.6 | 79.1 | 52.0* |
| Masked Prediction + Contrastive Learning: |  |  |  |  |  |  |
| STARS-3stage (Ours) | Single-stream | 86.3 | 90.7 | 79.3 | 80.6 | 52.2 |
| STARS (Ours) | Single-stream | 87.1 | 90.9 | 79.9 | 80.8 | 52.7 |

In all evaluation protocols, we report on STARS, the method proposed in section 3, as well as STARS-3stage. STARS-3stage involves a three-stage pretraining process. The second stage is divided into two parts: the Head Initialization stage, where only the projector and predictors are trained, and the contrastive tuning stage, where the encoder is fine-tuned along with the head modules. More details can be found in the supplementary materials.

Linear Evaluation Protocol: In this protocol, the weights of the pretrained backbone are frozen and a linear classifier is trained with supervision to evaluate the linear-separability of the learned features. We train the linear classifier for 100 epochs with a batch size of 256 and a learning rate of 0.1, which is decreased to 0 by a cosine decay schedule. We evaluate the performance on the NTU-60, NTU-120, and PKU-II datasets. As shown in Tab. 1, our proposed STARS outperforms other methods on both NTU benchmarks. On the PKU-II dataset, STARS achieves second-best result, and SkeAttnCLR [] outperforms it using a three-stream input method.

**Table 2: Performance comparison on NTU-60, NTU-120, and PKU-MMD in the KNN evaluation protocol (K=1).**
| Method | NTU 60 | NTU 120 |  |  |
| --- | --- | --- | --- | --- |
| XSub(%) | XView(%) | XSub(%) | XSet(%) |  |
| P&C [] | 50.7 | 75.3 | 42.7 | 41.7 |
| ISC [] | 62.5 | 82.6 | 50.6 | 52.3 |
| MAMP [] | 63.1 | 80.3 | 51.8 | 56.1 |
| CrosSCLR-B [] | 66.1 | 81.3 | 52.5 | 54.9 |
| CMD [] | 70.6 | 85.4 | 58.3 | 60.9 |
| I^2MD [] | 75.9 | 83.8 | 62.0 | 64.7 |
| STARS-3Stage (Ours) | 76.9 | 88.0 | 65.7 | 68.0 |
| STARS (Ours) | 79.9 | 88.6 | 67.6 | 67.7 |

KNN Evaluation Protocol: An alternative way to evaluate the pretrained encoder is by directly applying a K-Nearest Neighbor (KNN) classifier to their output features. Following other works [, , ], each test sequence is compared to all training sequences using cosine similarity and the test prediction is based on the label of the most similar neighbor (i.e. KNN with k=1). Tab. 2 compares different methods using KNN evaluation protocol. Notably, we find that MAMP cannot achieve competitive results compared to contrastive learning models, despite showing superior results on linear evaluation. We believe that this is because of the pretraining objective of contrastive learning models, which, by pushing different samples into different areas of the representation space, results in better-separated clusters. Our STARS approach leverages contrastive tuning to enhance the feature representation of MAMP, outperforming all other methods. This demonstrates the superiority of contrastive tuning over contrastive learning approaches.

**Table 3: Performance comparison on NTU-60, NTU-120, and PKU-MMD in terms of the fine-tuning protocol. The best and second-best accuracies are in bold and underlined, respectively. * TF stands for Transformer.**
| Method | Input | Backbone | NTU 60 | NTU 120 |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| XSub(%) | XView(%) | XSub(%) | XSet(%) |  |  |  |
| Other pretext tasks: |  |  |  |  |  |  |
| Colorization [] | DGCNN | Three-stream | 88.0 | 94.9 | - | - |
| Hi-TRS [] | Transformer | Three-stream | 90.0 | 95.7 | 85.3 | 87.4 |
| Contrastive Learning: |  |  |  |  |  |  |
| CPM [] | Single-stream | ST-GCN | 84.8 | 91.1 | 78.4 | 78.9 |
| CrosSCLR [] | Three-stream | ST-GCN | 86.2 | 92.5 | 80.5 | 80.4 |
| I^2MD [] | Single-stream | GCN-TF* | 86.5 | 93.6 | 79.1 | 80.3 |
| AimCLR [] | Three-stream | ST-GCN | 86.9 | 92.8 | 80.1 | 80.9 |
| ActCLR [] | Three-stream | ST-GCN | 88.2 | 93.9 | 82.1 | 84.6 |
| HYSP [] | Three-stream | ST-GCN | 89.1 | 95.2 | 84.5 | 86.3 |
| Masked Prediction: |  |  |  |  |  |  |
| SkeletonMAE [] | Single-stream | STTFormer | 86.6 | 92.9 | 76.8 | 79.1 |
| SkeletonMAE [] | Single-stream | STRL | 92.8 | 96.5 | 84.8 | 85.7 |
| MAMP [] | Single-stream | Transformer | 93.1 | 97.5 | 90.0 | 91.3 |
| Masked Prediction + Contrastive Learning: |  |  |  |  |  |  |
| W/o pre-training | Single-stream | Transformer | 83.1 | 92.6 | 76.8 | 79.7 |
| STARS-3stage (Ours) | Single-stream | Transformer | 93.2 | 97.5 | 89.8 | 91.3 |
| STARS (Ours) | Single-stream | Transformer | 93.0 | 97.5 | 89.9 | 91.4 |

Fine-tuned Evaluation Protocol: We follow MAMP and by adding MLP head on the pretrained backbone, the whole network is fine-tuned for 100 epochs with batch size of 48. The learning rate starts at 0 and is gradually raised to 3e-4 during the initial 5 warm-up epochs, after which it is reduced to 1e-5 using a cosine decay schedule. As shown in Tab. 3, both MAMP and STARS notably enhance the performance of their transformer encoder without pretraining. However, these results indicate that contrastive tuning following MAMP pretraining does not impact the fine-tune evaluation, and MAMP and STARS achieve nearly identical results, both outperforming other approaches.

Transfer Learning Protocol: In this protocol, the transferability of the learned representation is evaluated. Specifically, the encoder undergoes pretraining on a source dataset using a self-supervised approach, followed by fine-tuning on a target dataset through a supervised method. In this study, NTU-60 and NTU-120 are selected as the source datasets, with PKU-II chosen as the target dataset. Tab. 4 shows that when fine-tuned on a new dataset, masked prediction techniques like SkeletonMAE and MAMP demonstrate superior transferability compared to contrastive learning methods. Moreover, STARS enhances performance when pre-trained on NTU-60, but its effectiveness diminishes when pre-trained on NTU-120.

Few-shot Evaluation Protocol: This protocol evaluates the scenario where only a small number of samples are labeled in the target dataset. This is crucial in practical applications like education, sports, and healthcare, where actions may not be clearly defined in publicly available datasets. In this protocol, we pretrain the model on NTU-60 (XSub) and evaluate it on the evaluation set of 60 novel actions on NTU-120 (XSub) using $n$ labeled sequences for each class in $n$-shot setting. For the evaluation, we calculate the cosine distance between the test sequences and the exemplars, and use $n$-nearest neighbors to determine the action. Tab. 5 compares different methods in the few-shot settings. Notably, MAMP demonstrates poor generalization performance, in contrast to its robust performance in transfer learning and evaluations within the dataset. By applying contrastive tuning, STARS surpasses contrastive learning approaches in all settings, demonstrating its strength in various evaluations.

**Table 4: Performance comparison in the transfer learning protocol, where the source datasets are NTU-60 and NTU-120, and the target dataset is PKU-II.**
| Method | To PKU-II |  |
| --- | --- | --- |
| NTU 60 | NTU 120 |  |
| LongTGAN [] | 44.8 | - |
| MS2L [] | 45.8 | - |
| ISC [] | 51.1 | 52.3 |
| CMD [] | 56.0 | 57.0 |
| HaLP+CMD [] | 56.6 | 57.3 |
| SkeletonMAE [] | 58.4 | 61.0 |
| MAMP [] | 70.6 | 73.2 |
| STARS-3stage (Ours) | 71.8 | 72.7 |
| STARS (Ours) | 71.9 | 72.2 |

**Table 5: Performance comparison in the few-shot settings, where the model is pretrained on NTU-60 XSub and tested on 60 new samples of NTU-120 XSub.**
| Method | 1-shot | 2-shot | 5-shot |
| --- | --- | --- | --- |
| MAMP [] | 47.6 | 44.4 | 48.4 |
| AimCLR [] | 48.9 | 45.9 | 51.1 |
| HiCLR [] | 51.7 | 49.6 | 53.8 |
| ISC [] | 55.4 | 53.3 | 57.1 |
| HiCo-Transformer [] | 60.0 | 58.2 | 60.9 |
| CMD [] | 61.2 | 58.2 | 61.3 |
| STARS-3stage (Ours) | 59.3 | 57.8 | 61.5 |
| STARS (Ours) | 63.5 | 62.2 | 65.7 |

### 4.4 Ablation Study

**Table 6: Ablation study on the tuning strategy. The performance is evaluated on the NTU-60 XSub and NTU-60 XView datasets under the KNN evaluation protocol (K=10).**
| Tuning Strategy | NTU-60 |  |
| --- | --- | --- |
| XSub | XView |  |
| DINO | 77.6 | 86.3 |
| DINO${}_{\texttt{aug}}$ | 77.4 | 86.7 |
| MoCo | 72.2 | 86.7 |
| MoCo${}_{\texttt{aug}}$ | 73.9 | 88.0 |
| NNCLR | 81.9 | 89.6 |

**Table 7: Ablation study on the effect of augmentation. The performance is evaluated on the NTU-60 XSub and NTU-60 XView datasets under the KNN evaluation protocol (K=10).**
| Augmentation | NTU-60 |  |
| --- | --- | --- |
| XSub | XView |  |
| Spatial Flip | 85.0 | 90.6 |
| Rotation | 84.8 | 90.1 |
| Axis Mask | 81.2 | 89.0 |
| Shear | 83.2 | 90.4 |
| Spatial Flip + Rotation | 84.6 | 90.4 |
| No Augmentation | 84.5 | 89.6 |

Tuning Strategy Design: Tab. 6 compares the NNCLR strategy used in our STARS framework with DINO and MoCo. DINO [] employs a student-teacher framework. It updates the student’s weights by relying on the teacher’s output, which is constructed using a momentum encoder, as the target. Unlike contrastive learning methods, DINO does not need negative samples for contrast and employs centring and sharpening techniques to prevent collapse. MoCo [] is predominantly used by other contrastive learning approaches in action recognition [, , ]. It uses a memory bank to increase the negative samples in contrastive loss and a key encoder, which is updated via exponential moving average to maintain consistency. As shown in the Tab. 6, NNCLR significantly enhances KNN accuracy by forming better clusters for different actions, while not using any data augmentations. For the remaining two strategies, we also examined the impact of including augmentation through spatial flipping and rotation. Generally, adding augmentations helps the methods achieve better performance; especially for MoCo, which relies on augmentations to construct the positive samples. Note that it is expected for the other two methods to further improve by incorporating more augmentations, which is not the focus of this study. Additional details about the hyperparameters in this ablation study are provided in the supplementary material.

Figure: Figure 3: Ablation study on (a) layer-wise learning decay (b) Queue size. The performance is evaluated on the NTU-60 XSub dataset under the KNN evaluation protocol (K=10).
Refer to caption: x2.png

Figure: Figure 4: The t-SNE visualization of embedding features. We sample 15 action classes from the NTU-60 dataset and visualize the features extracted by our proposed STARS framework and compare it with AimCLR [], CMD [], HiCo-Transformer [], and MAMP [].
Refer to caption: x3.png

Effect of Augmentation: Tab. 7 shows that applying augmentation results in a minor improvement in the KNN evaluation protocol. However, we chose not to use augmentation as our main method since the type of augmentation works heuristically and can result in different behavior in new scenarios, sometimes even degrading performance in cases such as shearing or axis masking. Additionally, we tested data augmentation on different evaluation protocols, such as linear evaluation, and did not observe any performance improvement.

Layer-wise Learning Rate Decay: As shown in Fig. 3 (a), we observe a decrease in accuracy with higher learning decay. Our hypothesis is that increasing the decay causes the encoder to forget the robust representations learned in the initial stage, leading to performance degradation comparable to contrastive learning methods.

Queue size: Fig. 3 (b), shows that queue size has minimal influence in pretraining, and 8k is used for all of our evaluations.

Qualitative Comparison: Fig. 4 compares the t-SNE visualization of our proposed STARS method with AimCLR [], CMD [], HiCo-Transformer [], and MAMP []. CMD adds cross-modal mutual distillation loss to contrastive learning and by ensuring that various input modalities (joint, bone, and motion) exhibit the same neighborhood, it scatters actions across different areas of space and mitigates the impact of applying contrastive learning loss. On the other hand, AimCLR and HiCo-Transformer create distinct clusters through the use of extreme augmentations and by applying contrastive loss at different hierarchical levels, respectively. When compared to MAMP, these two contrastive learning methods exhibit a higher inter-cluster distance than MAMP. Interestingly, actions involving interactions between two individuals, such as kicking, giving objects, and shaking hands, create a distinct higher-level cluster compared to actions involving a single person across various methods. Specifically, MAMP shows the highest distance between these two cluster groups, whereas within each group, the action clusters are closely situated. By employing contrastive tuning, STARS effectively minimizes intra-cluster distance (as seen in examples like sneeze/cough) while maximizing inter-cluster distance. This leads to the formation of clearly separated clusters, each representative of different actions.

## 5 Conclusion

In this work, we proposed a sequential contrastive tuning method. We find that masked prediction methods, despite showing promising results in various within-dataset evaluations, cannot outperform contrastive learning based methods in few-shot settings. By using our STARS framework, we show that we can further enhance the masked prediction baseline while achieving competitive results in few-shot settings, outperforming other models in 5-shot setting. However, when the dataset size is limited for pretraining and for evaluations when encoder is fine-tuned with supervision, STARS cannot add significant value to its baseline.

## Appendix A 3-Stage Design

An alternative pretraining method is to do it in 3 stages. Figure 5 shows the 3-stage design. Initially, when we initialize the projector and predictor modules, we freeze the encoder weights. In the second stage, we exclusively train the projector and predictor modules until they can effectively differentiate between different sequences using the NNCLR approach. Finally, in the third stage, we fine-tune the encoder weights using layer-wise learning rate decay.

## Appendix B Semi-supervised Evaluation Results

In semi-supervised evaluation protocol, we follow previous works [, , ] and fine-tune the pretrained encoder in addition to a post-attached classifier while given a small fraction of the training dataset. Specifically, the performance on the NTU-60 is reported while using 1% and 10% of the training set. Since the training portions are selected randomly, we report the result averaged over 5 different runs as the final result. As shown in Tab. 9, STARS is more effective in all scenarios. Specifically, while using 1% of the training data, leading to an increase in accuracy for the MAMP baseline by 3.1% and 4.2% in cross-subject and cross-view evaluations, respectively.

**Table 8: MoCo hyperparameters for ablation study in tuning strategy design.**
| Hyperparameter | Value |
| --- | --- |
| Learning rate | 0.001 |
| Batch size | 32 |
| Augmentations | Mirroring & Rotation |
| Queue size | 32,768 |
| Momentum | 0.999 |
| Temperature | 0.07 |

## Appendix C Ablation Hyper-parameters

Tab. 10 shows the hyperparameters used in DINOTuning strategy. For simplicity and because of limitation in resources, we used only two global views and did not use any local views in DINO. As shown in Tab. 11, we can see that incorporating local views led to a small improvement in performance. However, it came at the cost of significantly more resources. To be specific, we introduced two local views that randomly trimmed a section of the sequence between 40% and 80% and fed it only to the student network. With these additional views, we had to reduce the batch size to 16 and double the training time. Consequently, in our other experiments, we stuck to using only global views. We also used Sinkhorn-Knopp centering [] a KoLeo regularizer [] to help the convergence. Tab. 8 shows the hyperparameters used in MoCoTuning. Similar to previous approaches [, ], we used 32K as the queue size, 0.999 for the momentum and 0.07 for the temperature.

**Table 9: Performance comparison on the NTU-60 dataset under the semi-supervised evaluation protocol, with the final performance reported as the average of five runs.**
| Method | NTU-60 |  |  |  |
| --- | --- | --- | --- | --- |
| XSub | XView |  |  |  |
| (1%) | (10%) | (1%) | (10%) |  |
| Other pretext tasks: |  |  |  |  |
| LongT GAN [] | 35.2 | 62.0 | - | - |
| ASSL [] | - | 64.3 | - | 69.8 |
| Contrastive Learning: |  |  |  |  |
| MS^2L [] | 33.1 | 65.1 | - | - |
| ISC [] | 35.7 | 65.9 | 38.1 | 72.5 |
| 3s-CrosSCLR [] | 51.1 | 74.4 | 50.0 | 77.8 |
| 3s-Colorization [] | 48.3 | 71.7 | 52.5 | 78.9 |
| CMD [] | 50.6 | 75.4 | 53.0 | 80.2 |
| 3s-Hi-TRS [] | 49.3 | 77.7 | 51.5 | 81.1 |
| 3s-AimCLR [] | 54.8 | 78.2 | 54.3 | 81.6 |
| 3s-CMD [] | 55.6 | 79.0 | 55.5 | 82.4 |
| CPM [] | 56.7 | 73.0 | 57.5 | 77.1 |
| Masked Prediction: |  |  |  |  |
| SkeletonMAE [] | 54.4 | 80.6 | 54.6 | 83.5 |
| MAMP [] | 66.0 | 88.0 | 68.7 | 91.5 |
| Masked Prediction + Contrastive Learning: |  |  |  |  |
| STARS-3stage (Ours) | 68.6 | 88.2 | 72.5 | 91.8 |
| STARS (Ours) | 69.1 | 88.0 | 72.9 | 91.8 |

**Table 10: DINOTuning hyperparameters for ablation study in tuning strategy design.**
| Hyperparameter | Value |
| --- | --- |
| Learning rate | 0.001 |
| Batch size | 32 |
| Augmentations | Mirroring & Rotation |
| Centering | Sinkhorn Knopp |
| KoLeo weight | 0.1 |
| # Global views | 2 |
| # Local views | 0 |
| Student temperature | 0.1 |
| Teacher temperature | 0.04 |
| Teacher momentum | 0.996 |

**Table 11: Effect of including local views on DINOTuning.**
| method | 10-NN | 20-NN | 40-NN |
| --- | --- | --- | --- |
| w/o local views | 77.4 | 77.1 | 77.0 |
| w/ local views | 77.8 | 78.0 | 77.6 |

### C.1 Algorithm psuedo code

Algorithm 1 demonstrates the process in PyTorch-style pseudo code.

Figure: Algorithm 1 PyTorch-style pseudo-code of contrastive tuning in the second stage.

## Appendix D Confusion Matrix

Fig. 6 illustrates the confusion matrix under KNN evaluation protocol when K=10 on NTU-60 XSub dataset. The errors depicted in the figure can be classified into two distinct categories. Firstly, there are errors stemming from a lack of contextual information. For instance, when only a skeletal sequence is provided, actions like ”play with phone/tablet” might be misinterpreted as ”reading” and ”writing.” Secondly, there are errors arising from subtle movements, such as distinguishing between ”clapping” and ”hand rubbing,” which pose challenges for the model in differentiation. In summary, these errors manifest due to either insufficient context or the intricacy of distinguishing minute actions, highlighting the complexities inherent in the task.

Figure: Figure 6: Confusion matrix of KNN evaluation in NTU-60 XSub dataset (k=10).
Refer to caption: extracted/5729889/Figures/confusion_matrix.png

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: Please refer to the weaknesses section. Additionally: 
- Do you use the same feature for linear evaluation, fine-tuning and kNN experiments ? Was one better over the other since a project was added ? 

- Typo: L310 ViT?
soundness: 3
strengths: - The approach is interesting and tries to leverage advantage of both masked-auto encoder and contrastive learning based pre-training. 
- Simplicity of the approach is a strength of the paper. Approach needs minimal additional training to achieve the improvements. 
- The paper is well written and easy to follow for the most part. 
- The presented ablations and design decisions could be helpful to the community.
summary: - The paper tackles the problem of self-supervised learning for skeleton-representation learning. 
- While previous state-of-the-art are MAE-based and achieve good fine-tuning performance, these suffer on linear evaluation, few shot learning due to poorly separated clusters. 
- The paper proposes to have a two stage pre-training of MAE followed by a contrastive learning based training. 
- The approach retains fine-tuning performance while being better at linear evaluation, and few-shot learning. 
- The approach shows that a short contrastive learning stage is enough to obtain good performance.
weaknesses: - Prior work: While simplicity is the strength for this paper. It proposes a combination of two existing approaches. The authors must make it clear if the two stages of training have any differences from the original approaches. A missing reference which also discusses differences in representations of MAE and CL-based pre-training for images and simple ways to use both [a]. Why was the proposed approach used instead of adapting one of the approaches in Section 2.2 for skeleton-based representation learning. 
- How does training with NNCLR loss together with MAMP compare with the proposed approach ? What happens if the final stage employs both MAMP and NNCLR ? 
- L231: "our method". Any changes from NNCLR must be clearly stated to make sure that the readers understand the differences. 
- Experiments; Table 6 - why k=10 ? Table 2 uses k=1 making it difficult to compare. It would be interesting to have the same number of clusters (or both settings) to make comparisons easier. I am surprised that other approaches are so bad (Table 6) especially with MoCo which as the setup used in CMD. 
- Since NNCLR was shown to be very effective post MAE training, do you have any baselines which uses NNCLR alone and compare it with MAMP and the proposed approach ? 
- Table 5: How sensitive is the approach to different runs. How many times was this experiment repeated? Why do we see a drop from 1-shot to 2-shot ?


[a] What do self-supervised vision transformers learn?

Please also refer to section on "Questions"

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: See the above weaknesses.
soundness: 3
strengths: 1. This paper proposes the STARS framework, which combines MAE with contrastive learning, and can significantly improve the output representation of the MAE encoder with only a small amount of contrastive tuning.
2. Extensive experiments and ablation studies have been conducted on three large-scale 3D skeleton action recognition datasets, effectively proving the effectiveness of the method, and in most cases, reaching the state-of-the-art performance level.
summary: This article proposes a human behavior analysis framework called STARS, aimed at improving the output representation of MAE encoders, thereby creating well-separated clusters without the need for any additional data augmentation. Extensive experiments and ablation studies conducted on three large-scale 3D skeleton action recognition datasets have verified the effectiveness of the STARS method.
weaknesses: 1. Compared to some other contrastive learning methods (such as AimCLR, CMD), the STARS method only relies on single-view sequences for operation and does not use two different augmented views. Theoretically, its performance may be limited under cross-view evaluation on the NTU dataset. However, the cross-view evaluation experiment results in Table 1 and Table 3 are better than them. The article lacks relevant explanatory analysis.
2. As shown in the experimental results of Table 1, when the pre-trained and fine-tuned encoders are on a limited dataset, the difference in effect between the STARS method and other mask prediction methods (such as Skeleton MAE, MAMP) is not obvious, indicating that its generalization capability on small datasets needs further improvement.
3. The method uses some tricks in the ablation experiments, such as hierarchical learning rate decay (Formula 7); in addition, the Backbone is not aligned with other methods (Table 3), making it difficult to determine whether these tricks have brought benefits.
Experiments to be carried out:
4. Since the Backbone applied by the STARS method is different from other methods, its universality and effectiveness on different Backbones need to be verified (experiments need to be supplemented in Table 3).

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. What is the difference between the core idea in this paper and the MAE-CT [1]? It seems that MAE-CT also trains MAE first and tunes the projector and predictor second. Besides, the pretraining pipeline utilized in this paper is MAMP, and the tuning framework used is the NNCLR. So, what is the novelty or the actual contribution of this work? Just employing them in the skeleton action recognition task? It is just an engineering work composed of several existing techniques.
2. Some state-of-the-art methods are not compared, e.g., PCM3 [2], UmURL [3]. The author should supply them for a comprehensive comparison.
3. Compared to the previous studies, the training time is decreased. However, the memory requirements look like they are increasing; 4 A40 GPUs are needed. Meanwhile, the queue size can influence this method's computation overhead. The author should supply the overhead comparison or other metrics to demonstrate the effectiveness of their method.

[1] Lehner J, Alkin B, Fürst A, et al. Contrastive tuning: A little help to make masked autoencoders forget[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2024, 38(4): 2965-2973.

[2] Zhang J, Lin L, Liu J. Prompted contrast with masked motion modeling: Towards versatile 3d action representation learning[C]//Proceedings of the 31st ACM International Conference on Multimedia. 2023: 7175-7183.

[3] Sun S, Liu D, Dong J, et al. Unified multi-modal unsupervised representation learning for skeleton-based action understanding[C]//Proceedings of the 31st ACM International Conference on Multimedia. 2023: 2973-2984.
soundness: 3
strengths: 1. The paper is well-written, and the techniques sound reliable.
2. The work provides comprehensive experiments that are effective.
summary: This paper proposes a new method for self-supervised skeleton action recognition. Specifically, it first pre-trains the encoder through the mask reconstruction in MAE, then a tuning strategy with contrastive learning to enhance the inter-class separability among actions. Extensive experiments demonstrate the superior performance of their method. Meanwhile, the code is released.
weaknesses: 1. The novelty of this paper is limited. It seems like the composition of existing methods and the contribution is not clear.
2. Multi-stages pertaining is more complex than previous studies. Although the training time is decreased, the computation overhead must be considered.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. Please provide a comprehensive description about training process of each model (in Fig. 1) or adopt more metrics, as detailed in the Weakness 2.

2. To provide a thorough evaluation of generalizability, it is suggested to add more empirical comparison (e.g., the naive MAE and its different representative variants), more qualitative analysis (e.g., the generalization performance of learned features on similar or different action classes), and more evaluation scenarios (e.g., other datasets rather than only NTU-60 and NTU-120). (see Weakness 3)

3.  It is suggested to quantify the required number of epochs instead of using "a few epochs" in the contribution part. Is the number of epochs fixed or does it need to be adjusted according to different scenarios? All these questions should be clearly clarified in the paper, for a more solid and improved presentation of this paper.

4.  How is the performance of other MAE-based approaches when applied to the proposed framework? (detailed in Weakness 5)

5. What is the reason for "use layer-wise learning rate decay Clark et al. (2020) to tune the second-half of the encoder parameters"? (see Weakness 6)

6. The result "52.0*" of MAMP evaluated on PKU-II (Table 1) is not the same as the original paper, do different GPUs influence the performance? (see Weakness 7)

7. The authors should provide comprehensive results in the ablation study. (detailed in Weakness 8)
soundness: 2
strengths: 1. This paper empirically compares the performance of masked prediction methods (MAE) and contrastive learning methods on action recognition, and conceptually introduces nearest-neighbor contrastive learning into MAE by partial self-supervised tuning to enhance the generalization performance. This idea is simple and effective.

2. This work conducts relatively comprehensive experiments under different cases, covering conventional protocols and new few-shot evaluation protocol, and compares with existing state-of-the-arts to show the effectiveness of method. 

3. The paper writing is easy to follow with clear technical description and presentation.
summary: This paper proposes an STARS framework that combines Masked Auto-Encoding (MAE) and contrastive learning to perform self-supervised tuning for 3D action recognition. It utilizes MAE as the pretext task and trains a contrastive head to partially tune the encoder, so as to learn distinct clusters for better action recognition. Experiments demonstrate the effectiveness of STARS on various benchmarks, including NTU-60, NTU-120, and PKU-MMD. This work also shows the limited generalizability of MAE approaches in few-shot settings, and verifies the higher efficacy of STARS under different protocols (linear evaluation, KNN evaluation, fine-tuned evaluation, etc.) in most cases.
weaknesses: 1. This work lacks sufficient novelty. It seems that the authors stack and combine existing technologies, such as MAE and contrastive learning approaches, to build the proposed framework for 3D action recognition. 

2. The comparison with other models in Fig. 1 is vague and not specific. The authors do not provide any information about training parameter size, training computational complexity (e.g., GFLOPs), etc. The training time can be influenced by many factors such as machines, GPU ability, I/O speed, etc. So I do not think the comparison in Fig. 1 can fairly show the efficiency or resource usage of the proposed method, unless providing a comprehensive description about training process of each model or the above metrics.

3. This work claims that "MAE approaches exhibit a lack of generalizability in few-shot settings", but it lacks sufficient experiments and deeper analyses to support this crucial claim. Authors seem to use only MAMP as an example of MAE approaches to compare in few-shot settings (Sec. 4.3), but other MAE methods are not included in the comparison and analyses. To provide a thorough evaluation of generalizability, it is suggested to add more empirical comparison (e.g., the naive MAE and its different representative variants), more qualitative analysis (e.g., the generalization performance of learned features on similar or different action classes), and more evaluation scenarios (e.g., other datasets rather than only NTU-60 and NTU-120).

4. It is suggested to quantify the required number of epochs instead of using "a few epochs" in the contribution part. Is the number of epochs fixed or does it need to be adjusted according to different scenarios? All these questions should be clearly clarified in the paper, for a more solid and improved presentation of this paper.

5. The paper mentioned that "any alternative MAE-based approach is also applicable", and stated that MAMP shows promising results. How is the performance of other MAE-based approaches when applied to the proposed framework? As we know, the generality of a method is not only related to a certain evaluation protocol such as the few-shot settings, but also related to its general applicability under different cases. Therefore, it is important to add compared experiments using different MAE-based approaches (there are many MAE methods in this area).

6. Some details of technical components should be added with their motivations. For example, what is the reason for "use layer-wise learning rate decay Clark et al. (2020) to tune the second-half of the encoder parameters"? The authors add this component but do not explain the necessity or importance.

7. Some results in the qualitative comparison seem problematic and not convincing. The result "52.0*" of MAMP evaluated on PKU-II (Table 1) is not the same as the original paper, do different GPUs influence the performance? Authors do not present the best result of CMD (Mao et al 2022), i.e., Three-stream as input, in Table 1. Authors need to carefully check and comprehensively and farily present all published results. The t-SNE visualization in Figure. 3 is also different from other papers, such as CMD. In the CMD paper, the visualization result is significantly better than that shown in this paper.

8. Some result points are missed in the ablation study (Figure 4). For example, the paper only shows the KNN accuracy of α=15, 25, 35, 55, but lacks the point when setting α=45. For queue size experiments, what are the results when setting k=6, 10, 12, 14? Authors should add these results to get a more holistic view of ablation study.

Meta-review: This paper introduces a human behaviour analysis method, STARS, aiming to enhance the output representation of MAE encoders. However, this work lacks sufficient novelty, and it seems to be a stack of existing techniques, like MAE and contrastive learning methods, to build the proposed method of 3D action recognition. Besides, multi-stages pertaining is more complex than previous works. The comparisons with other models are also vague and not specific. Due to the weaknesses, three reviewers recommend rejection.

Decision: Reject