Hyperbolic Self-paced Learning for Self-supervised Skeleton-based Action Representations
Authors: Luca Franco
Paolo Mandica
Bharti Munjal
Fabio Galasso
Venue: ICLR 2023 poster (2023)

Abstract: Self-paced learning has been beneficial for tasks where some initial knowledge is available, such as weakly supervised learning and domain adaptation, to select and order the training sample sequence, from easy to complex. However its applicability remains unexplored in unsupervised learning, whereby the knowledge of the task matures during training.
We propose a novel HYperbolic Self-Paced model (HYSP) for learning skeletonbased action representations. HYSP adopts self-supervision: it uses data augmentations to generate two views of the same sample, and it learns by matching one (named online) to the other (the target). We propose to use hyperbolic uncertainty to determine the algorithmic learning pace, under the assumption that less uncertain samples should be more strongly driving the training, with a larger weight and pace. Hyperbolic uncertainty is a by-product of the adopted hyperbolic neural networks, it matures during training and it comes with no extra cost, compared to the established Euclidean SSL framework counterparts.
When tested on three established skeleton-based action recognition datasets, HYSP outperforms the state-of-the-art on PKU-MMD I, as well as on 2 out of 3 downstream tasks on NTU-60 and NTU-120. Additionally, HYSP only uses positive pairs and bypasses therefore the complex and computationally-demanding mining procedures required for the negatives in contrastive techniques.
Code is available at https://github.com/paolomandica/HYSP.

Published as a conference paper at ICLR 2023
HYPERBOLIC SELF-PACED LEARNING FOR SELF- SUPERVISED SKELETON-BASED ACTION REPRESEN- TATIONS
Luca Franco Paolo Mandica
Bharti Munjal Fabio Galasso
Saaiienza University of Rome Technical University of Munich
ABSTRACT
Self-paced learning has been beneficial for tasks where some initial knowledge is available, such as weakly supervised learning and domain adaptation, to select and order the training sample sequence, from easy to complex. However its appli- cability remains unexplored in unsupervised learning, whereby the knowledge of the task matures during training.
We propose a novel HY perbolicc Self-Paced model (HYSP) for learning skeleton- based action representations. HYSP adopts self-suuervision: uses data augmen- tations to generate two views of the same sample, and it learns by matching one (named online) to the other (the targer). We propose to use hyperbolic uncertainty to determine the algorithmic learning pace, under the assumption that less uncer- tain samples should be more strongly driving the training, with a larger weight and pace. Hyperbolic uncertainty is a by-product of the adopted hyperbolic neural networks, it matures during training and comes with no extra cost, compared to the established Euclidean SSL framework counterparts.
When tested on three established akeleton-based action recognition datasets. HYSP outperforms the state-of-the-ar on PKU-MMD I, well as 2 out of 3 downstream tasks on NTU-60 and NTU-120. Additionally, HYSP only uses pos- itive pairs and bypasses therefore the complex and computationalll-denane mining procedures required for the negatives in contrastive techniques.
Code is available at /tttts::/gggtiiiiee com wwoom/pplooaaneee ca/ /YSP
INTRODUCTION
Starting from the seminal work of Kumar et al. (2010). the machine learning community has started looking self- self-paced learning, i.c. determining the ideal sample order, from easy complex, to improve the model performance. Self-paced learning has been adopted so far for weaakly-supervised learning (Liu et al. [202T, Wang et al. al.] [202T] [Sangineto et 2019), or where some initial knowledge is available, e.g. from a source model, unsupervised domain adaption (Liu et al.) 2021). Self- paced approaches use the label (or pseudo-label) confidence to select easier samples and train on those first. However labels
not available in self-supervised learning (SSL) (Chen et al. 2020aa He et al. 2020 Grill et al. 2020 Chen & He 22021) where the supervision comes from the data structure itself. from the sample embeddings.
We propose HYSP, the first HYperbolic Self-Paced learning model for SSL, In HYSP, the self- pacing confidence is provided by the hyperbolic uncertainty (Ganea et al., 2018 Shimizu et al., 202T) of each data sample. In more details, we adopt the Poincaré Ball model (Surís et al.) 2021 Ganea et al.] 2018 Khrulkov al. 2020) Ermolov al., (022) and define the certainty of each sample as its embedding radius. The hyperbolic uncertainty is a property of each data sample in hyperbolic space, and it is therefore available while training with SSL algorithms.
HYSP stems from the belief that the uncertainty of samples matures during the SSL training and that more certain ones should drive the training more prominently, with a larger pace, at each stage of training. In fact, hyperbolic uncertainty is trained end-to-end and it matures as the training proceeds,
Equul contribution

---

Published as a conference paper at ICLR 2023
Le. data samples become more certain. We consider the task of human action recognition. which has drawn growing attention (Singh et al. 2021: Li et al. 2021; Guo et al.) 2022at Chen et al. 2021a Kim al. 202I) due its vast range of applications, including surveillance, behavior analysis, assisted living and human-computer interaction, while being skeletons convenient light- rep-
resentations, privacy preserving and generalizable beyond people appearance (Xu et al. 2020 Lin
iet al. 2020a).
HYSP builds on top of a recent self-supervised approach, SkeletonCLR (Li al.
202 for train-
ing skeleton-based action representations. HYSP generates two views for the input samples by data
augmentations (He et al. 2020; Chen et al. 2020a Caron et al., 2020; Grill al.
2020 Chen & Hej
2021
Li et al. [202], which are then processed with two Siamese networks, to produce two sample
representations: an online and target. The training proceeds by tasking the former to match the lat- ter, being both of them positives, i.e. two views of the same sample. HYSP only considers positives during training and requires curriculum learning. The latter stems from the vanishing gradients of hyperbolic embeddings upon initialization, due to their initial overconfidence (high radius) (Guo ct al. 2022b). So initially, we only consider angles, which coincides with starting from the contor- mal Euclidean optimization. The former is because matching two embeddings in hyperbolic implies matching their uncertainty too, which appears iol-posed for negatives from different samples, i... uncertainty is specific of each sample at each stage of training. This bypasses the complex and procedures of negative mining, which contrastive techniques require" Both aspects are discussed in detail in Sec. 3.2
cnmpuutationally-dmmaann
We evaluate HYSP on three most recent and widely-adopted action recognition datasets, NTU-60, NTU-120 and PKU-MMD I. Following standard protocols, we pre-train with SSL, then transfer to a downstream skeleton-base action classification task. HYSP outperforms the state-of-the-art on PKU-MMD as well as on out of downstream tasks on NTU-60 and NTU-120.
2 RELATED WORK
HYSP embraces work from four research fields, for the first time, as we detail in this section: self- paced learning, hyperbolic neural networks, self-supervision and skeleton-based action recognition.
2.1 SELF-PACED LEARNING
Self-paced learning (SPL), initially introduced by Kumar et al. (2010) is an extension of curricu- lum learning (Bengio et al. 2009) which automatically orders examples during training based on their difficulty. Methods for self-paced learning can be roughly divided into two categories. One set of methods (Tiang et al. 2014 Wu et al., 2021) employ it for fully supervised problems. For example, Jiang et al. (2014), Wu et al. (2021) employ for image classification. Jiang et al. (2014) enhance SPL by considering the diversity of the training examples together with their hardness to select samples. (W et al. (202T) studies SPI. when training with limited time budget and noisy data. The second category of methods employ it for weakly semi supervised learning. Methods in (Sangineto et al.] (2019); Zhang et al. (2016a) adopt it for weakly supervised object detection, where (Sangineto et al. (2019) iteratively selects the most reliable images and bounding boxes, while Zhang et al. (0166) uses saliency detection for self-pacing. Methods in Peng et al. (2021); Wang et al. (202l I employ SPL for semi-supervised segmentation. |enng et al. (2021) adds a regu- larization term in the loss to learn importance weights jointly with network parameters, Wang el al. 2021 considers the prediction uncertainty and uses generalized Jensen Shannon Divergence loss. Both categories require the notion of classes and not apply to SSL frameworks, where sample em- beddings are only available.
2.2 HYPERBOLIC NEURAL NETWORKS
Hyperbolic representation learning in deep neural networks gained momentum after the pioneer- ing work hyperNNs (Ganea et al. 2018)), which proposes hyperbolic counterparts for the classical (Euclidean) fully-connected layers, multinomial logistic regression and RNNs. Other representa-
tive work has introduced hyperbolic convolution neural networks (Shimizu et al. 2021), hyperbolic
"Similarly to BYOL Grill
2020 HYSP is contrastive technique, as it only adopts positives.
2

---

Published as a conference paper at ICLR 2023
Memory Bank
stop gradiem
predictor
stop gradient
propectoo
projector
projector
projector o
encoder f
encoder
encodar
online cranch
target branch
online te branch
target branch
Inuu sequence
insur
(JJIY
Figure (a) SkeletonCLR is a contrastive learning approach based on pushing the embeddings of two views z and â of the same sample (positives) close to each other, while repulsing from embeddings of other samples (negatives). In the represented Euclidean space, on the left side, the attractingrrepulsive force is straight line. (b) The proposed HYSP maps the embeddings into hyperbolic space, where the sample uncertainty determines the learning pace. The hyperbolic Poincaré ball, on the right side, illustrates the embeddings, where the radius indicates the uncertainty. The attractive force proceeds on a circle, orthogonal to the boundary circle. So the force grows polynomially with the radius. In HYSP, learning is paced by the uncertainty, which the model learns end-to-end during SSL.
graph neural networks (Liu et al. 2019 Chami et al. 2019). and hyperbolic attention networks (Gul- cehre et al. 2019). Two distinct aspects have motivated hy hyperbolic geometry: their capability to en- code hierarchies and tree-like structures Chami al. (2020); Khrulkov al. (2020), also inspiring Peng et al. (2020) for -keleton-based action recognition; and their notion of uncertainty, encoded by the embedding radius (Surís et al. (202T); Atigh et al. (2022), which we leverage here. More re- cently, hyperbolic geometry has been used with self-supervision (Yan et al. 2021: Surís et al. 2021] Ermolov et al., (022).. Particularly, Surís et al. (202I) and Ermolov et al. (2022) let hicrarchical ac- tions and image categories emerge from the unlabelled data using RNNs and Vision Transformers, respectively. Differently, our HYSP uses GCNs and it applies to skeleton data. Moreover, unlike all prior works, HYSP leverages hyperbolic uncertainty to drive the training by letting more certain pairs steer the training more.
2.3 SELF-SUPERVISED LEARNING
Self-supervised learning aims learn discriminative feature representations from unlabeled data. The supervisory signal is typically derived from the input using certain pretext tasks and it is used to pre-train the encoder, which is then transferred to the downstream task interest. Among all the proposed methods (Noroozi & Favaro 2016 Zhang al. 2016b] Gidaris et al. 2018 Caron et al.) 2018 2020), a particularly effective one is the contrastive learning, introduced by SimCLR Chen et al. (2020a). Contrastive learning brings different views of the same image closer ( ("positive pairs' and pushes apart a large number of different ones ('negative pairs"). MoCo (He et al.] 2020 Chen
eet al., 2020b) partially alleviates the requirement of large batch size by using memory-bank of negatives. BYOL (Grill et al.) 2020) pioneers on training with positive pairs only. Without the use of negatives, the risk of collapsing to trivial solutions avoided by means of a predictor in one of the Siamese networks, and momentum-
encoder with stop-gradient in the other. SimSiam (Chen & He 202T) further simplifies the design by removing the momentum encoder. Here we draw on the BYOL design and adopt positive-only pairs for skeleton based action recognition. We experimentally show that including negatives harm training self-supervised hyperbolic neural networks.
2.4 SKELETON-BASER ACTION RECOGNITION
Most recent work in (supervised) Mkeleton-based action recognition (Chen et al. 2021a Wang al. 2020) adopts the ST-GCN model of Yan al. (208)) for its simplicity and performance. We
3

---

Published as a conference paper at ICLR 2023
21A
Figure 2: Analysis of the parts of eq. 5 namely (left) the angle between and h and (right) the quantity (1 || ||bl))), monotonic with the embedding uncertainty (1- -| during the training phases, i.e. initial (orange), intermediate (yellow), and final (light yellow). See discussion in Sec. 3.2.1
also use ST-GCN. The self--upervised skeleton- -bssed action recognition methods can be broadly divided into two categories. The first category uses encoder-decoder architectures to reconstruct the
input (Zheng et al. 2018 Su al. 2020 Paoletti et al. [022]] Yang et al. 202T Su et al. 2021). D. The most representative example is AF-I (Paoletti et al.) (021). which uses Laplacian leguaarizttion to
enforce representations aware the skeletal geometry. The other category methods (Lin et al.)
2020a Thoker et al. 202T Li et al.) 2021 Rao et al. (021) uses contrastive learning with negative samples. The most representative method is SkeletonCLR (Li et al. 2021) which uses momentum encoder with memory- augmented contrastive learning, similar to MoCo. More recent techniques, CrosSCLR (Li et al.) 2021) and AimCLR (Guo et al.) 2022a)), are based on SkeletonCLR. The former extends it by leveraging multiple views of the skeleton by introducing extreme augmentations, features dropout, and nearest neighbors mining for extra positives. We also build on SkeletonCLR, but we turn it into positive-only (BYOL), because only uncertainty between positives is justifiable (cf. Sec. FGX-fill-in). BYOL for skeleton-basee SSL has first been introduced by Moliner al. (2022). Differently from them, HYSP introduces self- -aced SSL learning with hyperbolic uncertainty.
mine more positíves; the latter
3
METHOD
We start by motivating hyperbolic self-paced learning, then we describe the baseline SkeletonCLR algorithm and the proposed HYSP. We design HYSP according to the principle that more certain samples should drive the learning process more predominantly. Self self-pacing is determined by the sample target embedding, which sample online embeddings are trained to match. Provided the same distance between target and online values, HYSP gradients are larger for more certain target fl Since SSL no prior information given about the samples, uncertainty needs to estimated end- to-end, alongside the main objective. This is realized in HYSP by means of hyperbolic mapping and distance, combined with the positives-only design of B YOL (Grill et al. 2020) and a curriculum learning (Bengio et al. 2009) which gradually transitions from the Euclidean the uncertainty- aware hyperbolic space Suris et al. 2021 Khrulkov et al.) 2020).
3.1 BACKGROUND
A few most recent techniques (Guo et al. 2022a Li 2021) for self-supervised skeleton rep- resentation learning adopt SkeletonCLR Li et al. (222)) That is contrastive learning approach, which forces two augmented views an action instance (positives) be embedded closely, while repulsing different instance embeddings (negatives).
Let us consider Fig. [a). The positives x and é are two different augmentations of the same action sequence, e.g. subject to shear and temporal crop. The samples are encoded into y = f(x, 0) and 9 f(t, 0) by two Siamese networks, an online and target encoder with parameters and respectively. Finally the embeddings = and & are obtained by two Siamese projector MLPs, y and ğ.
'Gradients generally depend on the distance between the estimation (here online) and the reference (here target) ground truth, function of the varying estimation. In self-paced learning, references vary during training.
4

---

Published as a conference paper at ICLR 2023
SkeletonCLR also uses negative samples which embeddings of non non-positivee instances, queued in a memory bank after each iteration, and dequeued according to a trsstin-first-ou principle. Positives and negatives employed within the Noise Contrastive Estimation loss (Oord et al. 2018):
exp(z 2/T)
Lnce
log
exp(2 8/7) E. exp(z mm//))/m))
where z .. is the normalised dot product (cosine similarity), m; is negative sample from the queue and r the softmax temperature nyper-parameter.
peculiar of SkeletonCLR, important for its robustness and performance: the mo- mentum encoding for f and the stop-gn gradient the corresponding branch. In fact, the parameters of the target encoder f are updated by back-propagation but by exponential moving average from the online encoder, i.e 8 at + a) where a is the momentum coefficient. similar notation and procedure is adopted for ÿ. Finally, stop-gratd gradient enables the model to only update the online branch, keeping the other as the reference "target".
Two aspects
3.2 HYPERBOLIC SELF-PACED LEARNING
Basing on SkeletonCLR, we propose learn by comparing the online and target h representations in hyperbolic embedding space. We do so by mapping the outputs of both branches into the Poincaré ball model centered at by an exponential map (Ganea al. 2018):
2
h Exph(2) = tanh(ye [+]])
Ve ||@]|
where e represents the curvature of the hyperbolic space and || the standard Euclidean L h and are compared by means of the Poincaré distance:
1 - Nll
Lpoin (h, h)) cosh -1 1 (1+2 (1
[|4]]) (1 ((]1))
with ||al]] and [ the radii of h and h in the Poincaré ball.
Following Surís et al. (2021 define the hyperbolic uncertainty of an embedding as:
UA = ||A0||
(4)
Note that the hyperbolic space volume increases with the radius, and the Poincaré distance is ex- ponentially larger for embeddings closer to the hyperbolic ball edge, i.e. the matching of certain embeddings is penalized exponentially more. Upon training, this yields larger uncertainty for more ambiguous actions, cf. Sec. 4.4 and Appendix A. Next, we detail the self-paced effect on learning.
3.2.1 UNCERTAINTY FOR SELF-PACED LEARNING.
Learning proceeds by minimizing the Poincaré distance via stochastic Riemannian gradient de- scent (Bonnabel) 2013). This is based on the Riemannian gradient of Eq. 3 computed w.r.t. the online hyperbolic embedding h. which the optimization procedure pushes to match the target h:
Loposa (h, 2Va IMP IMP Mp
Allk-& &1 181
(5)
[h-A]]
Learning is self-paced, because the gradient changes according to the certainty the target h, i.e.
the larger their radius [|Å]], the more certain h. and the stronger are the gradients V Lpgin (h, h),
irrespective h. We further discuss the training dynamics of Eq. 5 Let us consider Fig. 2 plotting for each training epoch the angle difference between h and h, namely Z(h h). and respectively the quantities
| and |
*The definition explicitly adopted in their code. This choice differs from Khrulkov 2020 which uses the Poincaré distance, i.e. u, Tppoi (h, 0). However both definitions provide similar rankings of uncertainty, which matters here. We regard this difference calibration issue, which do not investigate.
5

---

Published as a conference paper at ICLR 2023
At the beginning of training (Fig. 2 embeddings are at the edge of the Poincaré ball (1 (1-11/|2 1- |7i]2 0). Assuming that the representations of the two views be still not close (l) h| > 0), the gradient L (h, h) vanishes (Guo et al. 2022b). The risk of stall is avoided by means of curriculum learning
orange section), following random initialization, the
(see Sec. 3.2.3). which considers only their angle for the first 50 epochs;
At intermediate training phases (Fig. 2 yellow section), the radius of h is also optimized. h departs from the ball edge (1 || > in Fig. 2b and it moves towards h. marching along the geodesic (I/4 [[
-|l > 0). At this time, learning is self-paced, i.e. the importance of each sample is weighted according to the certainty of the target view: the higher the radius of ht the larger the gradient. Note that the online embedding h is pushed towards the ball edge, since h is momentum-updattd and it moves more slowly away from the ball edge, where it was initialized. This prevents the embeddings from collapsing to trivial solutions during epochs 50-200;
The training ends (Fig. 2 light yellow section) when à also departs from the ball edge, due to the updates of the model encoder, brought up by / with momentum. Both h and h draws to the ball origin, collapsing into the trivial null solution.
Next, we address two aspects of the hyperbolic optimization: i. the use of negatives is not well- defined in the hyperbolic space and, in fact, leveraging them deteriorates the model performance; ii. the exponential penalization of the Poincaré distance results in gradients which are too small at early stages, causing vanishing gradient. We address these challenges by adopting relatively positive pairs only (Grill ct al. 2020) and by curriculum learning (Bengio et al. 2009), detail next.
3.2.2 POSITIVE-ONLY LEARNING
The Poincaré distance is minimized when embeddings h and å match, both in terms of angle (co- sine distance) and radius (uncertainty). Matching uncertainty of two embeddings only really makes sense for positive pairs. In fact, two views of the same sample may well be required to be known to the same degree. However, repulsing negatives in terms uncertainty not really justifiable, as the sample uncertainty is detached from the sample identity. This has not been considered by prior work (Ermolov et al. 2022). but it matters experimentally, as we show in Sec. 4.3
We leverage BYOL Grill al 2020) and learn from positive pairs only. Referning to Fig. b), we add a predictor head g to the online branch, to yield the predicted output embedding p. Next, is mapped into the hyperbolic space get the embedding which is pushed match the target em- bedding E, reference from the stop-gradient branch, using Eq. 3 Note that this simplifies the model and training, as it removes the need for a memory bank and complex negative mining procedures.
3.2.3 CURRICULUM LEARNING
The model random initialization yields starting (high-dimensie digeessiinsion embeddings h and h with ran- dom dircctions and high norms, which results in vanishing gradients, cf. (Guo et al.] 2022b) and the former discussion training dynamics. We draw inspiration from currieulum learning (Bengio ltt al. 2009) and propose to initially optimize for the embedding angles only, neglecting uncertain- ties. The initial training loss is therefore the cosine distance of h and h. Since the hyperbolic space conformal with the Euclidean, the hyperbolic and Euclidean cosine distances coincide:
h h ||||||
Leos (h, h.)
The curriculum procedure may be written as the following convex combination of losses:
Luxse (h, h)) = a T (h, i) + a.) (h, h))
whereby the weighting term makes the optimization to smoothly transition from the angle-only to the full hyperbolic objective leveraging angles and uncertainty, after an initial stage. (See Eq. 8 in Appendix A).
*The reported MCC results refer to the ST-GCN encoder, for a fair comparison, i.e. this is adopted by all methods in the table.

---

Published as a conference paper at ICLR 2023
Table Results of linear, semi-supervised and finetuning protocols on NTU-60 and NTU-120.
Linear
Somi-sup. (10%)
Pinctune
Additionnll Techniques
NTU--120
NTU-60
Extm
Extra
Method
X3ub
xview
2507
33H0V
xveew
Xnub
2set
Neg.
Aug.
Por.
ME
PuCSu 2029
50.7
76.3
42.7
41.7
20720n
AS-CAT Roo
58.5
48.6
Skekdtnr Lelee of mi
2021
68.3
76.4
56.8
55.9
66.9
67.6
90.5
90.3
75.4
75.9
MCC Dacoler Sure 2027
55.6
59.9
83.0
89.7
79.4
80.8
Ain CRGGG
2022n
ISC Thoker erale M0249
76.3
85.2
67.9
67.1
65.9
72.5
HYYE (oWPS)
78.2
82.6
61.8
64.6
76.2
80.4
96.5
93.5
81.4
82.0
3s ST-GCN
85.2
91.4
77.2
77.1
S--SkeletonCCL er
3h Colorication Zonee tile 2027
75.2
83.1
71.7
78.9
$8.0
94.9
Je CrosSCLR ( wra 2024
77.8
83.4
67.9
66.7
74.4
77.8
96.2
92.5
80.5
80.4
3s-Ain CLR Gao er 20220
78.9
83.8
68.2
68.8
78.2
81.6
86.9
92.8
80.1
80.9
3s-HYSP
Table 2: Results of linear, semi-supervised and finetuning protocols on PKU-MMD
Linear eval.
Semi-sup. (10%)
Finetune
Method
Joint
Motion
3s
Joint
Bone
Motion
Motion
*S Lin ef 20206
64.9
70.3
85.2
Skeleton T. ya at 2027
80.9
72.6
63.4
85.3
ISC Thoker ata 2017
80,9
72.1
AimC
2022n
83.4
82.0
72.0
87.8
86.1
HYSP (ours)
83.8
87.2
70.5
88.8
85.0
87.0
77.8
88.7
94.0
94.9
91.2
96.2
4 RESULTS
Here we present the reference benchmarks (Sec. 4.1) and the comparison with the baseline and the ffateoof-the-rr (Sec. 4.2); finally present the ablation study (Sec. 4.3) and insights into the learnt skeleton-based action representations (Sec. 4.4
4.1 DATASETS AND METRICS
We consider three established datasets:
NTU RGB+D 60 Dataset (Shahroudy et al. 2016). This contains 56,578 video sequences divided into 60 action classes, captured with three concurrent Kinect V2 cameras from 40 distinct subjects. The dataset follows two evaluation protocols: cross-subject (xsub), where the subjects are split evenly into train and test sets, and cross-view (xview), where the samples of one camera are used for testing while the others for training.
NTU RGB+D 120 Dataset (Liu al.) 2020). This is an extension of the NTU RGB+D 60 dataset with additional 60 action classes for total of 120 classes and 113,945 video sequences. The samples have been captured in 32 different setups from 106 distinct subjects. The dataset has two evaluation protocols: cross-subject (xsub), same NTU-60, and cross-setup (xset), where the samples with even setup ids are used for training, and those with odd ids are used for testing.
PKU-MMD I (Chunhui et al. 2017). This contains 1076 long video sequences of 51 action cate- gories, performed by 66 subjects in three camera views using Kinect v2 sensor. The dataset is eval- uated using the cross-subject protocol, where the total subjects are split into train and test groups consisting of 57 and 9 subjects, respectively.
The performance of the encoder is assessed following the protocols Li et al. (0021)
Linear Evaluation Protocol. A linear classifier composed by fully-connected layer and softmax is trained supervisedly top of the frozen pre- pre-trained encoder.
Semi-supervvsee Protocol. Encoder and linear classifier are finetuned with 10% the labeled data. Finetune Protocol. Encoder and linear classifier are finetuned using the whole labeled dataset.

---

Published as a conference paper at ICLR 2023
Table 3: Ablation study on NTU-60 (xview). Top-I accuracy (%) reported.
HYSP
w/
w/o
w/o curr. learn.
w/ 3-stramm ensemble
hyper.
neg.
hyper.
downstream
82.6
69.7
76.9
73.9
85.2
75.5
4.2 COMPARISON TO THE STATE OF THE ART
For all experiments, we used the following setup: ST-GCN Yu et al.
2018 as encoder, Rieman-
nianSGD (Kochurov et al., 2020) as optimizer, BYOL (Grill al. 2020) (instead of MoCo) as
self-supervised learning framework, data augmentation pipeline from (Gu et al. 2022a) (See Ap- pendix B for all the implementation details).
NTU-60/120. In Table 1 we gather results of most recent SSL skeleton-based action recognition methods on NTU-60 and NTU-120 datasets for all evaluation protocols. We mark in the table the additional engineering techniques that the methods adopt: (3x) three stream, using joint, bone and motion combined; (Neg.) negative samples with or without memory bank; (Extra Aug.) extra ex- treme augmentations compared to shear and crop; (Extra Pos.) extra positive pairs, typically via nearest- neighhor or cross- view mining; (ME) multiple encoders, such as GCN and BiGRU, The up- per section of the table reports methods that leverage only information from one stream (joint, bone or motion), while the methods in the lower section use 3-stream information.
As shown in the table, on NTU-60 and NTU- datasets using linear evaluation, HYSP outperforms the baseline SkeletonCLR and performs competitive compared to other state-of-the-art approaches. Next, we evaluate HYSP on the NTU-60 dataset using semi-supervised evaluation. As shown, HYSP outperforms the baseline SkeletonCLR by large margin in both xsub and xview. Furthermore, un- der this evaluation, HYSP surpasses all previous approaches, setting a new state-of-the-art. Finally, we finetune HYSP on both NTU-60 and NTU-120 datasets and compare its performance to the rel- evant approaches. As shown in the table, with the one stream information, HYSP outperforms the baseline SkeletonCLR well as all competing approaches.
Next, we combine information from all streams (joint, bone, motion) and compare results to the relevant methods in the lower section of Table D As shown, on NTU-60 dataset using linear eval- uation, our 3S-HYSP outperforms the baseline 3s- 3s-SkeletonCLL and performs competitive to the current best method 3s-AimCLR. Furthermore, with semi- semi-supervised and fine-tuned evaluation, proposed SP sets new state-of-the on both the NTU-60 and NTU-120 datasets, surpass- ing the current best A..
PKU-MMD I. In Table 2 we compare HYSP to other recent approaches on the PKU MMD I dataset. The trends confirm the quality of HYSP, which achieves state-of-the-ard results under all three evaluation protocols,
4.3 ABLATION STUDY
Building up HYSP from SkeletonCLR is challenging, since all model parts required for the hyperbolic self- self-paced model to converge and perform well. We detail this Table 3 Jusing the linear evaluation on NTU-60 xview:
w/ neg. Considering the additional repulsive force of negatives (i.e, replacing BYOL with MoCo) turns HYSP into a contrastive learning technique. Its performance drops by 12.9%, from 82.6 to 69.7. We explain this as due to negative repulsion being ill-posed when using uncertainty, cf. Sec 3.2 w/o hyper. Removing the hyperbolic mapping. which implicitly takes away the self- self-paeing learning, causes significant drop in performance. by 5.7%. In this case the loss is the cosine distance, and each target embedding weights therefore equally (no self- self-pacing). This speaks for the importance of the proposed hyperbolic self- self-pacing.
w/o curr. learn. Adopting hyperbolic self-pacing from the very start of training makes the model more unstable and leads to lower performance in evaluation (73.9%), re-stating the need to only consider angles at the initial stages.
w/ 3-stream ensemble. Ensembling the 3-stream information from joint, motion and bone, HYSP
improves to 85.2%. Note that all downstream tasks take the SSL-trained encoder f, add to it a linear classifier and train
8

---

Published as a conference paper at ICLR 2023
0.035
(b) Writing (Radius 0.7712)
Cochee dicearce
6.025
0,010
(c) Touch back (Radius 0.8108)
0.000
0.25
0.30
0.15
0.20
0.25
Uncertanty: rediox
(a) Error VS Uncertainty Training
(d) Standing up (Radius 0.8538)
Figure 3: (a) Bar plot of the average cosine distance among positive pairs for specific intervals of uncertainty. The plot shows how cosine distance between embeddings and prediction uncertainty after the pre-text task are highly correlated. (b, c, d) Visualizations of skeleton samples. Uncertainty indicates the difficulty in classifying specific actions if they are characterized by less or more peculiar movements (a) and (b) or unambiguous actions (c).
with cross-entropy, as an Euclidean network. Since HYSP pre-trains in hyperbolic space, it natural to ask whether hyperbolic helps downstream too. A hyperbolic downstream task (linear classifier followed by exponential mapping) yields 75.5%. This is still comparable to other approaches, but below the Euclidean downstream transfer. We see two reasons for this: i. in the downstream tasks the GT is given, without uncertainty; ii hyperbolic self paced learning is most important in pre-traue tre-training, Le. the model has supposedly surpassed the information bottleneck (Achille & Soatto 2018 upon concluding pre-training, for which a self-paced finetuning is not anymore as effective.
4.4 MORE ON HYPERBOLIC UNCERTAINTY
We analyze the hyperbolic uncertainty upon the SSL self self-paced pre-training, we qualitatively relate it to the classes, and illustrate motion pictograms, the NTU-60 dataset.
Uncertainty as a measure of difficulty. We show in Figure the average cosine distance between positive pairs against different intervals of the average uncertainty (1 [M[]). The plot shows clear trend, revealing that the model attributes higher uncertainty to samples which are likely more diffi- cult, i.e. to samples with larger cosine distance.
Inter-class variability. We investigate how uncertainty varies among the 60 action classes, by ranking them using median radius of the class sample embeddings. We observe that: i) actions char- acterized by very small movements e.g writing, lie at the very bottom of the list, with the smallest radii; ii) actions that are comparatively more specific but still with some ambiguous movements (e.g- touch-back) are almost in the middle, with relatively larger radii; iii) the most peculiar and under- standable actions, either simpler (standing up) or characterized by larger unambiguous movements (handshake), lie at the top of the list with large radii. (See Appendix A for complete list).
Sample action pictograms Figure 3 shows pictograms samples from three selected action classes and their class median hyperbolic uncertainty: writing (3b), touch back (3c), and standing up (3d).
"Standing up" features a more evident motion and correspondingly the largest radius; by contrast, "writing" shows little motion and the lowest radius.
LIMITATIONS AND CONCLUSIONS
This work has proposed the first hyperbolic self- self-paced model HYSP, which re-interprets self self-pacing with self-regulating estimates of the uncertainty for each sample. Both the model design and the self-paced training are the result of thorough considerations on the hyperbolic space.
As current limitations, we have not explored the calibration of uncertainty, nor tested HYSP for the image classification task, because modeling has been tailored to the skeleton- based action recogni- tion task and the baseline SkeletonCLR, which yields comparatively faster development cycles. Exploring the overfitting of HYSP at the latest stage of training lies for future work. This is now an
active research topic (Ermolov et al. 2022 Guo et al.) 2021 2022b) for hyperbolic spaces.

---

Published as a conference paper at ICLR 2023
ETHICS STATEMENT
The use of skeletal data respects the privacy of the pictured actors, large extent. In most cases, privacy is affected when the actor's appearance is required, from the identity could be inferred. Although it might be possible to recognize people from their motion (e.g. recognition of their Gait), this is not straightforward. This makes potential misuse case for this work, however less likely, compared to the use images.
REPRODUCIBILITY STATEMENT
To ensure the complete reproducibility of our work, we have included in the abstract the link to the official github repository of the paper. The README file contains instructions on how to download the data, preprocess train the encoder self-supervisedly, and then evaluate the model the various protocols described in Sec. 4.1 The datasets are not included in the repository because they are publicly available for download and use. Furthermore, Appendix provides detailed information about implementation details.
8 ACKNONTEDGEMENT
We are most grateful for the funding from the Panasonic Corporation and the fruitful research dis- cussions with the Panasonic PRDCA-AI Lab. We would like to acknowledge the support of Nvidia AI Technology Center (NVAITC) for computing resources and of Giuseppe Fiameni for his exper- tise that, which enabled their effective use. Authors are also grateful to Anil Keshwani for initial discussions on contrastive learning for videos.
REFERENCES
Alessandro Achille and Stefano Soatto. Emergence of invariance and disentanglement in deep rep- resentations. 2018 Information Theory and Applications Workshop (IIA), pp. 1-9, 2018. doi: 10.1109/TA.200 20w.ITTA......5000000 8.800114.
Mina Ghadimi Atigh, Julian Schoep, Erman Acar, Nanne van Noord, and Pascal Mettes. Hyper- bolic image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4453-4462 June 2022.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th Annual International Conference on Machine Learning, pp. 41-48, New York, NY, USA, 2009. Association for Computing Machinery. ISBN 9781605585161. doi: 10.
1145/1553374 URLphttss dei ort//o/g///../ 11411114141 1tt.:1111/1137.1555555
Silvère Bonnabel. Stochastic gradient descent on riemannian manifolds. IEEE Transactions on Automatic Control, 58(9):2217-2229, 2013. doi: 10.1 10.1109/AA..2213.225555
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsu- pervised learning of visual features. In European Conference on Computer Vision, 2018.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. 2020.
Ines Chami, Zhitao Ying, Christopher Ré, and Jure Leskovec. Hyperbolic graph convolutional neural networks. Advances in neural information processing systems, 32:4868-4877 4879, 2019.
Ines Chami, Albert Gu, Vaggos Chatziafratiss and Christopher Ré. From trees to continuous embed- dings and back: Hyperbolic hierarchical clustering. In Advances in Neural Information Process- ing Systems (NeurIPS), 2020.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference Machine Learning, volume 119 of Pro- ceedings of Machine Learning Research, pp. 1597-1607, PMLR, 13-18 Jul 2020a. https::/prrceeddnng: r.press////eeee
URL
/iress/ve / chwn20..nn00.cm .html
10

---

Published as a conference paper at ICLR 2023
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Xinlei Chen, Haoqi Fan, Ross B. Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. ArXiv, abs/2003.04297, 2020b.
Yuxin Chen, Ziqi Zhang, Chunfeng Yuan. Bing Li, Ying Deng, and Weiming Hu. Channel-wiss topology refinement graph convolution for akeleton-based action recognition. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 13359-13368, 2021a.
Zhan Chen, Sicheng Li. Bing Yang, Qinghan Li, and Hong Liu. Multi-scale spatial temporal graph convolutional network for skeleton- baaeed action recognition. Proceedings of the AAAI Conference Artificial Intelligence, 35(2):1 13-1122, May 2021b.
Liu Chunhui, Hu Yueyu, Li Yanghao, Song Sijie, and Liu Jiaying. Pku-mmd: A large scale bench- mark for continuous multi-modal human action understanding. arXiv preprint arXiv:1703.0774: 2017.
Aleksandr Ermolov, Leyla Mirvakhabova, Valentin Khrulkov, Nicu Sebe, and Ivan Oseledets. Hy- perbolic vision transformers: Combining improvements in metric learning. In IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR), 2022.
Octavian Ganea, Gary Becigneul, and Thomas Hofmann. Hyperbolic neural networks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.). Advances in Neural Information Processing Systems, volume 31. Curran Asso- ciates, Inc., 2018. URL dbab2adc8 f 7800 930 fa81 42-.144r.Pppppppp
S. Gidaris, P. Singh, and N. Komodakis. Unsupervised representation learning by predicting image rotations. In ICLR, 2018.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap Your Own Latent: A new approach to self-supervised learning. In Neural Information Processing Systems, Montréal, Canada, 2020.
Caglar Gulcehre, Misha Denil, Mateusz Malinowski, Ali Razavi, Razvan Pascanu, Karl Moritz Hermann, Peter Battaglia, Victor Bapst, David Raposo, Adam Santoro, and Nando de Freitas. Hyperbolic attention networks. In International Conference Learning Representations, 2019.
URL / net coorummid=rJsae jR&FQ
Tianyu Guo, Liu Hong, Chen Zhan, Liu Mengyuan, Wang Tao, and Ding Runwei. Contrastive learning from extremely augmented skeleton sequences for self-supervised action recognition. In AAAI, 2022a.
Yunhui Guo, Xudong Wang, Yubei Chen, and Stella X. Yu. Free hyperbolic neural networks with limited radii. ArXiv, abs/2107.11472 2021.
Yunhui Guo, Xudong Wang, Yubei Chen, and Stella X. Yu. Clipped hyperbolic classifiers super- hyperbolic classifiers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2022b.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsupervised visual representation learning. 2020 IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2020.
Lu Jiang, Deyu Meng, Shoou-I Yu, Zhenzhong Lan, Shiguang Shan, and Alexander Hauptmann. Self-paced learning with diversity. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger (eds.), Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014.
11

---

Published as a conference paper at ICLR 2023
Valentin Khrulkov, Leyla Mirvakhabova, Evgeniya Ustinova, Ivan Oseledets, and Victor Lempit- sky. Hyperbolic image embeddings. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Hanul Kim, Mihir Jain, Jun-Tae Lee, Sungrack Yun, and Fatih Porikli. Efficient action recognition via dynamic knowledge propagation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 13719- 13728, October 2021.
Max Kochurov, Rasul Karimov, and Serge Kozlukov. Geoopt: Riemannian optimization in pytorch. CoRR, abs/2005.028 2020.
/tap:p::///xxv...o/ www.iv.orggabs////oopoooo 200pg//bs/2055.... 02819
M. Kumar, Benjamin Packer, and Daphne Koller. Self-paced learning for latent vari- able models. In J. Lafferty, C. Williams, J. Shawe-Taylor, R. Zemel, and A. Cu- lotta (eds.), Advances in Neural Information Processing Systems, volume 23, Curran Asso- ciates, Inc., 2010. URL http:: htttps::/proceedinneg newrips wwwpppppp.p.p...eeeeeeepppeepeee wttpc//pppe//10///iii..c e57c6b95 (11166628889989888855553333 Pape:e9977--appr...... pdf
Linguo Li, Minsi Wang, Binghing Ni, Hang Wang. Jiancheng Yang, and Wenjun Zhang. 3d hu- man action representation learning via cross-view consistency pursuit. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Lilang Lin, Sijie Song, Wenhan Yang, and Jiaying Liu. Ms2l: Multi-task supe-supe learning for skeleton based action recognition. In Proceedings of the 28th ACM International Conference on Multimedia, MM '20, pp. 2490-2498, New York, NY, USA, 2020a. Association for Computing doi: 10.1145/3394171.3413. 3331548.
Machinery. ISBN 4703998978
URL htttt:://ddot //di..
org/10. 1111..115//1 /355/333433 3394171- 11114441111343544
Lilang Lin, Sijie Song, Wenhan Yang, and Jiaying Liu. Ms21: Multi-task self-supervised learning for skeleton based action recognition. In Proceedings of the 28th ACM International Conference on Mullimedia, pp. 2490-2498, 2020b.
Jun Liu, Amir Shahroudy, Mauricio Perez, Gang Wang. Ling-Yo Yu Duan, and Alex C Kot. Ntu rgb+d 120: A large-scale benchmark for 3d human activity understanding. IEEE Transactions Pattern Analysis and Machine Intelligence, (2(10))2684-22)) 220-22011, 2020.
Qi Liu, Maximilian Nickel, and Douwe Kiela. Hyperbolic graph neural networks. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché Buce E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Asso- ciates, Inc., 2019. URL https: httttps://ppocceedinggg [0330005544111887201144233333333
Zhizhe Liu, Zhenfeng Zhu, Shuai Zheng, Yang Liu, Jiayu Zhou, and Yao Zhao. Margin preserv- ing self-paced contrastive learning towards domain adaptation for medical image segmentation. CoRR, abs/2103.08454, 2021. htth:t::: arxiv. org/obogg/..... wtt.////l... 221//21111212 08454
Olivier Moliner, Sangxia Huang, and Kalle Aström. Bootstrapped representation learning for skeleton-based action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 4154 4164, June 2022.
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solvingjigsaw puzzles. In ECCV, 2016.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding, 2018. htRLhttps://aatt
org/ http///ss/1100050
Giancarlo Paoletti, Jacopo Cavazza, Cigdem Beyan, and Alessio Del Bue. Unsupervised Human Action Recognition with Skeletal Graph Laplacian and Self-Supervised Viewpoints Invariance. In The 32nd British Machine Vision Conference (BMVC), 2021.
Jizong Peng, Ping Wang, Christian Desrosiers, and Marco Pedersoli. Self-paced contrastive learning for semi-supervised medical image segmentation with meta-labels. In M. Ran- zato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34. pp. 16686-16699, Curran Asso- ciates, Inc., 2021. URL http:////..prreeee....ennnniooo. pd
8b5c8441a8ff 1
12

---

Published as a conference paper at ICLR 2023
Wei Peng, Jingang Shi, Zhaoqiang Xia, and Guoying Zhao. Mix dimension in poincaré geometry for 3d skeleton- based action recognition. Proceedings of the 28th ACM International Conference Multimedia, 2020.
Haocong Rao, Shihao Xu, Xiping Hu, Jun Cheng, and Bin Hu. Augmented skeleton based con- trastive action learning with momentum lstm for unsupervised action recognition. Information Sciences, 569:90-109, 2021, doi: httpps//doi.orr//// 6/j 2021. .04.023.
Enver Sangineto, Moin Nabi, Dubravko Culibrk, and Nicu Sebe. Self paced deep learning for weakly supervised object detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41 (3):712-725, 2019.
doi: 10. 10.1109/TPAMI.20 8......2880 2804907.
Amir Shahroudy, Jun Liu, Tian-Tsong Ng, and Gang Wang. Ntu rgb+d: A large scale dataset for 3d human activity analysis. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1010-1019, 2016.
Ryohei Shimizu, YUSUKE Mukuta, and Tatsuya Harada. Hyperbolic neural networks++. In In- ternational Conference on Learning Representations, 2021. URL htttps://openreviie
net set//forum21dddscc5ttttt
Connor Shorten and Taghi M. Khoshgoftaar. A survey on image data augmentation for deep learning. Journal of Big Data, 6:60, 2019. ISSN 2196-1115. doi: https://do..or//0..1 1866 940537-019-0197-0.
Ankit Singh, Omprakash Chakraborty, Ashutosh Varshney, Rameswar Panda, Rogerio Feris, Kate Saenko, and Abir Das. Semi-supervised action recognition with temporal contrastive learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10389-10399, June 2021.
Kun Su, Xiulong Liu, and Eli Shlizerman. Predict & cluster: Unsupervised skeleton based ac- tion recognition. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9631 1641-9 2020.
Yukun Su, Guosheng Lin, and Qingyao Wu. Self-supervised 3d skeleton action representation learn- ing with motion consistency and continuity. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision (ICCV), pp. 13328-13338 October 2021.
Dídac Surís, Ruoshi Liu, and Carl Vondrick. Learning the predictability of the future. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12607-12617, 2021.
Fida Mohammad Thoker, Hazel Doughty, and Cees G. M. Snock. Ekeleton-Contraase 3D Action Representation Learning. Association for Computing Machinery, New York, NY, USA, 2021.
Minsi Wang, Bingbing Ni, and Xiaokang Yang. Learning multi-view interactional skeleton graph for action recognition. IEEE transactions on pattern analysis and machine intelligence, PP, 2020.
Ping Wang, Jizong Peng, Marco Pedersoli, Yuanfeng Zhou, Caiming Zhang, and Christian Desrosiers. Self-paced and self-consistent co-training for semi-supervised image segmentation. Medical Image Analysis, 73:102146, 2021. ISSN 1361-8415. doi: http::/ddo..rr/11.11111
media 2021.102146
Xiaoxia Wu, Ethan Dyer, and Behnam Neyshabur. When do curricula work? In International Conference Learning Representations, 2021. http//tpppp.///peeeieeeee.eeeees
id=tw4QEInpni
Jingwei Xu, Zhenbo Yu, Bingbing Ni, Jiancheng Yang, Xiaokang Yang, and Wenjun Zhang. Deep kinematics analysis for monocular 3d human pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Jiexi Yan, Lei Luo, Cheng Deng, and Heng Huang. Unsupervised hyperbolic metric learning. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12460- 12469, 2021. doi: 10. 10.1109CCVPR44477.2222 .01228
13

---

Published as a conference paper at ICLR 2023
Sijie Yan, Yuanjun Xiong, and Dahua Lin. Spatial temporal graph convolutional networks for skeleton based action recognition. In Sheila A. Mcllraith and Kilian Q. Weinberger (eds.), Pro- ceedings of the Thirty-Second AAAI Conference Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Ed- ucational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018. AAAI Press, 2018.
Siyuan Yang, Jun Liu, Shijian Lu, Meng Hwa Er, and Alex Chichung Kot. Skeleton cloud coloriza- tion for unsupervised 3d action representation learning. 2021 IEEE/CVF International Confer- ence on Computer Vision (ICCV), pp. 13403-13413, 2021.
Bing Yu, Haoteng Yin, and Zhanxing Zhu. Spatio-temporal graph convolutional networks: A deep learning framework for traffic forecasting. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18, pp. 3634-3640. International Joint Con-
ferences on Artificial Intelligence Organization. 7 2018. doi: 10.24963/iica.20188555 URL
wtw.z//...496..... sttps://dde //1 24963/ 2018
Dingwen Zhang, Deyu Meng, Long Zhao, and Junwei Han. Bridging saliency detection to weakly supervised object detection based self-paced curriculum learning. In Proceedings of the Twenty-Fift International Joint Conference on Artificial Intelligence, IJCAI' pp. 3538-3544. AAAI Press, 2016a. ISBN 9781577357704.
Richard Zhang, Phillip Isola, and Alexei Efros. Colorful image colorization. In ECCV, 2016b.
Nenggan Zheng, Jun Wen, Risheng I Liu,, Liangqu Long, Jianhua Dai, and Zhefeng Gong. Unsuper- vised representation learning with long-term dynamics for skeleton based action recognition. In AAAI, pp. 2644-2651, 2018.
A APPENDIX
Here we provide additional visualizations and analysis of the proposed model HYSP. First, we present analysis of the hyperbolic uncertainty, showing how it varies among different action classes; then, we illustrate how varies within the same action class with sample pictograms; fi- nally, we illustrate the relation between action variability and prediction error.
Inter-class variability In support of the discussion in Sec. 4.5 in the main paper, in Fig. we present the complete list of the 60 action classes from the NTU-60 dataset, ordered according to the median radius of their embeddings Note that the radius serves as a valid ranking metric for the median action uncertainty, however this work has explored the calibration of the actual uncertainty value (cf. the discussion on limitations in Sec. 5). Note in the figure how the radius ranks well the level of peculiarity and unambiguous motion in the sequence, e.g. the pictured "Pushing" action (with radius 0.9697) is easier to recognize than "Shake head" (with radius 0.7181).
In Fig. actions with smaller movements (e.g. writing, eat meal, playing with phone, or brush- ing teeth) have smallest radii, while actions with comparatively wider movements (c.g. touch-back, hopping or wearing on glasses) are almost in the middle with relatively larger radii. Finally, actions with the most peculiar and unambiguous movements, either simpler (standing up, throw) or involy- ing two people (handshake, walking apart), lie at the top of the list with large radii. Fig. 5 shows the pictograms of some selected action classes. For each action, we have selected representative sequence (with its radius close to the median of the class).
Intra-class variability We complement the above paragraph with qualitative examples of intra- class variability, i.e. how the action embedding radius (as a proxy to its uncertainty) varies for sequences within the same class.
As illustrated in Fig. 6 with reference to actions "Touch neck" (a, b, c) and "Staggering" (d, e, f), some sequences are more ambiguous (6al 6d), some little bit clearer (6 (ee and, finally, others are more specific and easier to represent by the model (6c 6f).
'hhe median is adopted (instead of the mean) hecause it is more robust estimator, less affected by outlier values of radii.
14

---

Published as a conference paper at ICLR 2023
52, puining sher parsa
15, leaging dhet passs
spart surdhing
to ving sonvthing other parss
hoong other pasan
54 rangrnudng
du watling toannss
54, point sharperan
poku
war
26, wear thse
nod headozn
store
45 nauses ar crvoriiing carettinn
stagporing
43 faiing
staddnng up ifom aiting pastion-
kickin
27 ammp
28 aut in raprao
areeawrcogg
45, to.s chest Morrachehhhhhutt
Action
20 cheer
18 wea plase
hupain
traating
wh jaanoute)
cake 47 touth racc insducts)
2 mas
11 medinn
use Dhe (wthi hand or pane ing som-
35, appeng
24 maken phone phwerphere
24, our the no togeter-
partting tovemeteeg with trgen
Hand souiin
3. broiiig
24 playing with choneeadiet
stave head
0.05
0.70
0.95
1.00
Median radius
Figure 4: Median radius per action class is an indicator of the understandability of each action class (vector graphics, please zoom-in for better readability).
drink water.
20, baticap.
shake
punching/slapping other person.
cat mealssnack
21. chter up.
36. wipe face.
brushing teeth.
22 hand waving.
37, salute.
kicking other per-
3. brushing buir drop.
23, kicking something.
38. put the pulms to- pether.
son.
24. reach into pocket.
pushing other per son.
5. pickup.
25. hopping (one foot jumping).
39, crass hands in front Isay
throw.
52, pat on person.
30. sneezotcough.
7. sitiing down.
26, jump up.
stagprring
stunding up ifrom
27. make phone
point fieger other person.
sitting powition).
coiliunswer phone.
falling.
clapping
28. plaing with
45. toach head (headache).
bugging other per son.
10, reauing.
phonetubiet.
11. writing.
29. typing on a key- board.
schachetheur poin).
44, touch chest (stom-
giving something to other person.
up paper.
30. thing with finger.
45. touch back (back- sche).
jacket.
touch other per- son's is's pocker
14. jacket.
31. tiking settle.
46. touch nocks (nock- ache).
handshiking.
off
32. check time ifrm wasch).
47. nunsen or voriiiing condition.
walking towads
17. wear on glaeses.
33. nub two hands to- gether.
48. use ftn (with puper)tfoeing wans.
each other.
off glasses.
walking apart each other.
19. put on baticap.
34. nod head/how.
15

---

Published as a conference paper at ICLR 2023
Pushing (Radius 0.9697)
(b) Hugging (Radius 0,9672)
Walking apart (Radius 0.9692)
(d) Take off hat (Radius 0.8143)
(e) Cheer (Radius 0.8091)
(l) Standing up (Radius 0.8538)
(g) Playing with phone (Radius 0.7679)
(h) Writing (Radius 0.7721)
(i) Shake head (Radius 0.7181)
Figure 5: Visualization inter-class variability through skeleton sequences with high (a,b,c), aver- age (d,c,f), and low (g.h.i) median radius.
A
(a) Touch neck (Radius 0.722)
(b) Touch neck (Radius 0.7976)
(c) Touch neck (Radius 0.9554)
(d) Staggering (Radius 0.7725) (e) Staggering (Radius 0.873)
(I) Staggering (Radius 0.9754)
Figure 6: Visualization of intra-class variability through skeleton sequences of the same class having low (a.d), average (b.e), and high (c,f) radius.
Relation between intra-class variability and prediction error We complement the study on action variability and uncertainty by relating those to the prediction error. In Fig. 7 we report the confusion matrix, whereby actions have been arranged by the median radius of the class embeddings. So, actions the rows and columns are sorted according to their median radii (increasing left- to-right and top-to-bottom), using the values from Fig. One observes that lower radii (bottom right corner) yield larger prediction errors among the actions. This is the case for actions such as writing (#11) and brushing teeth (#2), i.e. these are more ambiguous/uncertain because they characterized by less distinct motion. (The action IDs are illustrated after the confusion matrix.)
B APPENDIX
Transfer Learning protocol Following Thoker et al. (202)), 1, we additionally test models using the transfer learning protocol. We leverage the pre-trained models the NTU-60 and PKU-MMD I datasets and finetune them (pre-trained encoder and a classifier jointly) on PKU-MMD II. HYSP outperforms all competing methods in both transfers. Results are reported in Table
16

---

Published as a conference paper at ICLR 2023
39
89
50
56
58
52
34
16
47
42
Predicted Classes
23
066
19
40
84
24
21
17
65
004
18
46
10
43
27
30
13
23
11
29
3388
a = 2a Ground truth Classes
Figure 7: Confusion matrix among the actions of NTU60 (xview, single stream input, linear test protocol, top-1 accuracy) arranged by decreasing order of the median radii of their class sample embeddings (values from Fig. 4. Actions with lower radii (bottom-right corner) result in larger prediction errors. Those actions (IDs listed above) also more ambiguous and characterized by less distinct motions.
Table Results of transfer learning on PKU-MMD II, comparing the proposed HYSP against more recent techniques (performance scores other than HYSP are reported from Thoker et al. (202T). ) The evaluation protocol follows follows/Thoker al. (202T): after training an encoder on the source dataset (NTU-60 and PKU-MMD I), the pre-trained encoder and classifier are jointly fine-tuned on target dataset (PKU-MMD II) for action classification.
Transfer Learning (PKU-MMD II)
Method
PKU-MMD
NTU-60
S+P
Zheng et al.
2018
43.6
44.8
MS
L Lin et al.
2020b
44.1
45.8
ISC
Thoker el al.
2021
45.1
45.9
HYSP (ours)
50.7
46.3
Data pre-processing Following Li et al. (021). we pre-process each skeleton sequence by re- moving all invalid frames and resizing it to the length of 50 frames by linear interpolation. We use two sets of augmentations, normal and extreme. Normal augmentations include Shear (Rao et al. 2021) as the spatial and Crop (Shorten & Khoshgoftaar 2019) the temporal augmentation. Ex- treme augmentations (Guo et al. 2002a) include four spatial: Shear. Spatial Flip. Rotate, Axis Mask, two temporal: Crop, Temporal Flip, and two spatio-temporal augmentations: Gaussian Noise and
17

---

Published as a conference paper at ICLR 2023
Table 5: Top-1 accuracy varying batch size for the linear protocol. The dataset used in this table is NTU-60 xview.
Batch size 512
256
128
64
32
Top-1 Acc.
78.3
79.9
81.2
82.2
82.6
Gaussian Blur Normal augmentations are applied on the target, while extreme ones are applied on online sequences.
Sele-super Pre-training The encoder f is ST-GCN Yuu et al. 2018) with output dimension 1024. Following BYOL (Grill et al. 2020), the projector and predictor MLPs are linear layers with dimension 1024, followed by batch normalization, ReLU and final linear layer with dimension 1024. The model trained with batch size 512 and learning rate 0.2 in combination with Rie- mannianSGD (Kochurov al. 2020) optimizer with momentum 0.9 and weight decay 0.0001. For curriculum learning, across all experiments, we set ej = 50 and e2 = 100 Eq.
ife 6 i < < 62
a(c)
C-c1
(8)
if 2 02
Training on 4 Nvidia Tesla A100 GPUs takes approximately 8 hours.
Evaluation In downstream evaluation, the model is trained for 100 epochs using SGD optimizer with momentum 0.9 and weight decay 0. For the linear evaluation, the base Ir of the classifier is set 10, dropped by factor 10 epochs 60 and 80, keeping the encoder frozen. For supervise and fine-tuned evaluation, the base Ir of the encoder and classifier is set to 0.1, dropped by factor 10 at epochs 60 and 80. Exponential mapping not adopted evaluation phase.
C APPENDIX
Effect of batch size on linear protocol Here we report an interesting finding relating to the batch- size hyper-parameet when testing with the linear protocol. We find that lower batch size (32) results in a considerable performance increase with respect to a larger one (512). For the NTU-60, xview-subset, the top-1 accuracy grows from 78.3 to 82.6, resulting in performance increase of +4.3%. For this test, Table reports performances of various batch sizes.
Effect of higher/lower uncertainty samples on training We experiment compare the perfor- mance variation between using harder positives only (i.e. the samples with higher uncertainty) Vs. select
easier positives only (those samples with lower uncertainty). For the sake of comparison, an equal number of hard and easy positives, namely the first and second half of the total samples, after sorting them in decreasing order of uncertainty (as estimated upon the HYSP pre-training on all samples). It turns out that learning with the harder examples yields the top-1 accuracy of 70.9%, while the easier ones yield 74.5%. is expected that both results are well below the performance of the model trained on the full dataset (82.6% with the joint stream), because of using half the amount of data. As for the performance discrepancy, we speculate that easier samples have an edge on half-sized training set, while harder samples better support when larger amount of data is available. Let us also add that, as expected, the two new trainings have different durations: that one with harder samples converges and overfits earlier (at epochs 400 and 500 respectively) than with casier (epochs 400 and 700 correspondingly).
D APPENDIX
A Additional insights into the self-paced learning In Fig. we provide further insights into Eq the Riemannian gradient of the Poincaré loss, by illustrating the relation between the gradient magni- tude and the uncertainty of the target-view embedding [[]l]).. We consider the model checkpoint
18

---

Published as a conference paper at ICLR 2023
Gradlent nomm
0.065 0.085 0.105
0.125
0.15
0.17
0.19
0.21
0.23
0.25
Uncertanty: [1 radius)
Figure 8: Bar plot of the average Riemannian gradient norm for specific intervals of uncertainty. The plot experimentally explains the intuition behind Eq. described Sec 3.2.1. The gradient norm and prediction uncertainty after the pre-text task are highly correlated. Learning them is self-puced, because lower the uncertainty of h and the stronger is the gradient.
for epoch 100, which belongs to the second (the main) stage of training (the analysis of other epochs has produced similar statistics). Then we compute aggregate histogram statistics across all the train- ing samples. The plot of Fig. [ confirms the analysis of Sec. [32.1 the lower the uncertainty of the target view embedding h (left side of the plot), the larger the gradient. As result, learning is self- paced because the importance of each sample weighted, via the respective gradient magnitude, according to the level of certainty of the target-view embedding.
19


Reviews:
[{"review_id": "wJM50wSE2r", "reviewer": "Reviewer_Dsnu", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors present a method to perform skeleton-based action recognition using a novel self-paced learning on features in a hyperbolic embedding space. The learning is \"self-paced\" in the sense that the magnitude of the gradients for the learning optimization directly depends on the certainty of the learned features. The authors follow a curriculum scheduling approach to stabilize their self-paced model. They also propose a positive-only learning paradigm, showing that adding negative samples into the hyperbolic embedding space deteriorates performance. Overall, the authors show appreciable improvements in action recognition performance using their hyperbolic self-paced model and also demonstrate the benefits of the individual components of their method through ablation studies.", "summary_of_the_review": "The paper presents a sound and intuitive method for skeleton-based action recognition. The contributions are clear, and the experimental results sufficiently back up the authors' claims.", "strength_and_weaknesses": "**Strengths**\n1. The paper is clearly written and easy to follow. I was not familiar with prior work around hyperbolic embedding spaces and self-paced learning, but I found the concepts intuitive and the descriptions sound.\n\n2. The experimental results clearly show the contributions of the individual components of the proposed method. The authors back up all their claims around curriculum scheduling, positive-only sampling, and uncertainty in self-paced learning with appropriate experiments.\n\n3. I also found the limitations sections helpful to clearly understand the scope of the work.\n\n\n**Weaknesses**\n1. While I did not find any major concerns with the work, I am curious how the choice and the quality of positive samples affect performance. For example, how much does the performance depend on \"hard positives\"? Fig. 3a shows the correlation between the feature distances and the uncertainties. If only samples corresponding to the last few columns were to be used in the training (which I'm assuming are the hardest samples), how much would the performance drop? Since humans can perform similar actions with many diverse movements, real-world data can contain many such hard positives.\n\n2. In addition to showing the inter-class variability (Sec. 4.4 and appendix), it can be illuminating to also see the confusion matrix, showing how the variability might be correlated with the per-class performance.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "gUT08a1H4-6", "reviewer": "Reviewer_R1Eg", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "- The paper explores hyperbolic self paced learning for self supervised learning of action representations. \n- The key idea is to use hyperbolic geometry to automatically order the training samples from easy to hard. This determines the learning pace thus improving learning. \n- The framework is borrowed from BYOL, which does not need negatives. \n- The approach shows promising performance on the tasks considered. ", "summary_of_the_review": "The paper proposes a new technique for SSL on skeleton sequences. The improvements are decent, though not consistent on all tasks. I have some concerns on baselines and intuitions and hence my current score. ", "strength_and_weaknesses": "Strengths:\n- The paper is well written and easy to follow. The intuitions are clear for the most part. \n- I like the idea of self-pacing for self-supervised learning. The paper proposes a novel approach based on hyperbolic geometry to tackle this. \n- The paper combines ideas from four different areas. The combination is a novel contribution to the field. \n- The approach leads to good gains on some of the tasks considered.  \n\nWeaknesses:\n- While the approach shows good improvements on semi-supervised learning, these do not hold for the other settings. The comparison for finetuning would have been easier if there were numbers reported for ISC. \n- It is not clear from the paper why hyperbolic specifically. The Section 3.2, directly starts with explaining how a hyperbolic extension to SkeletonCLR would work. As far as I understand, the hyperbolic geometry is useful to encode hierarchies, tree-like structures. It is not clear to me why the problem being tackled by this paper needs that. \n- Useful intuition to clarify : What does it mean for a target to be uncertain. I found an exaplanation to this in caption for Figure 3. I suggest you include this earlier in the text to ease understanding. \n- A non-hyperbolic self-paced learning baseline : From Fig 3 caption \".. Uncertainty indicates the difficulty in classifying specific actions if they are characterized by less or more peculiar..\". Is it possible to weigh the usual contrastive learning loss using the distance between positives as a baseline to determine uncertainty? \n- Given that your approach is based on a different framework compared to state-of-the-art, you should include results of a non-hyperbolic baseline with BYOL. This will help determine how much of the improvement can be obtained using BYOL based baseline vs MoCo,  For example, for Image SSL tasks, BYOL is generally better than MoCo-v2. \n- Drop in performance with negatives : While the argument for why negatives cannot be generated in the h space seems convincing, what happens if you generate negatives/apply contrastive learning loss for this experiment on the z space (after projection g) ? \n- Suggestion : I think in addition to linear evaluation, you should evaluate your approach on transfer learning as well. For skeleton-based SSL, NTU -> PKU I/II is a good option. \n- The paper includes details on training time. It would be helpful to include details on how much extra compute/traning time is needed due to the use of Riemannian gradient descent. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "D0RmEshvty", "reviewer": "Reviewer_GGZs", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "While most works in self-supervised skeleton-based action recognition have used MoCo for modeling, this paper makes a case for using the positives-only design of BYOL instead. The paper proposes the HYperbolic Self-Paced model (HYSP) according to the principle that more certain samples should drive the learning process more predominantly. The authors empirically demonstrate that the proposed approach achieves superior performance on public datasets.", "summary_of_the_review": "The paper provides a new framework to model self-supervised skeleton-based action recognition. The paper provides extensive experiments for that and interesting observations and analyses showing the advantages of the proposed method. There are a few potential issues detailed in the previous section. Considering all the strengths and weaknesses pointed out in the Main review, I give the current rating.", "strength_and_weaknesses": "- Strength:\n    - The paper is well written and the problem being tackled is clearly described.\n    - The introduction of hyperbolic mapping and curriculum learning brings a different perspective to self-supervised skeleton action recognition.\n    - The paper has a detailed theoretical basis and the detailed visible analysis (Figure 2,4,5) further makes the idea more convincing.\n\n- Weakness:\n    - Parts of the approach do exist in prior works. The authors should further elaborate whether the HYSP is a combination of several methods (BYOL + Ganea et al. 2018 + Guo et al. 2022). However, I think that the paper presents a good study on some of the practices and makes a compelling case for using BYOL for the task.\n    - HYSP is not the first method to use the positive-only BYOL and the performance of HYSP is not that competitive. *''Moliner et al. Bootstrapped Representation Learning for Skeleton-Based Action Recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2022, pp. 4154-4164.''*\n    - Although mentioned in the conclusion, I still think that HYSP is more like a general improvement of the self-supervised framework for BYOL combined with hyperbolic space. Measurements on image classification tasks may be more convincing and necessary. I have not seen a unique improvement and analysis for the field of skeleton action recognition.\n\n- Other questions:\n    - Double check the last name and first name of the author of the cited paper, some are confused, such as \"Tianyu et al.\" should be \"Guo et al.\".\n    - Please explain Eq.5 in more detail. The conclusion behind Eq.5 cannot be intuitively understood from Eq.5.\n    - In Table 1, SkeletonCLR also uses negative samples.\n    - Does \"w/o hyper.\" in Table 3 mean using BYOL directly? If yes, I think the accuracy is unsatisfactory (see weekness2). If not, please explain in more detail and give the result of using BYOL directly.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "CL7_mkdUjpr", "reviewer": "Reviewer_V8KY", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a self-supervised skeleton representation learning method based on the BYOL and hyperbolic space. The experimental results show that the proposed HYSP model achieves state-of-the-art performances on three public datasets. ", "summary_of_the_review": "As mentioned in the 'Strength And Weaknesses', both the hyperbolic space and BYOL are not new in skeleton action recognition. So I tend to give the 'marginally below the acceptance threshold'.", "strength_and_weaknesses": "Pros:\n\n(1) This is the first self-supervised skeleton representation work on hyperbolic space.\n\n(2) This paper is well written. \n\n\nCons:\n\n(1) The proposal seems to be incremental based on prior works. HYSP is not the first method that uses the hyperbolic space in skeleton action recognition [1*] and uses the BYOL for self-supervised skeleton representation learning [2*]. \n\n(2) Why only BYOL?: In my opinion, a better way to show the hyperbolic space is useful is to combine the hyperbolic space with multiple self-supervised learning baselines. From the results in Table 3 (w/ neg.), the hyperbolic space seems not to work well on MoCo. \n\n(3)  The linear evaluation results are not as good as SOTA methods. And the transfer learning results are missing. These two evaluation protocols are more important than the semi-supervised and supervised settings, regarding showing the effectiveness of self-supervised methods, in my opinion.\n\n(4) Table 3: the BYOL-only result (w/o hyper.) may be not reasonable. \n\n\n  \n\n[1*] Mix Dimension in Poincar\\'{e} Geometry for 3D Skeleton-based Action Recognition. ACM MM 2020. \n\n[2*] Bootstrapped Representation Learning for Skeleton-Based Action Recognition. CVPRW 2022. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster