Graph Contrastive Learning for Skeleton-based Action Recognition
Authors: Xiaohu Huang
Hao Zhou
Jian Wang
Haocheng Feng
Junyu Han
Errui Ding
Jingdong Wang
Xinggang Wang
Wenyu Liu
Bin Feng
Venue: ICLR 2023 poster (2023)
Keywords: Skeleton-based Action Recognition

Abstract: In the field of skeleton-based action recognition, current top-performing graph convolutional networks (GCNs) exploit intra-sequence context to construct adaptive graphs for feature aggregation. However, we argue that such context is still $\textit{local}$ since the rich cross-sequence relations have not been explicitly investigated. In this paper, we propose a graph contrastive learning framework for skeleton-based action recognition ($\textit{SkeletonGCL}$) to explore the $\textit{global}$ context across all sequences. In specific, SkeletonGCL associates graph learning across sequences by enforcing graphs to be class-discriminative, i.e., intra-class compact and inter-class dispersed, which improves the GCN capacity to distinguish various action patterns. Besides, two memory banks are designed to enrich cross-sequence context from two complementary levels, i.e., instance and semantic levels, enabling graph contrastive learning in multiple context scales. Consequently, SkeletonGCL establishes a new training paradigm, and it can be seamlessly incorporated into current GCNs. Without loss of generality, we combine SkeletonGCL with three GCNs (2S-ACGN, CTR-GCN, and InfoGCN), and achieve consistent improvements on NTU60, NTU120, and NW-UCLA benchmarks. 

Published as a conference paper at ICLR 2023
GRAPH CONTRASTIVE LEARNING FOR SKELETON- BASED ACTION RECOGNITION
Xiaohu Huang 1. Hao Zhou
Jian Wang? 2
Haocheng Feng"
Junyu Han Bin Feng'
Errui Ding? 2 Jingdong Wang
Xinggang Wang
Wenyu Liu
School of EIC, Huazhong University of Science & Technology
Department of Computer Vision Technology (VIS), Baidu Inc., China
Ihuangxiachu, xgwang, 1iuwy, Fengbin) edu.
{ [zhouhaolg 44, wangjian jian33, fenghaocheng, hanjunyuJebaidu. (dingerrui, wang jangjingdong @ba du. com
ABSTRACT
In the field of skeleton-based action recognition, current top-performing graph convolutional networks (GCNs) exploit intra-sequence context to construct adap- tive graphs for feature aggregation. However, we argue that such context is still local since the rich cross-sequence relations have not been explicitly investigated. In this paper, we propose a graph contrastive learning framework for skeleton- based action recognition (SkeletonGCL) to explore the global context across all sequences. In specific, SkeletonGCL associates graph learning across sequences by enforcing graphs to be dlass-discrmminative, i.e., intra-class compact and inter- class dispersed, which improves the GCN capacity to distinguish various action patterns. Besides, two memory banks are designed enrich cross-sequence con- text from two complementary levels, ie., instance and semantic levels, enabling graph contrastive learning in multiple context scales. Consequently, SkeletonGCL. establishes new training paradigm, and it can be seamlessly incorporated into current GCNs. Without loss of generality, we combine SkeletonGCL with three GCNs (2S-ACGN, CTR-GCN, and InfoGCN), and achieve consistent improve- ments on NTU60, NTU120, and NW-UCLA benchmarks. The source code will be available at https: //gpt//gttttt//// com/ /lliverlivxxeieeeeeeeo eeleeenonC..
INTRODUCTION
Graph convolutional networks (GCNs) have been widely applied in skeleton-based action recogni- tion since they can naturally process non-grid skeleton sequences. For GCN-based methods, how to effectively learn the graphs remains core and challenging problem. In particular, ST-GCN (Yan et al., 2018) is a milestone work, using pre-defined graphs to extract the action patterns. However, the pre-defined graphs only enable each joint access the fixed neighboring joints but hard to capture long-range dependency adaptively. Therefore, a mainstream 2019; Shi et al., 2019; Zhang et al., 2020b;a; Ye et al., 2020; Chen et al., 2021b; Chi et al., 2022) take efforts to solve this issue by generating adaptive graphs. The adaptive graphs can dynamically aggregate the features within each sequence and thus show significant advantages in performance comparison.
subsequent works (Li et al.,
Generally, adaptive graphs are constructed by using intra-sequence context. However, such context will still be "local" when viewing the cross-sequence information as an available context. Therefore, we wonder: Is it possible involve the cross-sequence context graph learning? To find the answer, in Fig. 1, we visualize the adaptive graphs leamned from sequences of two easily confused classes ("point to something" and "take selfie"). The graphs are learned by a strong GCN, i.e.
Work done when Xiaohu Huang was an intern at Baidu VIS.
"ooresponning authors.

---

Published as a conference paper at ICLR 2023
Label: Paint to something Predict: Point something
Label: Take selfie Predict: Take
Label: Point to something Predict: selfie
(b)
(c)
Figure I: Graph visualization of sequences from two easily confused classes ("point to some- thing" and "take selfie"). The graphs are learned by CTR-GCN (Chen et al., 2021b). We take the tip of the hand that does the action as the anchor. The size of the red circles and the width of the blue lines both denote the strengths of connections between joints. For simplicity, only representative frames are visualized. (a) Three sequences from class "point to something" are correctly classified, where the graphs contain connections to the body joints. (b) Three sequences from class "take selfie" are correctly classified, where the graphs highly emphasize the connections to the hands, while the connections to the body are suppressed. (c) A sequence from class "point something" is misclassified as "take selfic", whose graph resembles the graphs in (b), but is dissimilar from graphs in (a). Hence, we realize that the class-ambiguous graph representations would make nega- tive impacts on recognition performance.
CTR-GCN (Chen et al., 2021b). From the visualization, we find that (1) For sequences that are correctly classified in Fig. 1 (a) and Fig. (b), the learned graphs in the same class look similar, while graphs in different classes have distinct differences. (2) For a misclassified sequence in Fig. I (c), the learned graph resembles the graphs from the misclassified class more than those from the ground truth class. These observations remind us that graph learning in current adaptive GCNs can implicitly learn class-specife graph representations some extent. But without explicit constraints, it leads to class-ambiguous representations in some cases, thereby affecting the GCN capacity to discriminate classes (in Tab. of Sec. 4.4, provide quantitative results to further support our hy- pothesis). Therefore, we speculate that if the cross- rros-sequence semantic relations are incorporated as guidance in graph learning. the class-ambies ambiguuty issue will be alleviated and the graph representations will better express individual characteristics of actions.
In recent years, contrastive learning has achieved great success in self-supervised representation learning (He et al., 2020; Chen et al., 2020; 2021a). It studies cross-sample relations in the dataset. The essence of contrastive learning "comparing", which pulls together the feature embedding from positive pairs and pushes away the feature embedding from negative pairs.
Based on the analysis above and the advances contrastive learning, we propose a graph contrastive learning framework for skeleton-basee action recognition in the fully-supervised setting, dubbed SkeletonGCL. Instead just using the local information within each sequence. SkeletonGCL ex- plores the cross-sequence global context to improve graph learning. The core idea is to pull together the learned graphs from the same class while pushing away the learned graphs from different classes. Since graphs reveal the action patterns of sequences, enforcing graph consistency in the same class and inconsistency among different classes helps the model understand various motion modes. In addition, enrich the cross-sequence context, we build memory banks store the graphs from historical sequences. In specific, an instance-level memory bank stores the sequence-wiss graphs, which hold the individual properties of each sequence. In contrast, semantic-level memory bank stores the aggregation of graphs from each class, which contains the class-level representation. The two banks are complementary to each other, enabling us to leverage more samples. Skele- tonGCL can be seamlessly combined with existing GCNs. Eventually, we combine SKeletonGCI. with three GCNs (2S-AGCN (Shi et al., 2019), CTR-GCN (Chen et al., 202 1b), and InfoGCN (Chi et al., 2022)), and conduct experiments on three popular datasets (NTU60 (Shahroudy et al., 2016), NTU120 (Liu et al., 2019) and NW-UCLA (Wang et al., 2014)). SkeletonGCL achieves consistent improvements with these models using different testing protocols (single- modal multi-modal) on all three datasets, which widely demonstrates the effectiveness of our design. Notably, SkeletonGCL only introduces small amount of training consumption but has no impact at the test stage.
2

---

Published as a conference paper at ICLR 2023
Though there exist some works that apply contrastive learning in skeleton- action recognition (Li et al., 2021; Guo et al., 2022; Mao et al., 2022), our method differs from them as follows: (1) The previous methods took pooled feature vectors to conduct contrastive learning et al., 2020; Chen et al., 2020), where the structural properties in skeletons are lost. In contrast, SkeletonGCI. uses graphs contrast, which maintains the structure details of skeletons and offers high-order connection information between joints. (2) The previous methods used memory banks to store instance-level representations only. Differently, our memory banks store both instance-level and semantic-level representations, allowing to leverage context from individual sequences and class-specific aggregations, which are complementary each other. (3) The previous methods were used in the pre-training stage, while SkeletonGCL. is incorporated into the fully-supervised setting without extra pre-training cost.
in (He
Summarily, the contribution of this paper can be concluded as follows:
We present a new perspective for graph learning of GCN models in skeleton-base action recognition. In specific, we propose to make use of the cross-sequence context to guide graph learning, whose goal is to enforce graphs to be intra-class compact and inter-class dispersed.
Motivated by the advances in contrastive learning, we smoothly combine the ideas of con- trastive learning and cross-sequence graph learning together, then propose new training paradigm for skeleton-bare bbase action recognition, called SkletonGCL. SkeletonGCL incor- porates an instance-level and semantic-level memory bank to enrich the cross-sequence context comprehensively. Besides, it can be seamlessly incorporated into current GCNs.
SkeletonGCL achieves consistent improvements combined with three GCNs (2S-AGCN, CTR-GCN, and InfoGCN) on three popular benchmarks (NTU60, NTU120, NW-UCLA) using both single-modal and multi-modal testing protocols. In addition, SkeletonGCL is training-efffciinn and has impact at the test stage.
2 RELATED WORKS
2.1 SKELETON-BASED ACTION RECOGNITION
Skeleton- based action recognition is to classify actions from sequences of estimated key points. The early deep-learning methods applied convolution neural networks (CNNs) (Chéron et al., 2015; Liu al. 2017b) or recurrent neural networks (RNNs) (Du et al., 2015; Lev et al., 2016; Wang & Wang, 2017; Liu et al., 2017a) to model the skeletons, but they could not explicitly explore the topolog- ical structure of skeletons, thus the performances were limited. Recently, PoseC3D (Duan et al., 2022) revisited the CNN-based method by stacking the heatmaps as 3D volumes, which maintained the spatial-temporal properties of skeletons and obtained marginal performance improvements. In the past few years, the mainstream works in skeleton-basee els. ST-GCN (Yan et al., 2018) was the first work that adopted GCN as the feature extractor, which heuristically designed fixed graphs to model the skeletons. The follow-up methods proposed spatial- temporal graphs (Liu et al., 2020), multi-scale graph convolutions (Chen et al., 2021c), channel- decoupled graphs (Chen et al., 2021b; Cheng al., 2020a) and adaptive graphs (Li al., 2019; Shi et al., 2019; Ye et al., 2020; Zhang et al., 2020b; Chen et al., 2021b; Chi et al., 2022) to improve the capacity of GCNs. Tracking the development of GCN-based methods, we find that graph learning has always been core problem and now the adaptive GCNs are leading since they can model the intrinsic topology of skeletons.
action recognition were GCN mod-
However, current adaptive GCNs generated the graphs based on the local context within each se- quence, where the cross-sequence relations have been neglected. In contrast, we propose to explore the cross-sequence global context to shape graph representations. In this way, the learned graphs can not only describe the individual characteristics within each sequence but also emphasize the similarity and dissimilarity of motion patterns across sequences.
2.2 CONTRASTIVE LEARNING
In recent years, numerous representation learning methods (Wu et al., 2018; Oord et al., 2018; He al., 2020; Chen et al., 2020; Wang et al., 2021) with contrastive learning have emerged, especially
3

---

Published as a conference paper at ICLR 2023
in self-supervised representation learning. The key idea is to pull together the positive pairs and push away the negative pairs in the feature space. Generally, the features are vectors obtained from feature extractors followed by pooling layer. A standard approach to obtaining the positive pairs is augmenting an original sample into two different views. The negative samples are selected randomly or using hard mining strategies (Khosla al., 2020; Robinson et al., 2020; Kalantidis et al., 2020). To increase the capacity of negative samples, the memory bank mechanism was devised in (He et al., 2020; Misra & Maaten, 2020) store more negative instances. By contrasting positive pairs against negative pairs, the model can learn to focus on semantic representations.
In the field of ikeleton-based action recognition. prior works (Li et al., 2021; Mao et al., 2022; Guo et al, 2022) proposed to apply contrastive learning in the pre- pre-training stage by roughly following the frameworks mentioned above. CrossCLR (Li al., 2021) mined positive pairs in the data space and explored the cross-modal distribution relationships. Further, CMD (Mao et al., 2022) transferred the cross-modal knowledge in a distillation manner. And AimCLR (Guo et al., 2022) used extreme augmentations to improve the representation universality.
Compared with the above methods, we use graph representations to contrast instead of using pooled feature vectors. Meanwhile, we establish two different memory banks at complementary levels, i.e. instance and semantic levels, to enrich the context scales. Besides, the proposed method is used with GCNs under the fully- supervvuee setting, which requires pre-training procedure.
METHOD
3.1 PRELIMINARY
We denote human skeleton as vertex set V {ui, 02...., UN }. where N denotes the number of vertices. For each vertex U, the feature dimension is set as C. Hence, a skeleton sequence with T frames can be denoted X E R Graph topology is used to represent the correlations between joints, formulated as g.
GCNs in Skeleton-Based Action Recognition. Generally, GCN models alternatively apply graph convolutions and temporal convolutions to extract the spatial configuration and motion pattern of skeletons, respectively. The graph g is vital for graph convolutions since it determines the message passing among joints. In current adaptive GCNs, is learned within each sequence and has different sizes, e.g., g E R x in 2S-AGCN (Shi et al., 2019) and g R Rxxxx NxN in -CN (Chen et al., 2021b). The Ks denotes the number of sub-graphs, normally set as 3. In general, the graph convolution defined as:
K
Xs
Σ
g
xw S:
where Xs. E RTxNXCC denotes the spatial extracted feature with C' channels, and Ws Kgx denotes the spatial feature transformation filters. Next, temporal convolutions are ap- plied on Xs, producing motion extracted feature XT E RT'NxC', After stacking layers of graph convolutions and temporal convolutions, global average pooling (GAP) layer summarizes the global features, then classification head (fully-connected layer) followed by a Softmax activa- tion function is applied to obtain the class prediction ý RC,, where Cg denotes the number of classes. Finally, a cross-entropy loss Les supervises the class prediction with the ground truth label y follows:
LCE -
Σ log
Self-Supervised Contrastive Learning. In the context of self-supervised contrastive learning, each input sample is processed by data augmentations produce positive pair: and I+_ Through feature extraction network, and /- are transformed into feature vectors f E RD and f+ E R As for the negative samples, they are selected from the dataset excluding and I+, represented as negative set ~ Each negative in N is denoted as f- E DR The similarity between two feature vectors is calculated as sim(ft f f+f-
InfoNCE (Gutmann & Hyvärinen, 2010; Oord
4

---

Published as a conference paper at ICLR 2023
Classffication Head
CromsEatrapy Loss Lert
Input
Inslance-low lovel
M Nemory Bank Mire
GCN Encoder
Enqueen
Sample
Positive
Conteaslive Loss N
Negalive
C.
Projection
Anchor
Contrastive
Nogattre
Mazsantaa Update
Average
Flatton
PC
Posttive
Semantiie-enss
K.N
M Memory Bank Mom
Figure 2: Overview of SkeletonGCI. An input skeleton sequence T is fed into GCN encoder, producing a feature vector f for classification and a learned graph for graph contrastive learning. The graph g is embedded into vector by a projection head. And two memory banks are built to store the embedded graphs. From the memory banks, we sample the positives and negatives according to the labels, then perform contrastive loss. The memory banks only used in the training stage but will be removed during the testing stage.
al., 2018) widely adopted for contrastive learning, which is formulated as:
fim(f,ft)/T LNCE log ff Er-eN- fim(f,f-)
(3)
where temperature
0 is hyper-parameter
3.2 GRAPH CONTRASTIVE LEARNING
The proposed SkeletonGCL is illustrated in Fig. 2. The framework consists of two branches, where the classification branch is parallel to the graph contrast branch. Taking a skeleton sequence / as input, the GCN encoder outputs feature vector f for classification and graph g for graph contrast.
Graph Projection Head. In order to contrast the graphs graphs into vectors by graph projection head. The projection heads for different GCNs are similar (see App. for details). In Fig. 2, taking the graph g E & learned in CTR-GCA (Chen et al., 2021b) as example, we first squeeze g along the channel dimension by an average pooling layer into R K Then,
common feature space, we embed the
flatten graph into 1D vector as g' E IR N2 and project g' into a vector E RC, by an FC layer We E N*K,N"XC, Since different channels in WG are specific to different vertices in the graph, the graph projection is vertex-aware and thus can encode the structures of skeletons. Afterward, we update two memory banks with v. The memory banks are illustrated in Fig, 2, and detailed next.
Memory Bank. enrich the cross-sequence context, we build memory banks to store the cross- batch graphs. In specific, two memory banks are constructed, i.e., an instance-level memory bank Mins x and a semantic-level memory bank Msen E CR xC, P denotes the number of instances stored for each class in Miss Particularly, each element in Mins denotes graph instance from a class. In contrast, each element Msem denotes the graph aggregation of a class. Therefore, the two memory banks bank emphasizes the instance discrimination each sequence, while the semantic-level memory bank covers the class properties across sequences.
complementary levels, where the instance-level memory
We update Mins in a first-in-first-out manner, which maintains the number of instances for each class P. As for Msen- we momentum update strategy, which integrates the graphs of the same class from the current timestamp and all previous timestamps, regarded as a long-term representation. The momentum update is defined follows:
m sem +- am Sem + (1 a)v,
where m.f.. is the representation for class c., C, is the class label for the input and 0 <<a<< is hyper-parameter.
5

---

Published as a conference paper at ICLR 2023
Loss. To achieve the graph contrast, we sample positives and negatives from the memory banks Mins and Msem- For Mins- vector v is set as the anchor, hence samples in the positive set Mis are with label ca, and samples in the negative set Nis are with different labels. Consequently, the InfoNCE loss in Eq. can be rewritten as:
sim sim (on(vvvt)) V/T
Σ
log
(5)
sim(V, vt) / Ev eN sima(v v-
VtENE
sim (v, vv,vv,
NCC
Σ log
sim(v, vtT Ev- sim(vv 3/T
(6)
V V-N
CNCE leverages multiple positives compared with Eq. 3 by using label information, which mines
more semantic-related samples. Similarly, we can define the InfoNCE loss NCE. which is specific for the memory bank Msem Summarily, the overall contrastive loss is written as follows:
LNCF L'CCE
ANCE
Sem
And the overall loss function is defined as follows:
L LSCE + LCE-
(8)
Hard Sampling. As the training continues, most samples become too easy, which contribute less to the training. Therefore, methods in (Tabassum et al., 2022; Robinson et al., 2020; Kalantidis et al., 2020; Wang et al., 2021) are proposed to use hard mining strategies to focus on informative samples. In this paper, considering the massive number of instances in Minss contrasting with all these instances naturally leads to redundancy and hinders the training. To alleviate this issue, we propose mine hard examples in M Specifically, as a criterion to evaluate hardness. Harder positives are with lower similarities, and harder negatives with higher similarities. In total, for Minss we select Ki hardest positive examples, K3 hardest negative examples, and Kg random negative examples.
take the similarity calculation sim(v, v')
4 EXPERIMENTS
4.1 DATASETS
NTU RGB+D. NTU RGB+D (NTU60) (Shahroudy et al., 2016) is large-scale skeleton-based action recognition dataset, which contains 60 action classes and 56,880 sequences. Each sequence is annotated as skeletons with 25 joints. All the sequences are performed by 40 subjects and filmed protocols
by 3 Kinect cameras from three different views. Generally, performances: (1) cross-subject (X-Sub): train data are performed by 20 subjects, and test data are performed by other 20 subjects. (2) cross-view (X-View): train data from cameras 2 and 3, and test data from camera 1.
used to evaluate the
NTU RGB+D 120. NTU RGB+D 120 (NTU120) (Liu al., 2019) is extension of NTU RGB+D dataset, which newly includes 57,367 skeletons of 60 extra classes. All the sequences are performed by 106 subjects and filmed by three cameras from three different views. In addition. NTU RGB+D 120 has 32 setups, where each denotes unique location. Generally, two protocols are used to evaluate the performances: (1) cross-subject (X-Sub): train data are performed by 53 subjects, and test data are performed by other 53 subjects. (2) cross-setup (X-Set): train data are samples with even setup IDs, and test data are samples with odd setup IDs.
Srrthwestern-UCL.. Northwestern-UCLA (NW-UCLA) dataset (Wang et al., 2014) contains 1494 sequences from 10 action classes. Each sequence is annotated as skeletons with 20 joints. All sequences performed by 10 subjects and filmed by three Kinect cameras from different views. We follow the official evaluation protocol: train data are captured by the first two cameras, and test data are captured by the third camera.
4.2 IMPLEMENTATION DETAILS
To thoroughly validate SkeletonGCL, we take three GCNs (2S-AGCN (Shi et al., 2019), CTR-GCN (Chen et al., 2021b), and InfoGCN (Chi et al., 2022)) baseline models. For CTR-GCN and

---

Published as a conference paper at ICLR 2023
Table 1: Top-1 accuracy comparison (%) with the state-of-ther the- methods on NTU 60 and NTU120 datasets. The numbers in gray indicate the results reported in their papers. indicates that we retrain the models using their officially released code. Particularly, 2S-AGCN retrained using stronger train recipe from CTR-GCN.
NTU
NTU 120
Setting
X-Sub
X- X-View
X-Sub
X-Set
Methou/Moodality
J+B
45
J+B
45
B
J+B
J+B
45
SGCN (Zhang at, 2020g)
89.0
94.5
79.2
81.5
ST-TR- GCN (Plizzari et al., 2021)
89.2
90.3
95.8
96.3
82.7
85.
85.0
87.1
Shift-GCN [Cheng at, 2020b)
87.8
89.7
90.7
95.1
96.0
96.5
80.9
85.9
83.2
86.6
87.6
DC. GCN-ADG (Cheng et, 2020s)
90.8
96.6
86.5
88.1
Dynamic GCN (Ye et 2020)
91.5
96.0
MS-G3D (Liu al., 2020)
89.4
90.1
91.5
95.0
95.3
96.2
86.9
88.4
MST-GCN (Chen al., 2021e)
89.0
89.5
91.1
91.5
95.1
95.2
96..4
96.6
822.8
84.88
187.0
18.55
84.5
86.3
88.3
88.8
2S-AGCN (Slá et 2019)
858.5
93.7
93.2
95.1
28.AGCN* (Shi al., 2019)
88.9
89.2
91.0
91.5
94.5
94.1
95.7
9.5.9
84.0
85.1
187.8
88.2
85.3
86.3
89.0
89.6
25-AGCN* WSKeletonGCI.
89.9
90.0
92.2
95.0
94.4
96.1
96.4
844.7
86.0
888.4
188.7
86.1
86.88
89.77
90.2
CTR-GCN (Chen al., 2021h)
92.4
8
8:5.7
8.899
187.5
[90.
90.6
CTR-GON* (Chon 202
89.8
90.2
92.4
94.8
94.8
96.3
966.8
84.9
85.7
88.77
88.9
86.7
87.5
90.1
90.5
CTR-GCN' w/SkeletonCCL
90.8
91.1
92.6
93.1
95.3
95.4
96.6
97.00
85.6
86.99
89.22
89.5
87.3
888.2
90.5
91.0
IntoCCN 2022)
89.8
92.7
56.55
87.35
199.4
86.3
90.7
InfoGCN (Chi at., 2022)
89.4
90.6
91.3
92.3
95.2
95.4
96.2
96.7
84.2
86.9
898.2
89.2
86.3
88.5
89.4
90.7
InfoGCN* w/SkeletonGCL
90.1
91.0
91.9
92.8
95.5
95.7
96.6
97.1
85.2
87.4
88.8
89.8
87.2
88.7
90.0
91.2
InfoGCN, we follow their training recipes. Particularly, for 2S-AGCN, since its training recipe is of date, we borrow the training recipe from CTR-GCN, which effectively improves its baseline performance, P. the number of stored instances for each class in Mins is set as 684 on NTU60 and NTU120, and 342 on NW-UCLA. The dimension of graph vector Cg is set to 256. For all datasets, the number of sampling examples KH KH and Kg are set as 128, 512, and 512, respectively. For different models used in different modalities, we experiment with temperature t of 0.5, 0.8, 1.0, and 1.5, and choose the best one. The hyper-parametts a for momentum updating is set as 0.85. Besides, we fix the random seed to ensure experiment reproducibility. All experiments are conducted using single NVIDIA V100 GPU.
4.3 COMPARED WITH THE STATE-OF-THE-ART
In this section, we combine our method with three GCNs, and compare them with the state-of-the- art (SoTA) methods. In Tab. and Tab. 2, we list current SoTA methods in skeleton-basee action recognition except PoseC3D (Duan al., 2022). PoseC3D is promising CNN-based method, but it uses non-official skeleton data and applies a multi-crop test protocol (GCN methods typically use one crop), which are unfair for comparison here. In evaluation, four modalities used: "joint stream" (J) denotes the joint coordinates, "bone stream" (B) denotes the coordinate difference be- tween spatially connected joints, "joint motion" (J-M) denotes the coordinate difference between temporally adjacent frames, and "bone motion" (B-M) denotes the bone difference between tem- porally adjacent frames. The 4-stream ensemble (4S) denotes using the four modalities together. Following the widely-adopted protocol, we evaluate models using J, B, J + B, and 4S modalities.
NTU60 and NTU120. Tab. lists the results on NTU60 and NTU120. From the results, we find that: (1) Combined with SkeletonGCI, all three baseline models achieve solid improvements on these two benchmarks over different settings and modalities. Taking the J modality on NTU60 X- Sub as example, 2S-AGCN improves by 1.0% (88.9% to 89.9%), CTR-GCN improves by 1.0% (89.8% to 90.8 %), and InfoGCN improves by 0.7% (89.4% to 90.1%). Considering NTU60 is dataset, such improvements are quite hard. (2) With SkeletonGCL. CTR-GCN and InfoGCN can set new SoTA performance.
an
NW-UCLA. Tab. 2 lists the results NW-UCLA. SkeletonGCL can still achieve consistent im- provements based on the three models. And new state-of-the-an performances are achieved by combining SkeletonGCI. with CTR-GCN and InfoGCN

---

Published as a conference paper at ICLR 2023
Top-1 accuracy Comparison (%) with the state-of-the-art methods on the NW-UCLA dataset. Numbers in gray denote the results reported in their papers. indicates that we retrain the models using their officially released codes. Partieularly, 28-AGCN is retrained using stronger train recipe from CTR-GCN.
Table 2:
Dataset
NW-UCLA
Mechod/Modality
B
J+B
45
AGC-LSTM (Si et al,, 2019)
DC-GCN+ADG (Cheug et 2020n)
95.3
ShantGGN (Cheng al, 20206)
92.5
94.2
94.6
2S-AGCN (Sli al, 2019)
28-AGCN (Shi et 2019)
92.0
92.2
95.0
95.5
25-AGCN* w/SkeletonGCI.
92.6
93.0
95.7
96.3
CTR CTR-GON GCN (Chen et 2021h)
96.5
CTR-GCN* (Chen 202
94.6
94.2
96.5
CTR GCN W/SkeletonGCLL
95.1
95.0
95.9
96.8
InloGUN (Chi cral. 2022)
94.0
InfoGCNN (Chi et 2022)
93.8
94.2
95.5
96.1
InfoGCN* w/SkeletonGCL
94.8
94.6
96.1
96.8
4.4 DIAGNOSTIC EXPERIMENTS
In this section, we conduct diagnostic experiments to verify the design of SkeletonGCL. Otherwise stated, we CTR-GCN the GCN encoder to perform the experiments the NTU60 dataset under the X-Sub setting using the joint modality (J). See App. 6.2 for more diagnostic experiments.
Table 3:
Comparison of intra-
Table 4: Comparison of feature and graph contrast.
Table 5: banks.
Impact of memory
batch and inter-batch contrast.
GCN)
(9))
Model (CTR-GCN)
Model (CTR-OCN) (%)
Baseline (wio contrast)
89.8
Baseline
89.8
Baseline
89.8
Intra-batch Contrast (No Banky
90.2
Feature Contrast
90.2
Instance Memory
90.3
Inter-batch Contrast
90.8
Graph contrast
90.8
Semantic Memory
90.2
Instance Somantic
90.8
Table 6: Impact of sampling strategics. Random; H: Hard.
Table 7: Comparison with using triplct loss.
Table 8: Training consumption on NTU60.
Model
Time (hours)
Sampling
Acc(%)
(CTR-GCN) (9)
28-AGCN
3.4
Positive
Negative
Baseline
89.8
28-AGUN
2.99
R
Triplct loss
90.7
CTR-GCN
11.4
R
I1
90.2
InfoNCE loss
90.8
CTR-GON wfours
11.7 1.66
R+H
90.5
InfoGCN
InfoGCN volours
4.6 1.0%%
H
H 90.6
R-H
R+H
R+H
90.3
Table 9: Graph Distance (dis.) comparison using Eu-
Table Performance (%) samples with different graph distance ranks to the correct class.
clidean distance (10-5)
Sample
Average dis. to classes
Dis. to correct class
Dis. to misclassified class
CTR
CTR w/Owr
+/-
rank
97.4
Pntorreelyy-Classinee Samples
rank2-5
92.1
92.9
+0.8
0.74 0.68
rank6-10
58.3
89.3
+1.0
Correally-Inssshee
rank 10
84.9
86.3
+1.4
Samples
0.47
2ank21-40
83.2
84.8
+1.6
rank41-60
82.9
+2.1
Intra-batch vs. Inter-batch Graph Contrast. Tab. 3, the effectiveness of introducing cross- sequence context is investigated. We find that only contrasting the graphs within one batch can bring improvement with 0.4% (89.8% to 90.2%), which owes to the cross-sequence relation mining. And further exploring the inter-batch relations can bring more improvements to 1.0% (89.8% to 90.8%), which explains that different batches provide richer context than a single batch.
Graph Contrast vs. Feature Contrast. In Tab. 4, the comparison of using features f to contrast and using graphs g to contrast is investigated. We find that feature contrast can improve the performance on the baseline with 0.4% (89.8% to 90.2%) But graph contrast obviously outperform it by 0.6% (90.2% to 90.8%). The results suggest that, due to the high-order structural information in graphs, graph contrast can better benefit graph convolution learning in GCNs.
8

---

Published as a conference paper at ICLR 2023
Memory Banks. In Tab. 5, the effectiveness of instance-level and semantic-level memory banks is investigated. We find that both memory banks benefit the recognition, and using them together achieves much higher performance. which proves their complementary properties.
Sampling Strategy. In Tab. 6, we compare different sampling strategies for SkeletonGCL. We find that selecting hard positive/negative examples can generally improve recognition. And also random negative samples are meaningful to recognition, which allows the contrastive loss to involve more negative samples.
InfoNCE Loss vs. Triplet Loss. In Tab. 7, we compare the performance of using another popular metric learning loss, ie., triplet loss (Schroff al., 2015). We find that using triplet loss can achieve similar performance compared to InfoNCE loss. The results indicate the generality of our idea that does not depend on certain loss but can boost the performance using different losses.
Traning Comsumption. In Tab. 8, we report the training consumption on NTU60. With our method, the training time only slightly increases with different baseline models, ranging from 2.6% to 7.0%. which proves the efficiency of the design.
Quantitative Results about Graph Similarities. As shown in Tab. 9, we statistically calculate the graph distances between each sample and all classes (detailed in App. 6.4). For incorrectly- classified samples, we find that: (1) The graph distance the misclassified class (0.68) is much lower than the average distance (1.70) to all classes.
(2) The graph distance to the misclassified class (0.68) is indeed slightly lower than the distance to the correct class (0.74), which explains that not learning class-specific graphs could truly degrade recognition performance. In addition, for samples, we notice that: (1) The average graph distance (2.20) is higher than that for the misclassified ones (1.70), which indicates that the inter-class graph representations more dispersed for the correcty-classifiedd samples. (2) The distance to the correct class (0.47) is lower than that for the misclassified ones (0.74), which reveals that the intra-class representations are more compact for the correctly classified samples. To sum up, these quantitative results illustrate the strong correlation between recognition performance and class-specific graph representation.
correctly-classifie
Performance vs. Graph Quality. In Tab. 10. we first calculate the graph distances between each sample and all classes (detailed in App. 6.4) for CTR-GCN. Then, we rank the distances from low to high. In Tab. 10, we report the recognition accuracies of samples according to their distance ranks the correct class. Here, higher ranks indicate that graphs are of higher quality (intra-class compact and inter-class dispersed), while lower ranks indicate that graphs are of lower quality (intra- class dispersed and inter-class aliasing). We note that: (1) Considering samples from lower ranks to higher ranks, performances improve monotonically, revealing the significant correlations between graph quality and recognition performance. (2) Combined with the proposed method, we improve performances in all cases, where larger improvements are obtained on the samples with lower-qualit graphs. These results prove that our method can alleviate the problem caused by learning oowquaaitty graphs.
CONCLUSION
In this paper, we establish a new training paradigm for skeleton-based action recognition, called SkeletonGCL, which explicitly explores the rich semantic context across sequences. Concretely, SkeletonGCL. contrasts the learned graphs among sequences, guiding the graph representations to be class-associated, hence improving GCN capacity to recognize different actions. We improve the current methods significantly to achieve SoTA on three benchmarks.
Limitation. In this paper, we push away the negative pairs from different classes in the same way without considering their intrinsic relations. Therefore, a comprehensive contrasting manner may be more suitable by delicately involving cross-class relations. We leave this for future work.
ACKNOWLEDGEMENTS
This research is supported by the NSFC (grants No. 61773176 and No. 61733007).
9

---

Published as a conference paper at ICLR 2023
REFERENCES
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference Machine Learning, pp. 1597-1607. PMLR, 2020.
Xinlci Chen, Saining Xie, and Kaiming He. An empirical study of training self- suff-super vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp- 9640-9649, 2021a.
Yuxin Chen, Ziqi Zhang, Chunfeng Yuan, Bing Li, Ying Deng, and Weiming Hu. Channel-wise topology refinement graph convolution for ikeleton-based action recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 13359-13368, 2021b.
Zhan Chen, Sicheng Li, Bing Yang, Qinghan Li, and Hong Liu. Multi-scale spatial temporal graph convolutional network for skeleton-bar based action recognition. In Proceedings of the AAAI Confer- ence on Artificial Intelligence, volume 35, pp. 1113-1122 112-,122,000 2021c.
Ke Cheng, Yifan Zhang, Congqi Cao, Lei Shi, Jian Cheng, and Hanqing Lu. Decoupling gen with dropgraph module for skeleton- based action recognition. In European Conference on Computer Vision, pp. 536-553. Springer, 2020a.
Ke Cheng, Yifan Zhang, Xiangyu He, Weihan Chen, Jian Cheng, and Hanqing Lu. Skeleton-based action recognition with shift graph convolutional network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 183-192, 2020b.
Guilhem Chéron, Ivan Laptev, and Cordelia Schmid. P-enn: Pose-based enn features for action recognition. In Proceedings of the IEFE International Conference Computer Vision, pp. 3218- 3226, 2015.
Hyung-gun Chi, Myoung Hoon Ha, Seunggeun Chi, Sang Wan Lee, Qixing Huang, and Karthik Ramani, Infogen: Representation learning for human skeleton-basee action recognition. Pro- ceedings of the JEFE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20186- 20196, 2022.
Yong Du, Wei Wang, and Liang Wang. Hierarchical recurrent neural network for skeleton based action recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1110-1118, 2015.
Haodong Duan, Yue Zhao, Kai Chen, Dahua Lin, and Bo Dai. Revisiting skeleton-based action recogmition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 2969-2978, 2022.
Tianyu Guo, Hong Liu, Zhan Chen, Mengyuan Liu, Tao Wang, and Runwei Ding. Contrastive learning from extremely augmented skeleton sequences for self-supervised action recognition. In Proceedings of the AAÁI Conference on Artificial Intelligence, volume 36, pp. 762-770, 2022.
Michael Gutmann and Aapo Hyvärinen. Noise- contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 297-304. JMLR Workshop and Conference Proceedings, 2010.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9729 9738, 2020.
Yannis Kalantidis, Mert Bulent Sariyildiz, Noe Pion, Philippe Weinzaepfel, and Diane Larlus. Hard negative mixing for contrastive learning. Advances Neural Information Processing Systems, 13:21798-21888 21809, 2020.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishman. Supervised contrastive learning. Advances in Neural Information Processing Systems, 13.18661-18611 2020.
10

---

Published as a conference paper at ICLR 2023
Guy Lev, Gil Sadeh, Benjamin Klein, and Lior Wolf. Rnn fisher vectors for action recognition and image annotation. In European Conference on Computer Vision. pp. 333-850. Springer, 2016.
Linguo Li, Minsi Wang, Bingbing Ni, Hang Wang, Jiancheng Yang, and Wenjun Zhang. 3d hu- man action representation learning via cross-view consistency pursuit. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4741-4755 2021.
Maosen Li, Siheng Chen, Xu Chen, Ya Zhang, Yanfeng Wang, and Qi Tian. oational-structin graph convolutional networks for skeleton-based action recognition. In Proceedings of the JEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 3595-3603, 2019.
Jun Liu, Gang Wang, Ping Hu, Ling-Yi Yu Duan, and Alex C Kot. Global context-aware attention Istm networks for 3d action recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1647-1656, 2017a.
Jun Liu, Amir Shahroudy, Mauricio Perez, Gang Wang. Ling- Yu Duan, and Alex C Kot. Ntu rgh+ 120: A large-scale benchmark for 3d human activity understanding. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(100)2684-2701. 2019.
Mengyuan Liu, Hong Liu, and Chen Chen. Enhanced skeleton visualization for view invariant human action recognition. Pattern Recognition, 68:346-362, 2017b.
Ziyu Liu, Hongwen Zhang, Zhenghao Chen, Zhiyong Wang. and Wanli Ouyang, Disentangling and unifying graph convolutions for skeleton-based action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 143-152, 2020.
Yunyao Mao. Wengang Zhou, Zhenbo Lu, Jiajun Deng, and Houqiang Li. Cmd: Self- supervised 3d action representation learning with cross-modal mutual distillation. arXiv preprint
arXiv:2208. 12448, 2022.
Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invarian representa- tions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6707 6717, 2020.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint 1.07vv/807.0374:: 2018
Chiara Plizzari, Marco Cannici, and Matteo Matteucci. Skeleton- based action recognition via spatial and temporal transformer networks. Computer Vision and Image Understanding. 208: 103:103219, 2021.
Joshua Robinson, Ching- Yao Chuang, Suvrit Sra, and Stefanie Jegelka. Contrastive learning with hard negative samples. arXiv preprint arXiv:2010.04592. 2020.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 815-823, 2015.
Amir Shahroudy, Jun Liu, Tian-Tsong Ng, and Gang Wang. Ntu rgb+ d: A large scale dataset for 3d human activity analysis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1010-1019, 2016.
Lei Shi, Yifan Zhang, Jian Cheng, and Hanqing Lu. Two-stream adaptive graph convolutional networks for skeleton- based action recognition. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 12026 12035, 2019.
Chenyang Si. Wentao Chen, Wei Wang, Liang Wang, and Tieniu Tan. An attention enhanced graph convolutional Istm network for skeleton- based action recognition. Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pp. 1227-1236, 2019.
Afrina Tabassum, Muntasir Wahed, Hoda Eldardiry, and Ismini Lourentzou. Hard negative sampling strategies for contrastive representation learning. arXiv preprint arXiv:2206.01 197, 2022.
11

---

Published as a conference paper at ICLR 2023
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
Hongsong Wang and Liang Wang. Modeling temporal dynamics and spatial configurations of ac- tions using two-stream recurrent neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 499-508, 2017.
Jiang Wang, Xiaohan Nie, Yin Xia, Ying Wu, and Song-Chun Zhu. Cross-view action modeling. learning and recognition. In Proceedings the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2649-2656, 2014.
Wenguan Wang, Tianfei Zhou, Fisher Yu, Jifeng Dai, Ender Konukoglu, and Lue Van Gool. Ex- ploring cross-image pixel contrast for semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7303-7313,200 2021.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 3733-3742, 2018.
Sijie Yan, Yuanjun Xiong, and Dahua Lin. Spatial temporal graph convolutional networks for skeleton-bar based action recognition. In Thirty-second AAAI Conference on Artificial Intelligence 2018.
Fanfan Ye, Shiliang Pu, Qiaoyong Zhong, Chao Li, Di Xie, and Huiming Tang. Dynamic gen: Context-enriched topology learning for skeleton-based action recognition. In Proceedings the 28th ACM International Conference on Multimedia, pp. 55-63, 2020.
Pengfei Zhang, Cuiling Lan, Wenjun Zeng, Junliang Xing, Jianru Xue, and Nanning Zheng. Semantics-guice -uiided neural networks for efficient skeleton-based human action recognition. Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1112- 1121, 2020a.
Xikun Zhang, Chang Xu, and Dacheng Tao. Context aware graph convolution for skeleton- based action recognition. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 14333-14342, 2020b.
APPENDIX
6.1 IMPLEMENTATIONS GRAPH PROJECTION HEADS FOR GCNS.
Average Pool
Flastium
Flatium
Average Pool
Flatlen
K2 X.C
x
KyNa
Kg X N
KyN²
TX KN
KyN²
C
DJ CTR-GEN
(b) I5-AGON
(C) InfoGINN
Figure Illustration of graph projection heads for GCNs.
In Fig. 3. we illustrate the implementation details of graph projection heads for different GCNs. Particularly, for CTR-GCN and InfoGCN, first apply average pooling layer summarize the information along the channel and temporal dimensions, respectively. Then, the same as 2S-AGCN, flatten the graphs and embed them with an FC layer.
12

---

Published as a conference paper at ICLR 2023
6.2 MORE DIAGNOSTIC EXPERIMENTS.
Table 11:
Comparison of using cross-entropy loss
Table 12: Impact of the FC layer in graph projection head.
to supervise graph learning.
Model (CTR-GCN)
Acc (%)
Model (CTR-GCN)
Acc (98)
Baseline
89.8
Bascline
89.8
WCCoss-entropyy loss
89.7
Graph Projection worRC.
90.1
w/SkeletoBCCL
90.8
Graph Projection w/F
90.8
Graph Projection WMLP
90.6
Table 13: Apply graph contrast on different layers.
Table 14: Impact of the size of
Table 15:
Performance compari-
Mis.
son with different Cg.
Model (CTR-GCN) Ace (%)
P
Acc
Ace (%)
Baseline 89.8
128
90.3
64
90.4
layer
342
90.5
128
90.6
8th layer
90.0
684
90.8
256
90.8
9th layer
90.4
90.5
10th layer
90.8
2736
90.0
Table 16: Impact
Table 17: Impact of a.
Table 18: Performance compari- son with different K
Ace (58)
a
Acc (18)
0.5
90.4
0.75
90.6
K y/
Acc (%)
0.8
90.6
0.80
90.7
64
90.6
90.8
0.85
90.8
128
90.8
1.5
90.5
0.9
90.6
256
90.6
512
90.5
Table 19: Performance comparison with different
Table 20: Performance comparison with diffcrent
Ki.
Kg.
K yr
R
128
90.3
128
90.6
256
256
512
90.8
512
90.8
1024
1024
Comparison of using cross-entropy loss. Since cross-entropy loss a widely used classification loss in learning slass-discimminative representations, in Tab. 11, we investigate its performance to supervise graph learning. We find that directly using cross-entropy loss for graph learning has negligible effects on the performance (89.8% 89.7%), which indicates that it is impractical to learn favorable class-discriminative graphs by naively using a classification loss. In this paper, we find a practical way achieve this goal by introducing the cross-sequence context for guiding graph learning.
Impact of FC in Projection Head. In Tab. 12, the effectiveness of transformation layer (FC layer) in the graph projection head is investigated. We find that the model achieves obvious improvement (90.1% to 90.8% equipped with the FC layer, which proves the importance of aertex-aware graph encoding. In addition, we find that using an MLP achieves similar but lower accuracy, hence we use a simple FC in the framework.
Which I Layer to Contrast Graphs? In Tab. 13, we apply graph contrast on different layers. We find that contrasting graphs on deeper layers outperform on shallower layers. One possible explanation is that deeper layers can provide higher-level semantics that is relevant to recognition.
Impact of the size of Mins In Tab. 14, the impact of the size of Mins is investigated, where we use different values of P to control the size. We find that appropriately increasing the size can effectively expand the crose-sequence context, and improve recognition performance. However, an over large memory bank stores old samples from few batches ago, which hinders representation learning.
Impact of dimension Cy. In Tab. 15, the influences of the dimension of graph vector C are investigated. For pursuing the best performance, set C 256.
Impact of temperature т. In Tab. 16, the influences of the temperature т are investigated. For pursuing the best performance, we as 1.0.
Impact of a. In Tab. 17, the influences of the momentum updating gated. For pursuing the best performance, we a as 0.85.
a are investi-
13

---

Published as a conference paper at ICLR 2023
Impact of the number of sampling examples. In Tab. 18, the impact of selecting K hardest positive examples is investigated. In Tab. 19, the impact of selecting K3 hardest negative examples investigated. In Tab. 20, the impact of selecting K random negative examples is investigated. For pursuing the best performance,
Kjj. KH. and K 128, 512 and 512, respectively.
The quantitative analysis of accuracy improvement. Tab. 21, the recognition accuracies of the top-10 hardest classes for CTR-GCN on NTU-60 presented. The improvements in four classes (i.e., "reading", "typing on a keyboard", "headache" and "point to something") are over 4%. Though performances in three classes decrease, they are relatively small (-1.5% on "writing", 1.0% "take off a shoe", and - 2.5% on "sneeze/cough") vs. others' increase. Overall, we obtain an average improvement the 10 classes of 2.7%.
In Tab. 22, the recognition accuracies of top-10 improved classes for CTR-GCN on NTU-60 are presented. The accuracy of the above 10 classes shows an average gain of 4.6%.
Table 21: Performance (%) on top 10 hardest classes for CTR-GCR
Table 22: Performance (%) on top improved classes for
classes
CTR-GCN
CTR-GCN w/GCL
+/-
classes
CTR-GCN CTR-GCN wAGCL
4/-
reading
58.2
49.6
reading
type on keyboard
66.3
70.9
+4,4
beadache
78.3
83.0
writing
67.3
65.8
-1.5
rub twa hands
87.3
92.0
+4.7
play with phone/tablet
68.0
70.9
+2.9
punchislap
8944
93.8
+4.4
71.6
type keyboard
70.9
take oil shoe
75.5
74.5
-1.0
point to something
81.9
85,9
+4.0
hendache
78.3
83.0
+4.7
clapping
82.1
85.7
+3.6
point something
+4.0
into pocket
Clapping
82.1
85.7
+3.6
meal
71.6
75,2
+3.6
snoeze/cough
82.2
79.7
-2.5
Book pain
88.0
91.3
+3.3
Average
73.2
Averago
78.6
83.2
6.3 QUALITATIVE RESULTS
In Fig. 4, we visualize the t-SNE distribution of graph and feature representations of sequences from six classes, illustrating the impact of SkeletonGCL. As shown in Fig. 4(a), SkeletonGCL can shape the graph representation structure, where the graphs from the same class get together and graphs from different classes spread out. Consequently, in Fig. 4(b), with SkeletonGCL, the features from different classes become more distinguishable, which indicates that graph contrast indeed improves the feature extraction capacity.
Feature Representation
CTR-GON
CTR w/SkcletonGCL
t-SNE visualization of graph representation.
CTR-GCN
CTR-GON w/ours
t-SNE visualization of feature representation.
Figure 4: t-SNE visualization. t-SNE (Van der Maaten & Hinton, 2008) visualization of graph and feature representations from sequences in the test set of NTU 60. Each color denotes a certain class. Best viewed with zoom in.
6.4 THE CALCULATION DETAILS GRAPH DISTANCE
In Tab. 9, the statistics of graph distance for all samples are investigated. In CTR-GCN, the graph g E K is learned for graph convolution. For the convenience of calculation, we use
an average pooling to squeeze g and reshape into = g E xIxNa as the graph embedding to conduct distance measurement.
14

---

Published as a conference paper at ICLR 2023
To acquire the graph embedding for each class Cx, we first calculate the centroid vector as follows,
N Bc. N Σ 9
where No, denotes the number of samples which belongs to class c.. To effectively reveal the rela- tion between graph quality and classification accuracy, we introduce three types of graph distance,
i.e., dall- door and dmis- dall measures the average distance to all classes. door measures the distance to the correct class dmis measures the distance to the misclassified class Cmis.
K
dall 1
11g
Scx 11?
(10)
K
k=1
deor ||9
Sc.
122
(11)
dnni
9
Scni
(12)
where K denotes the number of classes in the dataset.
15


Reviews:
[{"review_id": "r3dVTnQlR8", "reviewer": "Reviewer_YfcM", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors identify the lack of global context as a limiting factor of current state of the art, graph-convolution-based methods for skeletal action recognition. They propose a training paradigm based on a contrastive loss that is applicable to most existing approaches. Their proposed method is evaluated against existing work on the most widely used benchmarks and an ablation study investigating the impact of different model components is provided.", "summary_of_the_review": "The paper presents some improvements over state of the art in skeleton-based action recognition. The main claim of the paper (i.e., that graph structure and their lack of contrast to the misclassified class is responsible for limited performance of existing works,) is not backed by quantitative evidence. Similarly, the results are lacking an analysis if the proposed contrastive learning scheme solves the proposed problem. Obviously there is improvement, but there is no convincing analysis if the improvement roots in better graph structures.", "strength_and_weaknesses": "**Strengths**\n\n(a) technical idea\n- using two memory banks to explicitly control an instance-based and a semantic loss part is a simple and effective solution\n- the idea to explicitly learn graph contrasts to avoid similar graphs that lead to classification error is, to the best of my knowledge, novel\n\n(b) results\n- several recent state of the art benchmarks are consistently outperformed\n- the authors provide a fair comparison, where they report the original numbers from the referenced papers, but also retrain those methods and report the retrained results, although they are frequently better than the numbers reported in the original paper. I appreciate the care with which results are presented.\n- the t-SNE plot shows that the proposed method leads to clearer clusterings of produced graphs, which was the goal of the paper.\n\n**Weaknesses**\n\n(a) Novelty\nThe proposed method is essentially any existing graph convolutional method trained with an additional contrastive loss. While it is interesting to see the improvements this causes, the technical novelty of this approach is rather limited.\n\n(b) Motivation and Evaluation\nThe authors motivate their approach by the hypothesis \"From the visualization, we find that (1) For sequences that are\ncorrectly classified in Fig. 1 (a) and Fig. 1 (b), the learned graphs in the same class look similar,\nwhile graphs in different classes have distinct differences. (2) For a misclassified sequence in Fig.\n1 (c), the learned graph resembles the graphs from the misclassified class more than those from the\nground truth class.\"\nUnfortunately, they do not provide quantitative evidence of this hypothesis. While there is one qualitative example shown in Fig. 1, this is not evidence that the problem is a broad problem for GCN based action recognition. The hypothesis needs to be backed by some quantitative metrics, e.g. showing that misclassifications strongly correlate with graph distances to the misclassified class being lower than average or lower than to the correct class.\n\n(c) Results\nSimilar as for the motivation, the results are lacking evidence that the proposed method alleviates the problem that is claimed to cause lower performance, i.e., the problem that misclassified graphs are too similar to graphs from other classes.\nWhile the proposed approach leads to some improvement over state of the art methods, it is unclear if this improvement is caused by improved graph structures as claimed/motivated by the authors.\nAn in-depth analysis of the main claim of the paper is therefore missing.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "qU_n5LMqrEL", "reviewer": "Reviewer_vLtp", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a graph contrastive learning framework to explore the global context across all sequences for action recognition. It aims to enforce graphs to be class-discriminative.  More importantly, it proposes two memory banks for two complementary levels: instance and semantic levels. This method can be integrated into other graph convolution networks (GCN). The experimental results on three public datasets are better than other compared approaches.", "summary_of_the_review": "The graph contrastive learning is novel. In addition, it could be integrated into other GCNs and improves the performance of action recognition.", "strength_and_weaknesses": "Strength:\n\nIt proposes the graph contrastive learning framework for action recognition.  This framework can be integrated into other graph convolution networks. The experimental results have shown that the proposed framework is integrated into three GCNs and is able to improve performance.  \n\nWeaknesses:\n\nRegarding the instance-level memory bank, how to assign the number of instances to different classes? It would be great to add more descriptions. The infoNCE loss for the semantic part should be provided since this part is novel. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "lRPXBAC7GI", "reviewer": "Reviewer_Cq4q", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors present a method to perform skeleton-based action recognition using contrastive learning with graph sequences. To aid in contrastive learning, the authors propose two memory banks to store the graph-based data: an instance-level memory bank to store the properties of individual data samples and a semantic-level memory bank to store the aggregate properties of all the samples in each action class. The combined implementation of the two memory banks enables their learned network to cluster together intra-class samples and spread out inter-class samples. The authors integrate their contrastive learning approach with multiple graph convolutional techniques for skeleton-based action recognition and experimentally demonstrate the benefits of their proposed components.", "summary_of_the_review": "The authors tackle the well-known problem of skeleton-based action recognition that has mature solutions available, and propose a contrastive learning-based method that integrates with the current solutions and numerically offers only marginal benefits. While that by itself does not discount the importance of their contribution, I recommend the authors describe the differences between their memory banks more rigorously and better explain where and how their proposed method performs best to get a clearer understanding of the quality of their work.", "strength_and_weaknesses": "**Strengths**\n1. The proposed approach of using contrastive learning and the memory bank descriptions are technically sound.\n2. The paper is easy to follow overall and well-organized.\n\n\n**Weaknesses**\n\nMy main concern is regarding the results, particularly in the net improvement the authors' proposed techniques achieve.\n\n1. All the experiments show a 1% or less improvement in accuracy, but the authors haven't discussed whether such improvements are significant. Assuming these are the mean accuracy numbers, what are the corresponding standard deviations?\n\n2. Do the authors have confusion matrices to show the intra-class and inter-class performances? Given that the baseline accuracies are already close to 90%, it can perhaps help to show how much the proposed method improves performance just on hard data samples (note that this is different from the performance improvement using hard sampling that the authors report in Table 6).\n\n3. Somewhat along these lines, I see visual evaluations (Fig. 3) that provide a better perspective of how contrastive learning helps. Can the authors further explain why the graph representation changes dramatically (Fig. 3a) while the feature representations stay almost the same (Fig. 3b)?\n\n4. For Table 5, have the authors performed any additional experiments with the instance-level memory bank size $P$?\n\n5. I am also not fully clear on the need for a separate semantic-level memory bank that aggregates the graph-based features across all intra-class samples. How different is that from aggregating the features in the instance-level memory bank per class? I can see that the memory bank size $P$ can play a role here, how large is the value of $P$ compared to the class populations?\n\n6. Further, how does the hard sampling work in conjunction with the FIFO setup of the instance-level memory banks? Which samples are removed from the queue after the similarity calculation?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "JRPs-RFRxou", "reviewer": "Reviewer_YiMJ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces a graph neural network-based approach to tackle the skeleton-based action recognition problem. The work builds upon the intuition that current methods are not explicitly exploiting the information shared by all available sequences but are only using the intra-sequence features to learn proper graph representations. Following such an intuition, the paper formulates the problem of learning better representation through a contrastive learning approach that uses graph embeddings to pull graphs for the same action together, while pushing away graphs for other classes. This is combined in a solution that exploits a contrastive learning approach and a separated classification head. Experiments on 3 benchmark datasets show that the approach yields limited improvements with negligible computational cost. ", "summary_of_the_review": "I found the paper easy to read and well-motivated. The solution, despite being simple, provides (marginal) improvements to different existing approaches. This demonstrates the capabilities of the approach that can also be extended to other domains. The computational effort at training time is negligible and that there is no impact at inference time (wrt to the adopted base model). In light of all such considerations, the paper seems to have enough merits to justify a publication.", "strength_and_weaknesses": "**Strengths**\n+ The paper is well-written, easy to follow, and presented concisely. All the most relevant details are described with enough depth providing the reader with enough information to properly grasp the overall idea and the rationale behind the choice of the implemented components. The implementation details seem to be enough to replicate the proposed solution. This would be very helpful to help further studies.\n+ The approach seems quite simple, yet it shows interesting performance. The paper proposes to exploit inter-sequence information as well as intra-sequence features to learn a proper representation to recognize actions with a graph-based approach. This comes with the plus that, as shown in the experimental results, the approach adds negligible computational effort to current schemes.\n+ The proposed approach can be applied on top of existing solutions. The method introduces a separate branch that exploits the graph learned through an existing approach (e.g., CTR-GCN, 2s-AGCN, etc.) to improve its generalization capabilities by exploiting the contrastive learning scheme. \n\n**Weaknesses**\n\nOverall, the paper does not have critical issues that may hamper a publication. There are a few minors that can be addressed with a review round. These include:\n- While describing the evaluation methodology the paper states that there are four modalities being considered to compute the results, i.e., (J), (B), (J-B), (B-M), which can also be used together as in (4S). However, the tables (e.g., Table 1 and Table 2) report on the performance considering (J), (B), (J+B), and (4S). Clarifications/fixes on these are needed.\n- There are a few hyperparameters listed in the experimental details that have no justification. These are the number of instances in each class $P$, the dimension of the graph vector $C_g$, and the dimension of the samplings for loss computation $K^H$'s and $K^R$ 's. It would be interesting to hear if these are found through experimental validation, cross-validation, etc.\n- The submission does not come with a discussion about the limitations of the proposed approach. It would be interesting to hear what are the limiting factors that the authors came across.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster