Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper introduces a graph neural network-based approach to tackle the skeleton-based action recognition problem. The work builds upon the intuition that current methods are not explicitly exploiting the information shared by all available sequences but are only using the intra-sequence features to learn proper graph representations. Following such an intuition, the paper formulates the problem of learning better representation through a contrastive learning approach that uses graph embeddings to pull graphs for the same action together, while pushing away graphs for other classes. This is combined in a solution that exploits a contrastive learning approach and a separated classification head. Experiments on 3 benchmark datasets show that the approach yields limited improvements with negligible computational cost. 
strength_and_weaknesses: **Strengths**
+ The paper is well-written, easy to follow, and presented concisely. All the most relevant details are described with enough depth providing the reader with enough information to properly grasp the overall idea and the rationale behind the choice of the implemented components. The implementation details seem to be enough to replicate the proposed solution. This would be very helpful to help further studies.
+ The approach seems quite simple, yet it shows interesting performance. The paper proposes to exploit inter-sequence information as well as intra-sequence features to learn a proper representation to recognize actions with a graph-based approach. This comes with the plus that, as shown in the experimental results, the approach adds negligible computational effort to current schemes.
+ The proposed approach can be applied on top of existing solutions. The method introduces a separate branch that exploits the graph learned through an existing approach (e.g., CTR-GCN, 2s-AGCN, etc.) to improve its generalization capabilities by exploiting the contrastive learning scheme. 

**Weaknesses**

Overall, the paper does not have critical issues that may hamper a publication. There are a few minors that can be addressed with a review round. These include:
- While describing the evaluation methodology the paper states that there are four modalities being considered to compute the results, i.e., (J), (B), (J-B), (B-M), which can also be used together as in (4S). However, the tables (e.g., Table 1 and Table 2) report on the performance considering (J), (B), (J+B), and (4S). Clarifications/fixes on these are needed.
- There are a few hyperparameters listed in the experimental details that have no justification. These are the number of instances in each class $P$, the dimension of the graph vector $C_g$, and the dimension of the samplings for loss computation $K^H$'s and $K^R$ 's. It would be interesting to hear if these are found through experimental validation, cross-validation, etc.
- The submission does not come with a discussion about the limitations of the proposed approach. It would be interesting to hear what are the limiting factors that the authors came across.
clarity,_quality,_novelty_and_reproducibility: **Clarity/Reproducibility:** as discussed in the strengths, the work is properly presented and discussed. All the most relevant points are defined with enough details, hence can be exploited to reproduce the submitted work (implementation settings are also properly listed).

**Quality:** the submitted paper covers a challenging task by proposing a simple, yet effective, approach that can be applied on top of the different existing solutions by introducing a negligible computational effort. 

**Novelty:** in my personal view, the paper has properly clarified the differences with the existing literature thus making clear how it differs from the current works. Hence, the proposed approach is novel enough to possibly justify a publication. 
summary_of_the_review: I found the paper easy to read and well-motivated. The solution, despite being simple, provides (marginal) improvements to different existing approaches. This demonstrates the capabilities of the approach that can also be extended to other domains. The computational effort at training time is negligible and that there is no impact at inference time (wrt to the adopted base model). In light of all such considerations, the paper seems to have enough merits to justify a publication.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors identify the lack of global context as a limiting factor of current state of the art, graph-convolution-based methods for skeletal action recognition. They propose a training paradigm based on a contrastive loss that is applicable to most existing approaches. Their proposed method is evaluated against existing work on the most widely used benchmarks and an ablation study investigating the impact of different model components is provided.
strength_and_weaknesses: **Strengths**

(a) technical idea
- using two memory banks to explicitly control an instance-based and a semantic loss part is a simple and effective solution
- the idea to explicitly learn graph contrasts to avoid similar graphs that lead to classification error is, to the best of my knowledge, novel

(b) results
- several recent state of the art benchmarks are consistently outperformed
- the authors provide a fair comparison, where they report the original numbers from the referenced papers, but also retrain those methods and report the retrained results, although they are frequently better than the numbers reported in the original paper. I appreciate the care with which results are presented.
- the t-SNE plot shows that the proposed method leads to clearer clusterings of produced graphs, which was the goal of the paper.

**Weaknesses**

(a) Novelty
The proposed method is essentially any existing graph convolutional method trained with an additional contrastive loss. While it is interesting to see the improvements this causes, the technical novelty of this approach is rather limited.

(b) Motivation and Evaluation
The authors motivate their approach by the hypothesis "From the visualization, we find that (1) For sequences that are
correctly classified in Fig. 1 (a) and Fig. 1 (b), the learned graphs in the same class look similar,
while graphs in different classes have distinct differences. (2) For a misclassified sequence in Fig.
1 (c), the learned graph resembles the graphs from the misclassified class more than those from the
ground truth class."
Unfortunately, they do not provide quantitative evidence of this hypothesis. While there is one qualitative example shown in Fig. 1, this is not evidence that the problem is a broad problem for GCN based action recognition. The hypothesis needs to be backed by some quantitative metrics, e.g. showing that misclassifications strongly correlate with graph distances to the misclassified class being lower than average or lower than to the correct class.

(c) Results
Similar as for the motivation, the results are lacking evidence that the proposed method alleviates the problem that is claimed to cause lower performance, i.e., the problem that misclassified graphs are too similar to graphs from other classes.
While the proposed approach leads to some improvement over state of the art methods, it is unclear if this improvement is caused by improved graph structures as claimed/motivated by the authors.
An in-depth analysis of the main claim of the paper is therefore missing.
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written and I believe it can be reproduced from the description of the method.
The novelty is limited: on the technical side, any existing GCN is combined with a contrastive loss on the graphs. On the conceptual side, a novel claim is made but unfortunately is not backed by evidence.
summary_of_the_review: The paper presents some improvements over state of the art in skeleton-based action recognition. The main claim of the paper (i.e., that graph structure and their lack of contrast to the misclassified class is responsible for limited performance of existing works,) is not backed by quantitative evidence. Similarly, the results are lacking an analysis if the proposed contrastive learning scheme solves the proposed problem. Obviously there is improvement, but there is no convincing analysis if the improvement roots in better graph structures.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a graph contrastive learning framework to explore the global context across all sequences for action recognition. It aims to enforce graphs to be class-discriminative.  More importantly, it proposes two memory banks for two complementary levels: instance and semantic levels. This method can be integrated into other graph convolution networks (GCN). The experimental results on three public datasets are better than other compared approaches.
strength_and_weaknesses: Strength:

It proposes the graph contrastive learning framework for action recognition.  This framework can be integrated into other graph convolution networks. The experimental results have shown that the proposed framework is integrated into three GCNs and is able to improve performance.  

Weaknesses:

Regarding the instance-level memory bank, how to assign the number of instances to different classes? It would be great to add more descriptions. The infoNCE loss for the semantic part should be provided since this part is novel. 
clarity,_quality,_novelty_and_reproducibility: The paper is well written. The main contribution of this paper is the new training paradigm – graph contrastive learning, which can be integrated into other GCNs. It would be great if the authors can release their code. It may be hard to reproduce the results 
summary_of_the_review: The graph contrastive learning is novel. In addition, it could be integrated into other GCNs and improves the performance of action recognition.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors present a method to perform skeleton-based action recognition using contrastive learning with graph sequences. To aid in contrastive learning, the authors propose two memory banks to store the graph-based data: an instance-level memory bank to store the properties of individual data samples and a semantic-level memory bank to store the aggregate properties of all the samples in each action class. The combined implementation of the two memory banks enables their learned network to cluster together intra-class samples and spread out inter-class samples. The authors integrate their contrastive learning approach with multiple graph convolutional techniques for skeleton-based action recognition and experimentally demonstrate the benefits of their proposed components.
strength_and_weaknesses: **Strengths**
1. The proposed approach of using contrastive learning and the memory bank descriptions are technically sound.
2. The paper is easy to follow overall and well-organized.


**Weaknesses**

My main concern is regarding the results, particularly in the net improvement the authors' proposed techniques achieve.

1. All the experiments show a 1% or less improvement in accuracy, but the authors haven't discussed whether such improvements are significant. Assuming these are the mean accuracy numbers, what are the corresponding standard deviations?

2. Do the authors have confusion matrices to show the intra-class and inter-class performances? Given that the baseline accuracies are already close to 90%, it can perhaps help to show how much the proposed method improves performance just on hard data samples (note that this is different from the performance improvement using hard sampling that the authors report in Table 6).

3. Somewhat along these lines, I see visual evaluations (Fig. 3) that provide a better perspective of how contrastive learning helps. Can the authors further explain why the graph representation changes dramatically (Fig. 3a) while the feature representations stay almost the same (Fig. 3b)?

4. For Table 5, have the authors performed any additional experiments with the instance-level memory bank size $P$?

5. I am also not fully clear on the need for a separate semantic-level memory bank that aggregates the graph-based features across all intra-class samples. How different is that from aggregating the features in the instance-level memory bank per class? I can see that the memory bank size $P$ can play a role here, how large is the value of $P$ compared to the class populations?

6. Further, how does the hard sampling work in conjunction with the FIFO setup of the instance-level memory banks? Which samples are removed from the queue after the similarity calculation?
clarity,_quality,_novelty_and_reproducibility: **Clarity**

1. It seems that the memory banks are a part of the authors' main contributions. But they are not listed as part of the main contributions (end of Sec. 1).

2. I found the use of subscripts and superscripts to indicate positives and negatives inconsistent. For example, in Sec. 3.1, the authors write positive input samples and features as $I^+$ and $f^+$, but write their negative set as $\mathcal{N}_-$ (more such examples in the "Loss" and "Hard Sampling" paragraphs). If there is any specific reasoning behind these differences, I would be happy to learn it.

3. I would recommend explicitly writing down the loss function for $\mathcal{L}_\textrm{NCE}^\textrm{Sem}$ for completeness.


**Quality and Novelty**

Without any additional insight into the results (please refer to my concerns under "weaknesses"), the contributions of the proposed method appear marginal, with most of the heavy lifting already done by the current graph convolutional methods. Also, explaining the differences between aggregating the instance-level memory bank features and using the semantic-level memory banks more rigorously can help in understanding the quality of the contributions.


**Reproducibility**

The authors sufficiently describe their proposed components so their method is reproducible.
summary_of_the_review: The authors tackle the well-known problem of skeleton-based action recognition that has mature solutions available, and propose a contrastive learning-based method that integrates with the current solutions and numerically offers only marginal benefits. While that by itself does not discount the importance of their contribution, I recommend the authors describe the differences between their memory banks more rigorously and better explain where and how their proposed method performs best to get a clearer understanding of the quality of their work.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes to better leverage the global context (intra-sequence context) to construct adaptive graphs for skeleton based human action recognition. The designed method associates graph learning over different sequences and drives the graphs to be class discriminative, which improves the graph convNet's capability to distinguish various action patterns. Most of the reviewers (and also the area chair) agree that this view is interesting. Specifically, the performance improvement on hard classes is good. However, the major weakness is that the overall performance gain on the whole dataset is ~1% only. Yet, considering that the performance improvements reported by state-of-the-arts are also at a similar scale, and also, the method obtains more obvious improvement on hard classes, the AC recommends accept for this paper. Authors need to incorporate the new results and content provided in rebuttal to the final version of the main paper.
justification_for_why_not_higher_score: Though the method is interesting and novel, the main weakness is that performance gain is not very obvious.
justification_for_why_not_lower_score: It is an interesting perspective to take advantage of the cross-sequence context information to enhance the graph discrimination capability.
Decision: Accept: poster