Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors present a method to perform skeleton-based action recognition using a novel self-paced learning on features in a hyperbolic embedding space. The learning is "self-paced" in the sense that the magnitude of the gradients for the learning optimization directly depends on the certainty of the learned features. The authors follow a curriculum scheduling approach to stabilize their self-paced model. They also propose a positive-only learning paradigm, showing that adding negative samples into the hyperbolic embedding space deteriorates performance. Overall, the authors show appreciable improvements in action recognition performance using their hyperbolic self-paced model and also demonstrate the benefits of the individual components of their method through ablation studies.
strength_and_weaknesses: **Strengths**
1. The paper is clearly written and easy to follow. I was not familiar with prior work around hyperbolic embedding spaces and self-paced learning, but I found the concepts intuitive and the descriptions sound.

2. The experimental results clearly show the contributions of the individual components of the proposed method. The authors back up all their claims around curriculum scheduling, positive-only sampling, and uncertainty in self-paced learning with appropriate experiments.

3. I also found the limitations sections helpful to clearly understand the scope of the work.


**Weaknesses**
1. While I did not find any major concerns with the work, I am curious how the choice and the quality of positive samples affect performance. For example, how much does the performance depend on "hard positives"? Fig. 3a shows the correlation between the feature distances and the uncertainties. If only samples corresponding to the last few columns were to be used in the training (which I'm assuming are the hardest samples), how much would the performance drop? Since humans can perform similar actions with many diverse movements, real-world data can contain many such hard positives.

2. In addition to showing the inter-class variability (Sec. 4.4 and appendix), it can be illuminating to also see the confusion matrix, showing how the variability might be correlated with the per-class performance.
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written. I could follow it without prior knowledge of self-paced learning and hyperbolic embedding spaces. The technical contributions are sound and also make intuitive sense. The experimental results also highlight the novelty of the work. All parts of the method are fully described, making the work reproducible. The authors also provide their source code.
summary_of_the_review: The paper presents a sound and intuitive method for skeleton-based action recognition. The contributions are clear, and the experimental results sufficiently back up the authors' claims.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: - The paper explores hyperbolic self paced learning for self supervised learning of action representations. 
- The key idea is to use hyperbolic geometry to automatically order the training samples from easy to hard. This determines the learning pace thus improving learning. 
- The framework is borrowed from BYOL, which does not need negatives. 
- The approach shows promising performance on the tasks considered. 
strength_and_weaknesses: Strengths:
- The paper is well written and easy to follow. The intuitions are clear for the most part. 
- I like the idea of self-pacing for self-supervised learning. The paper proposes a novel approach based on hyperbolic geometry to tackle this. 
- The paper combines ideas from four different areas. The combination is a novel contribution to the field. 
- The approach leads to good gains on some of the tasks considered.  

Weaknesses:
- While the approach shows good improvements on semi-supervised learning, these do not hold for the other settings. The comparison for finetuning would have been easier if there were numbers reported for ISC. 
- It is not clear from the paper why hyperbolic specifically. The Section 3.2, directly starts with explaining how a hyperbolic extension to SkeletonCLR would work. As far as I understand, the hyperbolic geometry is useful to encode hierarchies, tree-like structures. It is not clear to me why the problem being tackled by this paper needs that. 
- Useful intuition to clarify : What does it mean for a target to be uncertain. I found an exaplanation to this in caption for Figure 3. I suggest you include this earlier in the text to ease understanding. 
- A non-hyperbolic self-paced learning baseline : From Fig 3 caption ".. Uncertainty indicates the difficulty in classifying specific actions if they are characterized by less or more peculiar..". Is it possible to weigh the usual contrastive learning loss using the distance between positives as a baseline to determine uncertainty? 
- Given that your approach is based on a different framework compared to state-of-the-art, you should include results of a non-hyperbolic baseline with BYOL. This will help determine how much of the improvement can be obtained using BYOL based baseline vs MoCo,  For example, for Image SSL tasks, BYOL is generally better than MoCo-v2. 
- Drop in performance with negatives : While the argument for why negatives cannot be generated in the h space seems convincing, what happens if you generate negatives/apply contrastive learning loss for this experiment on the z space (after projection g) ? 
- Suggestion : I think in addition to linear evaluation, you should evaluate your approach on transfer learning as well. For skeleton-based SSL, NTU -> PKU I/II is a good option. 
- The paper includes details on training time. It would be helpful to include details on how much extra compute/traning time is needed due to the use of Riemannian gradient descent. 

clarity,_quality,_novelty_and_reproducibility: - The paper is well written and easy to follow for the most part. Please look at the weaknesses for some suggestions on how to improve clarity. 
- The approach combines ideas from four different areas in a novel way. 
summary_of_the_review: The paper proposes a new technique for SSL on skeleton sequences. The improvements are decent, though not consistent on all tasks. I have some concerns on baselines and intuitions and hence my current score.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a self-supervised skeleton representation learning method based on the BYOL and hyperbolic space. The experimental results show that the proposed HYSP model achieves state-of-the-art performances on three public datasets. 
strength_and_weaknesses: Pros:

(1) This is the first self-supervised skeleton representation work on hyperbolic space.

(2) This paper is well written. 


Cons:

(1) The proposal seems to be incremental based on prior works. HYSP is not the first method that uses the hyperbolic space in skeleton action recognition [1*] and uses the BYOL for self-supervised skeleton representation learning [2*]. 

(2) Why only BYOL?: In my opinion, a better way to show the hyperbolic space is useful is to combine the hyperbolic space with multiple self-supervised learning baselines. From the results in Table 3 (w/ neg.), the hyperbolic space seems not to work well on MoCo. 

(3)  The linear evaluation results are not as good as SOTA methods. And the transfer learning results are missing. These two evaluation protocols are more important than the semi-supervised and supervised settings, regarding showing the effectiveness of self-supervised methods, in my opinion.

(4) Table 3: the BYOL-only result (w/o hyper.) may be not reasonable. 


  

[1*] Mix Dimension in Poincar\'{e} Geometry for 3D Skeleton-based Action Recognition. ACM MM 2020. 

[2*] Bootstrapped Representation Learning for Skeleton-Based Action Recognition. CVPRW 2022. 
clarity,_quality,_novelty_and_reproducibility: Clarity: 
This paper is well-written.

Quality:
The paper has minor technical flaws.

Novelty:
The novelty of the proposed method is marginal 

Reproducibility:
The authors have provided the code in the supplemental materials.
summary_of_the_review: As mentioned in the 'Strength And Weaknesses', both the hyperbolic space and BYOL are not new in skeleton action recognition. So I tend to give the 'marginally below the acceptance threshold'.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: While most works in self-supervised skeleton-based action recognition have used MoCo for modeling, this paper makes a case for using the positives-only design of BYOL instead. The paper proposes the HYperbolic Self-Paced model (HYSP) according to the principle that more certain samples should drive the learning process more predominantly. The authors empirically demonstrate that the proposed approach achieves superior performance on public datasets.
strength_and_weaknesses: - Strength:
    - The paper is well written and the problem being tackled is clearly described.
    - The introduction of hyperbolic mapping and curriculum learning brings a different perspective to self-supervised skeleton action recognition.
    - The paper has a detailed theoretical basis and the detailed visible analysis (Figure 2,4,5) further makes the idea more convincing.

- Weakness:
    - Parts of the approach do exist in prior works. The authors should further elaborate whether the HYSP is a combination of several methods (BYOL + Ganea et al. 2018 + Guo et al. 2022). However, I think that the paper presents a good study on some of the practices and makes a compelling case for using BYOL for the task.
    - HYSP is not the first method to use the positive-only BYOL and the performance of HYSP is not that competitive. *''Moliner et al. Bootstrapped Representation Learning for Skeleton-Based Action Recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2022, pp. 4154-4164.''*
    - Although mentioned in the conclusion, I still think that HYSP is more like a general improvement of the self-supervised framework for BYOL combined with hyperbolic space. Measurements on image classification tasks may be more convincing and necessary. I have not seen a unique improvement and analysis for the field of skeleton action recognition.

- Other questions:
    - Double check the last name and first name of the author of the cited paper, some are confused, such as "Tianyu et al." should be "Guo et al.".
    - Please explain Eq.5 in more detail. The conclusion behind Eq.5 cannot be intuitively understood from Eq.5.
    - In Table 1, SkeletonCLR also uses negative samples.
    - Does "w/o hyper." in Table 3 mean using BYOL directly? If yes, I think the accuracy is unsatisfactory (see weekness2). If not, please explain in more detail and give the result of using BYOL directly.

clarity,_quality,_novelty_and_reproducibility: - Clarity and Quality: The paper is well organized.
- Novelty: The contributions are somewhat new. Aspects of the contributions exist in prior work.
- Reproducibility: The author provides the code. There is reason to believe that the paper is reproducible.
summary_of_the_review: The paper provides a new framework to model self-supervised skeleton-based action recognition. The paper provides extensive experiments for that and interesting observations and analyses showing the advantages of the proposed method. There are a few potential issues detailed in the previous section. Considering all the strengths and weaknesses pointed out in the Main review, I give the current rating.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This work introduces a self-supervised representation learning method for skeleton action data, which is based on BYOL and the hyperbolic space. The proposed model achieves state-of-the-art performance on three datasets. The idea of self-pacing for self-supervised  skeleton representation learning is interesting, and the authors designed a novel approach based on hyperbolic geometry to tackle this. All reviewers recommend acceptance for this paper due to the novelty and efficacy of the method. Authors need to improve the camera ready version following reviewers' comments.
justification_for_why_not_higher_score: Two reviewers pointed out that parts of the approach already exist in prior works. Hyperbolic space and BYOL have been used for skeleton action recognition before.

justification_for_why_not_lower_score: The overall method is interesting and novel. Also the results show the efficacy of such a design.
Decision: Accept: poster