Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a two-stream model including the raw videos and the keypoint sequences for sign language recognition and sign language translation. The two-stream model also utilize a variety of techniques, including bidirectional lateral connection, sign pyramid network, and frame-level self-distillation to further improve their model. Empirically, the two-stream model outperforms the previous best method.
strengths_and_weaknesses: Strengths: 
1. This paper takes advantage of the keypoint sequences and a variety of techniques and achieve excellent results.
2. This paper is well written.
Weaknesses:
1. It may be confused that why the keypoint sequences can achieve such an effect improvement.  
2. This paper introduce the robustness of previous models, which suffer from dramatic performance degradation when the background or signer is mismatched between training and testing, but is the two-stream model can solve this problem?
questions: 1. It may be confused that why the keypoint sequences can achieve such an effect improvement.  
2. This paper introduce the robustness of previous models, which suffer from dramatic performance degradation when the background or signer is mismatched between training and testing, but is the two-stream model can solve this problem?
limitations: The authors have addressed the limitations of their work.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: In the SLR task, the paper introduces two separate streams to model both the raw videos and the keypoint sequences. For the two streams to interact with each other better, the paper proposes a variety of techniques, including bidirectional lateral connection, sign pyramid network, and frame-level self-distillation. In the SLT task, the paper extends TwoStream-SLR to TwoStream-SLT by attaching an MLP and NMT. Experimental results show that TwoStream-SLR and TwoStream-SLT achieve SOTA performance on SLR and SLT tasks in three datasets.
strengths_and_weaknesses: Strengths:

1. The paper proposes Two-Stream Network, including bidirectional lateral connection, sign pyramid network, and frame-level self-distillation methods, to interact RGB videos and keypoint sequences for advancing SLR and SLT. 
2. The paper reported an improvement on SLR and SLT tasks.

Weaknesses:

1. These lack the evaluation of visual redundancy and interact with both the raw videos and the keypoint sequences.

2. It is unclear about the usage of domain knowledge and key sequences.

questions: 1. The logical coherence of the abstract needs strengthening. For example, in line4-line7, the link between the questions about visual redundancy and the need to incorporate domain knowledge is rather jumpy, thus affecting the fluency of the reading.
2. In the 3.1 section, the details of the selection of the key point are not clear enough. Why were the 79 key points chosen and what were the reasons? The choice of key points was not explored further in the experiment part.
3. In the SLT experiments, in past studies, the results of experiments with added gloss supervision were higher than the gloss-free method. However, the opposite results were obtained in this experiment. It is necessary to analyze and explain the reasons for this. 
4. Suggest that some methods of the visual field need to provide an explanation to increase the readability of the paper.
limitations: N/A
review_title: null

final_score: rating: 9: Very Strong Accept: Technically flawless paper with groundbreaking impact on at least one area of AI/ML and excellent impact on multiple areas of AI/ML, with flawless evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a two-stream network for sign language recognition and translation. The main idea behind this paper is using two separate S3D networks to encode RGB modality and human keypoint modality respectively according to the fact that sign languages use both manual articulations and non-manual elements to convey information. To make the two streams interact with each other, authors propose bidirectional lateral connection, sign pyramid network with auxiliary supervision, and a frame-level self-distillation strategy. Elaborate ablation studies have verified the effectiveness of each proposed component. The results are very solid. TwoStream-SLR which is designed for sign language recognition task, achieves 18.8 WER on Phoenix-2014, 19.0 on Phoenix-2014T and 25.3 on newly published CSL-Daily, which greatly outperforms prior methods by large margins. As for TwoStream-SLT, it also achieves  state-of-the-art performance on Phoenix-2014T and CSL-Daily datasets.
strengths_and_weaknesses: Strengths:
1.	This paper is very well written and easy to follow. The motivation is very clear.
2.	The proposed TwoStream network with bidirectional lateral connection, sign pyramid network, auxiliary supervision and frame-level self-distillation is technically sound. I believe this paper will facilitate this research direction.
3.	System-level experiments are very solid. This paper achieves state-of-the-art performance on two sign language understanding tasks, i.e. continuous sign language recognition and sign language translation across several datasets. It is worth mentioning that TwoStream-SLR outperforms previous best methods by large margins, especially on CSL-Daily dataset.
4.	A variety of ablation studies verify the effectiveness of each proposed component, as shown in Table 3 and 4.(a)-(d), as well as the Tables in appendix.

Weakness:
1.	Authors could move the formulation of CTC loss and translation loss from Appendix to the main paper to make the paper clearer. 

questions: 1. Self-distillation loss brings improvement as shown in Table 3. Can authors study the weight of this distillation loss in Eq 1. 
2. As described in L169, authors utilize a Gaussian function to generate keypoint heatmaps. I don’t see studies about sigma, as well as the resolution of heatmaps. I hope authors could provide these studies since how to model keypoints is a key contribution, adding these experiments will make the submission stronger.

limitations: Due to the data bias and data scarcity issue, there are unpredictable recognition/translation errors as shown in Table 4 in Appendix. Authors also discuss the limitations and societal impact adequately.
review_title: null

Meta-review: metareview: This paper extends models for sign language recognition and translation with a dual encoder where, first, keypoint sequences are estimated using an off-the-shelf model, then fused with the video sequence. It is a minor technical contribution to add the keypoint estimations as input since no new information was introduced; however, the authors demonstrated strong execution of experimental results. This paper can be categorized with pipeline/cascade approaches which rely on domain knowledge for engineered feature extraction and combination. 
The paper presents many experimental results for architecture changes to improve results: bidirectional lateral connection, sign pyramid network, and frame-level self-distillation. 
The authors convinced the reviewers with more experimental results during the rebuttal period leading to two solid, and one borderline accept votes.

confidence: Less certain
Decision: Accept