Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: null
review_content: Apologies for both the brevity and the last-minute nature of this review.

First and foremost, I applaud the authors for this resource. Working with minoritized and under-resourced communities always introduces ethical challenges that can be avoided by sticking to well-trodden ground, and unless more researchers take up the challenges (as these authors have done) to contend with the difficulties implicit in working with scanter sources and challenging power dynamics, the degree to which these communities are ignored and under-resourced -- or even shut out of public imagination -- will only grow. It's great to see work that tackles those challenges so thoughtfully and responsibly.

It's worth noting that acknowledging the YouTube creators (which is the just and responsible thing to do, and will give the dataset further credibility as being for the community rather than an extraction project) also makes those creators slightly more vulnerable to de-anonymization. This is mentioned not as a problem that the authors can or should solve -- as noted, I think that acknowledging those creators is the right thing to do, and there is no halfway version that extends recognition to these creators without making them available for a different kind of recognition. Rather, I want to highlight this as a balancing concern for future developments or iterations of this project; there might be an instance in which the risks to creators outweighs the benefit of being acknowledged. 
review_title: brief, last-minute ethics review

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: summary and contributions: This paper provides three sign language datasets which by itself is already useful (sign languages are largely under-resourced/under-studied). The authors also provide in-depth analysis of the extraction process, data attributes and a few baseline accuracy numbers using a graph-convolution network. Overall the paper is very well structured, easy to follow and critical for sign language ML research in my opinion. 
strengths: (1) SignCorpus is the largest ever collected sign language pretraining dataset. As the authors show this is very useful for finetuning neural network models for multilingual sign identification and finger spelling. I think there are plenty of other applications for such a pretraining dataset, for ex: for unsupervised MT into sign languages or even simply for augmenting sign language training with generated signs/poses.
(2) Well explained data collection process and solid experiment results for benchmarking.

weaknesses: The only weakness I see is that a lot of manual effort went into annotating the dataset by the authors/researchers themselves. Now that they have decent ML models for sign language prediction,  I wonder if this can be used to speed up the data collection process itself (some kind of life-long learning paradigm). Another related question is how large can such a dataset be given that sign language speakers are a minority. Are there enough youtube videos out there to build larger models? We may need to use different approaches depending on the answer to this. 
correctness: The claims made by the authors seem correct and the data looks sound.
clarity: The paper is well written and structured (even the appendix).

relation to prior work: Introduction and related work sections provide a good summary of prior works including datasets, models and results.

additional feedback: N/A
review_title: Important sign language dataset!

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper focuses on resource scarcity across sign languages and makes the following datasets contributions:
- Release a multilingual sign language dataset SignCorpus comprising about 4.6K hours of signing data across 10 sign languages.
- Create a multilingual and label-aligned dataset MultiSign-ISLR of sequences of pose keypoints from 14 labelled datasets across 7 sign languages.
- Create a new finger-spelling dataset MultiSign-FS across 7 sign languages.

Additionally, the authors release a pre-trained sign language model Sign2Vec pre-trained with the dataset SignCorpus for downstream sign language tasks and demonstrate its effectiveness on MultiSign-ISLR and MultiSign-FS.
strengths: 1. The dataset contributions are sufficient.
2. Useful for the group of the Deaf and Hard of Hearing (DHH) and positive social implications.
weaknesses: 
1. The datasets are only partly uploaded.
2. Lack some manual evaluations for dataset quality.
correctness: Yes.
clarity: Yes, well written.
relation to prior work: Yes, clearly discussed.
additional feedback: 1. Suggest adding some manual evaluations for dataset quality, such as the accuracy of image to pose keypoints conversion
review_title: Review

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: summary and contributions: A hard, bold and much needed contribution for Natural Language Processing on Sign Language.
The authors 
strengths: The data gathering procedure is well described and justified, using techniques inspired of other tasks, datasets and models that have similar obstacles, such as low resource.
The model architecture of the Sign2Vec pre-trained model is well described and justified.
The experimental results show realistic improvements on multiple benchmarks and competitive results on the others. 
Given that the task is difficult, understudied and it's medium and representation is demanding, we find this bold try a success.
weaknesses: As the authors mention, there are still many other pretraining strategies and model architectures to be tested.
correctness: The claims in the papers are well justified and explained; the step descriptions are correctly specified and detailed and the experiments are reasonable.
clarity: The paper is clearly written, well explained and easy to follow.
relation to prior work: Related work and prior work from the authors and other teams is clearly presented, discussed and referenced.
additional feedback:
review_title: A hard, bold and much needed contribution for Natural Language Processing on Sign Language.

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The authors propose a new sign language dataset that contains sign language videos in different languages. The authors also propose a novel network,  Sign2Vec, which is a large-scale pretrained model for sign language tasks. 
strengths: 1. The proposed dataset is of great interest to the community.
2. The proposed technique can improve the state of several downstream tasks.
3. The paper is well written, and the data collection technique is clearly mentioned
weaknesses: No major weaknesses. 
The paper should cite BOBSL and How2Sign datasets too.
correctness: The paper is technically sound
clarity: The paper is well written
relation to prior work: The paper covers most of the related work. But there is no mention on How2Sign and BOBSL.
additional feedback: None
review_title: Unified Vocabulary Dataset for Sign Languages

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: summary and contributions: This submission introduces a multilingual benchmark of signed languages, including both local spoken language glosses and a unified multi-SL to English gloss annotation for single-language translation/decoding. The benchmark includes both fingerspelling (boring, but more common and easy; usually not multilingual!) and sign recognition (cooler, harder, usually not multilingual!) /and/ sequential signing (hardest, realest, usually not multilingual). The benchmark tasks are sign recognition (cool!) and fingerspelling (yawn, but we "get it").

strengths: The released data is large scale and the benchmark unifies the sign recognition task across multiple signed languages. The benchmark is necessary and timely.

weaknesses: The key weakness of the submission is the lack of human toplines for native signers for each of the 10 languages considered attempting the task on the pose information alone, which is what models have to work with. In particular, the extracted poses may be noisy or inaccurate, and so we need to know whether a native signer is able to translate the extracted poses released as the dataset into the target local language gloss / identify the SL sign.

A methodology question: For "MultiSign-ISLR" what about signs that don't have a clear, single meaning in the common label set? Are these thrown out? Problem not acknowledged?

A metholdogy question: Section 3.3 filtering does not explicitly check that videos labeled with a particular sign all actually depict that sign being surfaced. In automated scrapes of data, it is exceedingly easy to get noisy productions of neighbor signs or signs embedded in phrases. Can the authors expand on how they ensured that each labeled video depicted the sign in isolation (when necessary) and that all labeled videos are in fact depicting the same underlying sign?

correctness: The experiments are complete and include both existing state of the art sign rec models and a new pretrained model based on the large, released corpus.

clarity: There are some typos but the core arguments and presentation of the paper are clear.

relation to prior work: Yes, the work is well-positioned.

additional feedback: "Sign2Vec as a multilingual foundation model" the reviewer requests the authors to re-consider using the term "foundation model" in one of two ways. Either (1) cite the Stanford paper that establishes this term or (2) avoid using the term because it is a Stanford power-grab to make the "Stanford Center for Foundation Models" synonymous with large-scale pretraining, which was not invented at Stanford. The reviewer favors option (2), but if you're going to call your model a "foundation model", cite the paper that introduces the terminology, because it is non-standard. Sign2Vec is a large-scale, multimodal pretrained model, which you can also be proud of, mouthfull-tho-it-may-be.

Typo line 38 "1,00,000 signs"

Line 75 and elsewhere citation should be parenthetical but appears as noun. Line 132 (and elsewhere) opposite problem, citation appears parenthetically that should be compiled as a noun. 

Super weird spacing around the numbers in line 271; maybe just drop the commas in these.
review_title: Necessary and timely benchmark

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: In this work, several sign multilingual datasets are introduced, described, and modeled: 1) SignCorpus – dataset with ten languages represented by sequences of pose keypoints for SignCorpus 2) Sign2Vect – model pretrained on SignCorpus, 3) MultiSign-ISLR - dataset with pose sequences of seven languages, 4) MUltiSign-FS- finger-spelling dataset of seven languages, 5) fine-tuned Sign2Vect models on 7 languages. The submission refers to OpenHands library, developed and presented at 60th Annual Meeting of the Association for Computational Linguistics (Selvaraj et al.; doi: 10.18653/v1/2022.acl-long.150), and it seems that most work that would be interesting for sign language – related researchers, which is related to the dataset preparation, including carving the data, labeling, and determination of pose keypoints, has been already done while preparing the paper of Selvaraj et al. The prepared models are of lesser importance since many neural network-based solutions are introduced each year; hence, the usage of pretrained Sign2Vect would be limited in the next few years. The recent research is more focused on continuous sign language recognition models. This subject is neglected in this submission, limiting the usability of the models.


strengths: The sequences of pose keypoints for multilingual videos are the greatest contribution of this work. The developed models, including pretrained or fine-tuned models, would interest some researchers.  
weaknesses: Most sequences of pose keypoints for multilingual videos were created in previous authors' work (Selvaraj et al.; doi: 10.18653/v1/2022.acl-long.150). The developed models, including pretrained or fine-tuned models, would be of inferior interest to the research community as most researchers develop their own models and there is no consensus on the way the neural network for sign language recognition should look like. The lack of video information would confine the users of the dataset to a set of keypoints produced MediaPipe Holistic software. The usage of software allows for maintaining the anonymity of the signers. However, it is not clear whether authors were granted permission to use and process videos crawled from Internet sources. The recent research is more focused on continuous sign language recognition models. This subject is neglected in this submission, limiting the usability of the models. Since the accuracy of some models is presented, the submission could be treated as a benchmark for new models, but it should be described differently in this case.
correctness: Yes, the submission is correct. The submission is a dataset and it contains necessary information. However, since the accuracy of some models is presented without detailed comparison and an explicitly defined data division into training and testing subsets, its usage as a benchmark requires further work. 
clarity: The introduction of the dataset is quite chaotic and poorly laid out as the number of videos, justification of the number of used languages, the addition of finger-spelling, data origin, extraction, and models are introduced revealing many details that at first hinder the justification and purpose of the dataset. The paper requires proofreading. Some abbreviations are used before the first introduction (e.g., PII in Abstract).
relation to prior work: It is written that “in [Selvaraj et al., 2022], the training was done only on Indian Sign Language (ISL), while this work expands the training to 10 sign languages and increases the dataset size by 4×. As we show, this increased diversity of pretraining results in improved accuracy.” However, this is not a research paper in which neural network models are introduced but a dataset paper. And, looking at the content of the [Selvaraj et al., 2022], it seems that most efforts with data curation, crawling, and labeling were put while preparing previous work. Here, some additional languages and finger-spelling datasets are new, which cannot be seen as a novel contribution but rather an extension of the existing and published dataset. It is not true that “For Indian Sign Language (ISL), in our earlier work we presented a pretraining corpus of 1,129 hours [Selvaraj et al., 2022].149 Hence we focus on the remaining 9 languages in this work.” since in previous work results for several (other) languages are shown, indicating that the pose data for them was prepared for experiments. A clear distinction between what was done previously and where is not shown.


additional feedback: The dataset is not balanced. This makes its usage for training and evaluation of different models more complex. The implications of the structure of the datasets are not discussed in the paper.  Also, it would be nice to see the addition of which dataset in the fine-tuning is the most contributing. And the investigation of why the MFT model decreased the accuracy (Table 4) in some cases requires further attention. However, taking into account above suggestions would transform the dataset into benchmark submission, which, in my opinion, would be more interesting for researchers, as the proposed pretrained models would be rarely used by other researchers and the benchmark already has such potential.
review_title: Set of multilingual sign language databases and pretrained models with significant limitations

Meta-review: metareview: This paper presents a significant collection of unlabelled signing data (4.6K hours) across 10 sign languages, which have been pre-processed and converted to pose-keypoints to remove identifiable information. The paper also presents Multi-ISLR a multilingual dataset with label-alignment extracted from 11 other labelled datasets for 7 sign languages. Furthermore, the paper also provides a multilingual model, Sign2Vec, pre-trained on the unlabelled data, and fine-tuned on the labelled data, which shows the SOTA results for known tasks. Lastly, the paper also introduces a dataset for fingerspelling across 7 languages. It's likely that this work will pave the way for new research on multilingual signed language tasks.

**pros:**
* The paper provides a significant expansion for unlabelled and labelled signed data (previous work focused only on Indian Signed Languages)
* The data gathering procedure is well documented and justified
* The experimental results provide evidence of the value of the data.


**cons**
* For the SignCorpus there is a lack of a human verification process, that assesses the quality of the pose keypoints; i.e. whether they can be used by native signers to extract the target gloss.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept