Kati8/ICLR_2023_2025 huggingface dataset:
authors: Zhengsheng Guo
authors: Zhiwei He
authors: Wenxiang Jiao
authors: Xing Wang
authors: Rui Wang
authors: Kehai Chen
authors: Zhaopeng Tu
authors: Yong Xu
authors: Min Zhang
reviews: 
Comments: The paper studies a new setting for sign language understanding: unsupervised sign language translation and generation (USLNet), which exploits information from abundant single-modality but non-parallel data. More specifically, UNMT pretrains its text encodes/decoder and video/encoder decoder by reconstruction tasks. To address the misalignment issue between video and texts, the authors further propose a sliding window based aligner.
Strengths: 1. The idea is sound. Due to the data scarcity issue in the sign language understanding systems, it is important to explore non-parallel data.
2. The text-video-text and video-text-video back-translation strategies are novel in the sign language community.
3. Detailed ablation studies.
Weaknesses: 1. The major issue is the experimental setting.

1.1. Intuitively, leveraging abundant data should be helpful to the model performance. For example, in MMTLB (Chen et al., 2022), using a translation network pretrained on large natural language corpus can boost sign language translation performance. But I didn't see similar conclusions in the experiment section. In Table 1, the authors directly compare a supervised model with the proposed USLNet, and get a worse result on BLEU-4. I understand that the unsupervised performance must be worse, but what is this comparison for? I hope to see that for example, fine-tuning USLNet on parallel corpus can give better results, i.e., similar to the conclusion in MMTLB.

1.2. The performance is **too bad**. Although it may not be the authors' fault (maybe the dataset is too difficult), the poor performance make the comparison less convincing. Experiements on other widely-adopted benchmarks, e.g., Phoenix-2014T and CSL-Daily, shall be considered.

2. It seems that there is a factual error in the sliding window based aligner. The text and video are not monotonically aligned. In fact, only video and glosses are monotonically aligned, e.g., 1-10 frames for the first gloss, and 11-20 frames for the second gloss. Thus, it is questionable for the design of the sliding window-based aligner.

3. The descriptions for the process of the aligner should be more clear. The current form is a bit difficult to understand.

4. The process of two back-translation strategies are simialr to dual learning. The authors may consider adding a subsection in related works to discuss dual learning.
Recommendation: 5: marginally below the acceptance threshold
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 2 fair
Clarity: 2 fair
reviews: 
Comments: Inspired by the success of unsupervised NMT approaches, this paper proposes USLNet, an unsupervised SL translation and generation approach. USLNet has three main components, namely: text reconstruction module, video reconstruction module and finally cross-modality back translation module. The authors also propose a sliding window based approach to address the alignment issues that are inherent in broadcast SL datasets. The proposed approach is evaluated on BOBSL, however the reported results suggest the proposed approach does not meet the expectation of a translation system (~0.2 BLEU4 score).
Strengths: To the best of my knowledge this is the first bi-directional (translation/generation) SL approach that is trained in an unsupervised manner. Although the results are not promising, the proposed method is sound, and further studying the unsupervised training approach might yield promising results.
Weaknesses: Although I like the idea of using pretrained large-scale models and unsupervised learning, I'd expect quantitative results to back up the benefits of employing these ideas. Sadly, the presented results does not suggest the presented approach to be "working" (~0.2 BLEU-4 score on BOBSL, while the state of the art is above 2 https://openaccess.thecvf.com/content/ICCV2023W/ACVR/papers/Sincan_Is_Context_all_you_Need_Scaling_Neural_Sign_Language_Translation_ICCVW_2023_paper.pdf) 

That being said, the reviewers and the readers should acknowledge how challenging the BOBSL dataset is, and that we still need several breakthroughs to progress in large scale SL translation/generation. 

Therefore to strengthen the paper, I'd have considered/expected the following:

(1): Experiment on different datasets, such as Phoenix-2014T, or the larger OpenASL and YoutubeASL, which have more state-of-the-art results, hence more data points to gauge the performance/benefits of the proposed approach.

(2): Frame the approach as a pretraining method, and do a final supervised finetuning step (with varying amounts of data). One would expect the unsupervised pretraining on unaligned data to yield better performance than straightforward supervised translation approach, which would have strengthened the utility of the proposed method. 

(3) Having some qualitative results and failure analysis for translation/generation would have helped the paper immensely. Relying solely on b1 and b4 results does not give enough insights to the reader, and possibly is not doing the proposed approach justice. 

As is, I do not think the reviewer/reader has enough signals to evaluate the benefits of the proposed approach, and I'd highly recommend the authors to consider the suggestions mentioned above.
Recommendation: 5: marginally below the acceptance threshold
Reviewer confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness/correctness: 3 good
Clarity: 3 good
reviews: 
Comments: The model is proposed for cross-modal unsupervised learning. It focuses on unsupervised sign language translation and generation and it learns the task without requiring parallel sign language data. The model consists of four modules: text reconstruction,  video reconstruction, text-video-text translation, and video-text-video reconstruction.
Strengths: The overall writing quality is good although there are some issues.

The method is unsupervised which is important in the area as it requires experts to annotate. Also, inspired by unsupervised machine translation and applying the idea to another domain is the originality of the method.

The proposed methods support the writing with detailed formulation and figures.
Weaknesses: Discussion about existing text-to-video aligner algorithms is not sufficient. For example, although text2video[1] is a text-based talking face generation model, it uses an aligner for phoneme-to-pose. 

It seems back translations are highly similar to reconstruction loss that is used in image generation, especially in unpaired I2I tasks for cycle consistency. So you might consider elaborating this in the manuscript.

There are no visual results on the manuscript and limited visual results on the supplementary materials. I think it needs to be more convincing that the model is capable of generating sign language videos with high quality.


[1] Zhang, Sibo, Jiahong Yuan, Miao Liao, and Liangjun Zhang. "Text2video: Text-driven talking-head video synthesis with personalized phoneme-pose dictionary." In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2659-2663. IEEE, 2022.
Recommendation: 5: marginally below the acceptance threshold
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 2 fair
Clarity: 3 good