Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper authors propose the unit edit distance to measure robustness of self-supervised speech representations for spoken language modeling, and based on it, adding a multi-layer perceptron (MLP) trained using CTC to improve model robustness. Experimental results based on multiple self-supervised learning methods show effectiveness of the proposed approach.
strength_and_weaknesses: Strength: Overall, this propose approach is technical sound. Experiments and analysis are carefully designed for validation, including several widely used self-supervised learning representations. 

Weakness: This proposed metric and approach are mostly based on existing technologies (Levenshtein distance, MLP-based module, CTC loss etc). Also I think more details are needed for explaining design and results. (See questions in the summary section below).
clarity,_quality,_novelty_and_reproducibility: Overall, this paper clearly presents the proposed methodology. This work is reproducible with information presented in the paper. It is mostly based on existing technologies, with limited originality. 
summary_of_the_review: Overall, the methodology presented in this paper is technical sound, and the experiments are well designed to show its effectiveness. Several widely used self-supervised learning techniques are considered. This proposed approach is mostly based on existing technics (Levenshtein distance, MLP, CTC etc), with limited originality. Also I'd suggest authors consider addressing the following questions:

1. For the proposed unit edit distance, how to address class permutation (between the two inputs) when computing the Levenshtein distance? Also please double check if dimensions match for the composition of E, f and g in formula (1).
2. As said in paper, the UED monotonically increases with the number of units used. Are there thoughts on how to define UED more robust re the number of units?
3. In Section 4.1, it's said "learning the parameters of the encoder performs worse than freezing them." Is there a hypothesis on why this is the case?
4. Why are std only added for those results in Table 1, but not 2 & 3?
5. For "Our quantizer is composed of three fully connected layers" mentioned in Section 5.1, what are the layers' sizes?
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work looks into capturing the semantic information of self-supervised learning for speech. In particular the central idea is that the semantic of the speech should be robust to the small amount of noise in the speech, and therefore the robustness of a self-supervised learning algorithm toward noise should correlate with how well its learned representation captures the semantic of the speech. The approach uses speech augmentation to study the robustness of existing self-supervised learning algorithms, and proposes to apply a noisy-student approach with CTC loss to distill the quantizer. The experiments show the distillation approach provides a better quantization based on ABS test, spot-the-word test, and the acceptability judgment test, and there is some improvement on a downstream speech-to-speech translation task.
strength_and_weaknesses: Strength: the approach of using augmentation robustness as an objective for improving speech representation in self-supervised learning is well-motivated.
Weaknesses: both the noisy-student learning and the quantization distillation are existing approaches, and the idea of correlating semantics with noise robustness is also well known. This makes the novelty of the work limited. The experiments on downstream tasks are also insufficient to demonstrate the benefits of the approach.

clarity,_quality,_novelty_and_reproducibility: The article is easy to follow, but the novelty is limited.
summary_of_the_review: This work demonstrates that by applying noisy-student training on quantizers learned from self-supervised learning, the resulting student quantizer can provide some improvement. Overall the novelty of the work is limited, and there should be more experiments to demonstrate the benefit of the approach.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Setting: self-supervised representations for speech (spoken language modeling)

Tasks: speech-to-speech translation

The paper wants to improve self-supervised representations for speech.

Tests how variations of time-stretch, pitch-shift, additive-noise, and reverberation alter the learned representations. It should not really affect them, as those properties should not really be relevant, at least for most downstream tasks.

A new method using pseudo-labeling and CTC training is introduced to improve the robustness.

strength_and_weaknesses: 
Strength:

- Tests on time-stretch, pitch-shift, additive-noise, and reverberation and how they affect the learned representation.
- Pseudo-labeling using CTC, and an iterative variant improve speech-to-speech translation.

Weaknesses:

- The paper claims to study the self-supervised representation in general for spoken language modeling. However, in the end it really only tests it for speech-to-speech translation. This limits the scope of the paper. I would have expected tests for other downstream tasks as well, to really get an understanding on the robustness of self-supervised representation, as the title says.
- The actual tests on robustness are very limited. It just has a single experiment on speech-to-speech translation. I would have expected more experiments to verify the robustness aspect. Note that this is a separate weak point to the previous. Here in this point, I specifically mean that the robustness experiments are too limited. Of course, both are related. When experiments are done on other tasks as well, this would automatically extend also the actual tests on robustness. But there are potential other ways to extend such tests as well.

clarity,_quality,_novelty_and_reproducibility: UED definition 3.1: Where exactly is the deduplication in there? Function E outputs the same length as the input?

Figure 2, UED is in percentage? Or otherwise, shouldn't it be in the range 0 and 1?

Speech-to-speech translation results: I don't really see how they directly test the influence of their method. They just compare it to some result from the literature. But they should have a baseline without the proposed method, and then do a comparison with the proposed method, to directly see the difference.

Pseudo-labeling: It's not really clear to me: What are the targets for CTC? Those quantization indices, these found units? I think this should be made more clear. Further, this depends on a previous pretrained model?

summary_of_the_review: I think the scope of the work is too limited. When studying the learned representation, other downstream tasks should be tested, which would test the robustness much more directly, such as speech recognition.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper first studies how presence of different kinds of noise affect the discrete speech representations from self-supervised models used for spoken LM. The authors utilize the Levenstien distance between sequences of discrete token IDs obtained from HuBERT as an indicator for the presence of noise. Then, a pseudo-labelling task is introduced that aims to reduce this noise. The proposed techniques improve model performance on a variety of tasks.
strength_and_weaknesses: Strengths:

The paper is written clearly and is easy to follow. The proposed techniques themselves are simple yet effective for getting better discrete representations from speech. Evaluation is done on a variety of tasks and a significant improvement is achieved on all of them.

Weaknesses:

My main concern about this paper is how the premise is set. The main premise in this paper seems to be that spoken language modeling is sensitive to noise and discretized speech representations are not robust to it. My question is how often is noise a problem in speech generation/synthesis for downstream tasks like end-to-end speech translation. Maybe I am missing something but it seems that speech processing for spoken LM is not subject to noise from outside as it processes its own generated speech and not human speech. Also, was there a source of noise in tasks that involve autoregressive generation like E2E S2S translation?

Maybe I am missing something here, but it seems that the UED score analysis does not add anything noteworthy to the paper. I understand that it shows how a discrete representation of speech may change in the presence of noise but it’s not very surprising that it changes as representations always change in the presence of noise. The question is how much is the downstream task affected by these noises. There are no experiments which show the effect of artificial noise on downstream tasks but only UED is used to show the effect which is rather obvious. The fact that UED improves after a teacher-student like training with CTC loss is also not surprising. If a UED score is high, does it necessarily mean that one representation is better than the other?

clarity,_quality,_novelty_and_reproducibility: The paper is clearly written for the most part and doesn’t have significant quality issues. The paper has moderate novelty to the best of my knowledge. It is not clear if the authors have or intend to release their code for reproducibility.

summary_of_the_review: I found the CTC based pseudo-labelling and its iterative counterpart to be quite ingenious and interesting. It improves the overall discrete representation quality of the speech through a teacher-student like framework. But I do question how the story is set up by showing that representations change by addition of synthetic noise through the use of UED. I found that part rather redundant. Maybe the authors can convince me as to why it’s important to this paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper studies the robustness capabilities of self-supervised speech representations for generative tasks, in particular spoken generative language models. The authors present a method for a teacher-student like training with CTC loss, and a metric for robustness evaluation. Experiments are conducted on a speech-to-speech translation task. The paper is clearly written and easy to follow. At the same time, the paper is considered by the reviewers as having moderate novelty and limited experiments.  
justification_for_why_not_higher_score: Limited novelty and experiments.
Decision: Reject