Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper aimed at alleviating the acoustic multimodality problem for speech-to-unit approaches for direct speech-to-speech translation (S2ST) system. To address this problem, the authors developed bilateral perturbations (BiP) and leveraged the CTC fine-tuning technology. The bilateral perturbations include information enhancement on CTC input and style normalization on CTC target. Information enhancement applies information bottleneck on acoustic features to create speech sample variants in different acoustic conditions (considering rhythm, pitch, energy) while preserving linguistic content. In contrast, style normalization aims to eliminate acoustic-style information and creates acoustic-agnostic “pseudo text”.  After the bilateral perturbation, ASR system with CTC decoding would do better to generate deterministic representations, hence alleviating the acoustic multimodality problem.  The whole idea of bilateral perturbation and leveraging CTC fine-tuning is simple and intuitive. Analysis of unit error rates showed effectiveness of proposed Bilateral Perturbation in reducing acoustic multimodality. With bilateral perturbation, the authors developed NAR S2ST (TranSpeech), while applying the standard KD technology in NAR to reduce linguistic multimodality and prior work of mask-predict NAR unit decoder. Experiments showed effectiveness of BiP over AR baseline system (basic conformer) by 1.2-4.4 BLEU points and NAR baseline system (Transpeech-Distill) by 0.6-1.8 BLEU points. 
strength_and_weaknesses: Strengths:

(1) The proposed bilateral perturbation approach for CTC fine-tuning to address the  acoustic multimodality problem is simple, intuitive and novel.

(2) With the reduced acoustic multimodality, the paper explored NAR for direct S2ST. As shown in the results, NAR TranSpeech with KD achieved significant speedup over AR direct S2ST without a big degradation in translation accuracy.

(3) The authors investigated a bag of tricks proposed in prior works, including conformer, KD for NAR, mask-predict NAR unit decoder, and other decoding choices including target length beam and NPD. Experiments evaluated efficacy of these tricks and analyzed performance-speed tradeoff, which are useful for future NAR efforts on direct S2ST. 

(4) Experiments and analyses are comprehensive. Experimental results on three language pairs of CVSS-C showed good gains in BLEU from the proposed BiP approach for both AR and NAR direct S2ST.  Analysis of unit error rates showed effectiveness of proposed Bilateral Perturbation in reducing acoustic multimodality.  The authors also compared to cascaded systems, which shows that there is still a lot of room for improvement for direct S2ST systems.

Weaknesses:

(1) The paper is overall clearly written. However, some technical details and discussions are missing and need to be included. Please check out the questions under Clarity.

clarity,_quality,_novelty_and_reproducibility: Clarity and Quality: 

The paper is generally clearly written, but some technical details and discussions are missing and need to be included:

(1)	The style normalization stage did not consider normalizations in rhythm. This needs to be explained.

(2)	The HuBERT representations are pre-trained on natural speech. It would be useful to explain the quality of encoded representations by applying pre-trained HuBERT encoder on the style-normalized speech (whether encoding may suffer from reduced naturalness in the style-normalized speech).

(3)	For pitch information in Information Enhancement, please elaborate the motivation for the choice of the chain function in this order, instead of other orders or other designs. Appendix C only provides details on each function, still misses answers to this question.

(4)	For energy information in Information Enhancement,  please clarify how the average from a log-mel spectrogram is used.

(5)	In the ablation analysis, it would help to add ID=4 results in Table 2. Based these results, it seems that adding SN on top of Basic Conformer + IE only brought quite a small gain on BLEU, as 0.41, 0.05, 0.03 respectively. This discussion needs to be added and more insights on how to improve effect of SN are expected.

(6)	Note that there are some typos, for example, Section 5.1, “BELU” should be “BLEU”.

Novelty: 

The proposed BiP approach is novel, simple, and intuitive. Experiments showed effectiveness of BiP in reducing acoustic multimodality and improving translation accuracy. The NAR direct S2ST achieved significant speedup over AR direct S2ST without a big gap in translation accuracy.

Reproducibility:  

Although no source code is released, Section 5.1 and Appendix B and Appendix C provide sufficient details for reproducibility.


Presentation: 

There are some grammatical errors. For example,
(1)Introduction, “Among the conventional method”, method -> methods
(2)	Introduction, “recently proposed direct S2ST literature demonstrate…”,  demonstrate -> demonstrates 
(3)	There are quite a few subject-verb agreement errors.
(4)	Section 2, “demonstrates its outperformed translation results…”, outperformed -> outperforming

summary_of_the_review: This paper proposed a simple and intuitive bilateral perturbation (BiP) approach leveraging CTC fine-tuning for tackling acoustic multimodality in unit-based direct S2ST. The proposed BiP approach is shown to reduce acoustic multimodality. With this approach, the paper built non-autoregressive direct S2ST, TranSpeech, and achieved significant speedup over AR direct S2ST without a big degradation in translation accuracy. The authors also investigated other prior approaches such as conformer, KD, mask-predict unit decoder and other decoding choices for TranSpeech. The experiments are comprehensive. The proposed BiP approach achieved good BLEU gains over both AR and NAR baselines.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work proposes two approach for an end-to-end speech-to-speech (S2S) translation: bilateral perturbation to focus the learning step on the linguistics information from the speech (i.e., disregarding acoustic properties such as rhythm and pitch that potential affects the consistency of the translation), and a non-auto-regressive unit decoding mechanism (mask-predict). Experimental results show the two approach to improve translation quality and unit decoding latency.   
strength_and_weaknesses: Strengthes
- The proposed style normalization step (that removes acoustic style information), and the enhancement step (that creates variable speech samples without affecting the linguistic information), are the main strength of the proposed S2S model. In addition to generate deterministic representation of the speech signal, the approach can be applied in several use cases as it will allow to better exploit speech representation learned from a self-supervised pre-training step incorporating both linguistic and acoustic signals. 
- Experimental evidences show the proposed approach (I and II), improves translation quality and a significant decoding latency. These improvements can enable the proposed approach to have multiple application use cases (including simultaneous S2S translation).  


Weaknesses
- Although, the proposed approaches demonstrate improvements for the respective metrics (quality and latency), its is hard to connect the two approaches (or at least it is not well motivated in the paper). I understand, the perturbation approach delivers by making NAR decoding of units in S2S translation is relatively easier, given the simplification on the acoustic features. However, its still unclear which is the main theme of the paper (I will assume the NAR approach is a by-product). 
- Experiments from a single benchmark - given the focus on the acoustic attributes, dataset from variable benchmarks (potentially with high variability in the acoustics information) could provide more strength for the experimental section.        
clarity,_quality,_novelty_and_reproducibility: Clarity
- The paper is well written, and understandable with enough background. The scope covering basic definitions of linguistics and acoustic information, illustrating and discussing the proposed approach (Figure 2), through the experiment and discussion part keeps the flow. 

Quality
- There are instance, that require a proof reading, to tune some of the parts of the work (for instance "The
parallel decoding algorithm requires as few as 2 iterations to generate outperformed samples," --> ... to generate samples that outperform ... ?). 

Novelty 
- The proposed approach follows previous work (see Figure 1a), and different from related work, the proposed approach explores more variants of acoustic features (rhythm, pitch, and energy) of the speech representation.
- NAR decoding

Reproducibility
- Model parameters are provided
- 
summary_of_the_review: This work shows interesting results for a S2S translation task, for quality and decoding latency. The proposed approach addresses the modeling of more acoustic attributes that are not explored in previous work. This work allows for a better use of pre-trained self-supervised models, by generating a deterministic representation. Despite the improvements, the experiment setup is narrow and the motivation to combine quality improvement with latency is poorly stated.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper address the issues in the recent proposed S2ST framework and propose to solve the problems with two different aspect.
First of all, the paper proposes to alleviate the acoustic multimodal problem by using bilateral perturbation.
Obvious improvement is achieved with this method.
Next, the paper proposes to solve the high latency problem by introducing a non-autoregressive approach.
This leads to a significant speed up compared with the baseline.
strength_and_weaknesses: Strength:
This paper propose two methods to improve the translation accuracy and the inference speed of S2ST systems. Significant improvement is achieved.
It is the first implementation of non-autoregressive S2ST

Weaknesses:
Here are my questions to the paper: 

The author should check the definition of the word "multimodality" and the common usage of this word in the community.

"The proposed bilateral perturbation eases acoustic multimodality and makes it possible for non-autoregressive (NAR) generation" is stated in the introduction. Why the method of bilateral perturbation and the NAR issue is related? 

"the indeterministic training target for speech-to-unit translation fails to yield good results" is stated in section 3.1. What is the reason behind this judgement? The units, which generated from the speech with an unsupervised way, contain noise and it was known since HuBERT. The vocoder behind the S2UT should be able to cover this and generated the target waveform. In other way round, this might be a good thing as the acoustic variation is preserved.

How much BLEU is sacrificed from moving an auto-regressive system to a NAR system?

The major aim of unit-based S2ST is to deal with unwritten languages. 
Without this motivation and without the use of discrete units, I would consider the unit-based S2ST approach as a cascade of speech2text translation and TTS. 
While this paper is claiming their work on the foundation of unit-based S2ST, they are not evaluating  their models with unwritten language translation task.
I doubt that the comparison in Table 2 is not fair as the strength of the baselines should be on unwritten language tasks.



 


 

clarity,_quality,_novelty_and_reproducibility: Clarity: The writing is ambiguous and hard to read. E.g. the first sentence in the abstract.

Quality: Standard.

Novelty: Significant.

Reproducibility: The codes are not provided in the paper.


summary_of_the_review: I would recommend the paper to be accepted if my questions are settled.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: ### Overview

The paper aims to generate much deterministic representation for end-to-end speech translation by characterizing style normalization and information enhancement. Overall, I like the paper on incorporating those ideas for S2ST tasks. 

In general, the author's design of both style normalization and acoustic information enhancement methods could be considered as one attribute-based enhancement sharing with some similar findings on latent space prompting or speech input reprogramming via perturbation to drive the model under different models and conditions. Both works are related to characterizing the perturbation for fine-tuning tasks. 

The authors could add some discussion on those fine-tuning pre-training AM works via perturbations to improve the overall impact of this work. 

I strongly recommend accepting this work. 
- Original 6; to 8 after rebuttal. 


***

**References**

A. Voice2series: Reprogramming acoustic models for time series classification, ICML 21

B. WAVPROMPT: Towards Few-Shot Spoken Language Understanding with Frozen Language Models, Interspeech 2022




strength_and_weaknesses: ### Pros.

1. Startforward but effective approach via the perturbation with fine-tuning CTC 

### Cons.

1. Some related perturbation-based references are missing. 

2. Since Rhythm, Pitch, and Energy have all been used as information enhancement. The impact of each terms are not very clear. 


### Questions. 

1. It is not clear to me that the "ASR model with CTC decoding" has been used as one major setup in this work. Is that any difficulty in using the seq2seq prediction model under the proposed scheme?

2. I heard some high-pitch background noise for Fr-En (https://transpeech.github.io/) examples. Could the authors provide some SNR or speech quality assessment since additional information has been introduced in the proposed method?

***

### Post-Rebuttal

I read the authors' updated version and reply. 

- In general, my Q1 (seq2seq modeling) and Q2 (SNR) have been addressed. The authors have also stated a code release in their future version. 

- After re-reading, I think the major contribution (normalization and enhancement via perturbations) in section 3 is still a little blur in terms of writing since the speech module and notations are defined in their section 4. The authors could consider refine the proposed method in section with mathematical notation (e.g., $X$ and $p$) to improve their future clarification for general readers. 

- I updated my score to accept. This is a ~8 score paper to me. 
clarity,_quality,_novelty_and_reproducibility: Overall the paper with good clarity. Some motivation for the perturbation based fine-tuning could be included. 

In general, there are yet theoretical discussions on the fine-tuning process. But some connections to the previous perturbation-based theoretical study [A] could be included. 

The authors do not mention their code and license. I give a question mark on the reproducibility. [updated: authors state code is to be released. The issue seems to be resolved.]
summary_of_the_review: In general, it is one good work on tackling characterized perturbation and S2ST. Some references are missing but in general, the area of fine-tuning SSL with a customized approach is still new and worthy of more studies.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper aims to generate more deterministic representation for direct speech-to-speech translation (S2ST) system, where the system often suffers from various acoustic conditions (e.g., rhythm, pitch, energy) that make generating consistent semantic units challenging. The authors propose bilateral perturbation to address the challenge, where information enhancement and style normalization are leveraged such that CTC decoding can generate units that preserve linguistic information and be invariant across speech sample variants in dierent acoustic conditions. The author also implemented S2ST with non-autoregressive unit decoding mechanism. Efficacy of proposed approaches are showed empirically with several well-known benchmarks in translation quality and unit decoding latency. 

Strength of the paper:
1. Proposed approaches are simple, intuitive and novel.
2. Authors proposed a bag of tricks and show the efficacy of those tricks empirically in performance (BLEU scores) and translation latency. Conducted experiments and analyses are comprehensive and allows readers to understand the effectiveness of each trick.
3. Beyond S2St, proposed approaches can benefit other applications that required units stably encoding semantic information, such as ASR. 

Weakness of the paper:
1. Some technical details and discussion are missing (e.g., the connection between bilateral perturbation and NAR, why indeterministic units are harmful for performance, how much BLEU is sacriced from moving an auto-regressive system to a NAR system, why the authors choose to experiment with main stream languages such as English, even though S2ST is most beneficial for languages that can't be written). The authors did a great job in discussion phase, and provided many additional experiment results and explanation. Most of the concerns are mitigated, if not  cleared. However, the connection between bilateral perturbation and NAR approaches are not as strong as the reviewers/AC are hoping to, which makes the work be viewed more as a bag of (effective) tricks, instead of one clean framework. Lack of one clean framework could limit the impact of this paper.
2. Reviewers and AC found "multimodality" used in the paper (refer to variations) is a bit different from how it's generally interpreted in community (mulitiple sensory or channels of information, such as text, audio, image)

justification_for_why_not_higher_score: - See the bullet 1 of weakness. The connection among proposed approaches (i.e., bilateral perturbation and NAR) are not as strong as the reviewers/AC are hoping to. The lack of strong connection makes the work be viewed more as a bag of tricks (yet effective), instead of one clean/simple framework, which could limit the impact of this paper. 
- Also the proposed approaches can benefit S2St and several speech applications that require semantic units, such as ASR. However, how can the proposed approaches and experiment results be generalized to other community is an open questions.
 
justification_for_why_not_lower_score: - See the strength of the paper for more details. The paper proposed several very helpful approaches for S2ST and potentially beneficial for ASR and other speech applications. The experiment and analysis are comprehensive. The reviewers and AC have confidence that accepting the paper in ICLR would benefit the community and drive research further.
Decision: Accept: poster