Learning Monotonic Attention in Transducer for Streaming Generation

Authors: Zhengrui Ma, Yang Feng, Min zhang
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: streaming generation, simultaneous translation, Transducer

Abstract:
Streaming generation models are increasingly utilized across various fields, with the Transducer architecture being particularly popular in industrial applications. However, its input-synchronous decoding mechanism presents challenges in tasks requiring non-monotonic alignments, such as simultaneous translation, leading to suboptimal performance in these contexts. In this research, we address this issue by tightly integrating Transducer's decoding with the history of input stream via a learnable monotonic attention mechanism. Our approach leverages the forward-backward algorithm to infer the posterior probability of alignments between the predictor states and input timestamps, which is then used to estimate the context representations of monotonic attention in training. This allows Transducer models to adaptively adjust the scope of attention based on their predictions, avoiding the need to enumerate the exponentially large alignment space. Extensive experiments demonstrate that our MonoAttn-Transducer significantly enhances the handling of non-monotonic alignments in streaming generation, offering a robust solution for Transducer-based frameworks to tackle more complex streaming generation tasks. Codes are publicly available in supplementary materials.

Decision: reject
Reviewer scores: 3, 6, 5, 5

Reviews and discussion:
<thread>
creation date:
1730454741492

thread_type:
official_review

<thread_object>
creator: reviewer_2lwh

<content>
summary:
This paper presents a solution to the challenges faced by Transducer-based streaming generation models in handling non-monotonic alignments, particularly in simultaneous translation tasks. The authors introduce a learnable monotonic attention mechanism that integrates with Transducer decoding. Utilizing the forward-backward algorithm, they infer alignment probabilities between predictor states and input timestamps, allowing adaptive adjustments in attention scope. Experimental results show significant performance improvements of the MonoAttn-Transducer, offering a robust method for complex streaming generation tasks. Their code is publicly available.

strengths:
1. This paper improves the Transducer architecture, making it more suitable for streaming generation tasks.  
2. The experiments demonstrate that the proposed method achieves commendable performance in the Speech-to-text Simultaneous Translation task.

weaknesses:
1. The experiments in this paper only provide results on the Speech-to-Text Simultaneous Translation task, and do not validate the method on other streaming generation tasks.  
2. Under lower latency conditions (about 1000 ms) for the EnDe and EnEs tasks, the performance of MA-T appears to be inferior to that of CAAT, another transducer-based method. This may indicate that the proposed method in this paper is not highly effective.  
3. The proposed method in this paper shares a very similar model structure with both CAAT (Liu et al. 2021) and  MILK (Arivazhagan et al 2019), which raises concerns about the lack of novelty in the article.

**Reference**

[1]Dan Liu, Mengge Du, Xiaoxi Li, Ya Li, and Enhong Chen. Cross attention augmented transducer networks for simultaneous translation. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing.

[2]Naveen Arivazhagan, Colin Cherry, Wolfgang Macherey, Chung-Cheng Chiu, Semih Yavuz, Ruoming Pang, Wei Li, and Colin Raffel. Monotonic infinite lookback attention for simultaneous machine translation. In Anna Korhonen, David Traum, and Llu´ıs Marquez (eds.), ` Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.2019.

questions:
1. After incorporating a unidirectional encoder, the model structures of MA-T and CAAT are very similar. What are the specific differences between the two? How does MA-T manage to perform attention computations while ensuring that Memory Overload remains O(1)?  
2. Using AL as a latency metric may not accurately assess the phenomenon of over-generation. Could you provide alternative latency metrics, such as LAAL or LAAL-CA?  
3. Regarding the comparison between MA-T and TAED, could you provide more experimental results to demonstrate that your method truly achieves better performance than MA-T? From the perspective of AL and BLEU, while TAED has a larger computational overhead, it does achieve higher translation quality at a lower latency and the overhead is still acceptable.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730533606041

thread_type:
official_review

<thread_object>
creator: reviewer_ak6x

<content>
summary:
This paper describes an extension for Transducers for tasks where the input and output are not monotonically aligned, such as speech translation. The method is evaluated on two languages of the MuST-C benchmark and it is compared to existing methods.

strengths:
* The paper addresses an interesting setting which has practical implications
* The method is described in detail and a large portion of the paper is dedicated to this description.

weaknesses:
* The method is fairly complex, requiring a range of steps in addition to Transducer training as shown in Algorithm 1 (which is already complex).
* The description of the method is sometimes not very clear. It would be helpful to provide intuition in addition to equations.
* The evaluation was done on languages with relatively similar word order: En-Es and En-De differ in their word order but much less so, then say En-Chinese or English and any non-European language. There are speech translation benchmarks which enable these settings, e.g., FLEURS and which would provide more interesting results.
* It seems a bit surprising that the BLEU improvements for En-Es and En-De between Transducer and the new method seem fairly similar across most chunk sizes (about 1 BLEU, Table 2). I would have expected En-De to benefit more the new method than En-Es given that the word order of En-De is more different.

questions:
* Did you consider evaluating on other settings than En-De/En-Es where word order is more different?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730713045541

thread_type:
official_review

<thread_object>
creator: reviewer_uli8

<content>
summary:
This paper proposes MonoAttn-Transducer, a novel approach to enhance Transducer models with learnable monotonic attention for streaming generation tasks. The key contributions are:

- A method to integrate monotonic attention into Transducer's architecture while maintaining its efficient training through the forward-backward algorithm.
- A training algorithm that uses posterior alignment probabilities to estimate context representations for monotonic attention.
- A chunk synchronization mechanism to bridge the gap between training and inference.
- Extensive experiments demonstrating improved performance on simultaneous translation tasks.

strengths:
Technical Innovation:

- The proposed method cleverly solves the exponential state space problem by using posterior alignments to estimate context representations
- The solution maintains the same computational complexity as vanilla Transducer
- The chunk synchronization mechanism shows thoughtful consideration of practical deployment

Theoretical Foundation:

- The approach is well-grounded in probability theory and previous work on Transducers
- The mathematical derivations are sound and clearly explained
- The relationship between prior and posterior alignments is well-analyzed

Empirical Results:

- Comprehensive experiments on MuST-C dataset
- Strong improvements over baseline Transducer (+0.75-1.0 BLEU, +0.95-2.06 COMET)
- Thorough ablation studies and analysis
- Competitive performance against SOTA methods

weaknesses:
Limited Experimental Scope:

- Experiments focus only on speech-to-text translation.
- Only two language pairs (En->De, En->Es) are tested.
- No experiments on other streaming generation tasks like ASR or TTS.

Training Efficiency:

- The paper doesn't discuss training time comparison with baseline Transducer.
- Memory usage analysis could be more detailed.
- No discussion of potential overhead from posterior alignment calculation.


Algorithmic Limitations:

- The method still requires chunk-based processing.
- The impact of chunk size on performance could be better theoretically explained.
- The choice of diagonal prior distribution seems somewhat arbitrary.

questions:
/
</content>
</thread_object>

</thread>
<thread>
creation date:
1730718981175

thread_type:
official_review

<thread_object>
creator: reviewer_zys6

<content>
summary:
This paper discusses the challenges faced by Transducer-based streaming generation models, particularly in tasks requiring non-monotonic alignments, such as simultaneous translation. These challenges arise from the input-synchronous decoding mechanism of Transducers, which can result in suboptimal performance. To address this, the authors propose integrating a learnable monotonic attention mechanism within the Transducer architecture. This mechanism uses a forward-backward algorithm to calculate the posterior probability of alignments between predictor states and input timestamps. Consequently, it allows for the estimation of context representations of monotonic attention during training, enabling the model to adaptively adjust its attention scope based on predictions. This innovative approach eliminates the need to explore the exponentially large alignment space. Experiments reveal that the proposed MonoAttn-Transducer significantly improves performance in streaming generation tasks dealing with non-monotonic alignments. The codes for this study are available in the supplementary materials.

strengths:
1. The authors propose a learnable monotonic attention mechanism within the Transducer architecture to solve the non-monotonic alignments in streaming generation.

2. The propsoed method is effectiveness and easy to reproduce.

weaknesses:
1. I am unable to discern the difference between the proposed monotonic attention mechanism and the cross-attention mechanism.

2. Furthermore, the authors should compare the cross-attention approach (Liu et al., 2021; Tang et al., 2023) with other methods that utilize input history.

questions:
N/A
</content>
</thread_object>

</thread>
<thread>
creation date:
1732236267737

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Kind Reminder

comment:
Dear Reviewers,

As the rebuttal phase is nearing its end, we would greatly appreciate it if you could kindly take a moment to review our responses. We would be pleased to discuss further questions if you have any.

Best regards,  

Authors
</content>
</thread_object>

</thread>
<thread>
creation date:
1732472623178

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Rebuttal Summary

comment:
***UPDATE 2024/12/4***

**The discussion period is finished. Sadly, NONE of the reviewers are willing to engage in the rebuttal discussion. However, we provide a summary of our rebuttal here and hope it will be helpful to anyone interested in this paper. We believe this reflects the value of OpenReview.**

> **Intuition of this work (Question from Reviewer ak6X)**

Transducer is a widely applied streaming architecture. However, it is less suitable for tasks where the alignment is non-monotonic (e.g., simultaneous interpretation from SVO language to SOV language). The non-monotonicity problem arises from its synchronization mechanism, which relies solely on an MLP-based joiner. It lacks an attention module to facilitate target outputs reordering. However, introducing attention module to streaming generation architecture is not trivial, as you have no idea how many source tokens a target token can attend to during training. **The major contribution of this work is providing a low-complexity algorithm to introduce the attention mechanism into Transducer, effectively addressing the issue of non-monotonic alignment in transducer-based streaming generation.** 

> **Why our method performs attention computations while ensuring the memory overload in forward-backward algorithm remains $O(1)$? (Question from Reviewer 2LWH)**

We solve this problem by estimating the posterior probability of alignment between source and target (Eq. 5 and Eq. 7 in paper). This estimation from output lattice directly gives us clues of how much source each target token can attend to in training. We use this posterior alignment probability to estimate the expected representation for each predictor state, thus ensuring that Memory Overload remains $O(1)$.

> **Questions about Training Efficiency (Question from Reviewer uLi8)**

***Training Time***  
We first analyze each step in Algorithm 1 to theoretically compare the training time differences between MonoAttn-Transducer and baseline.
Firstly, we observe that Lines 1, 2, and 6 involve naive matrix computation without requiring gradients, which do not significantly impact the overall overhead.
The additional time overhead introduced by our method arises from Lines 3, 4, and 5. **Specifically, this includes an additional forward pass of the predictor component and the computation for the posterior alignment.
The overhead from the posterior alignment calculation is approximately equivalent to the overhead incurred during loss calculation, as both rely on the same forward-backward algorithm.
Emperically, we found MonoAttn-Trasducer is 1.33 times slower than Transducer baseline with the same configuration on Nvidia L40 GPU.**

***Training Memory***  
Compared to Transducer baseline, **the additional memory overhead of MonoAttn-Transducer comes solely from its Monotonic Attention module.**
The extra forward pass of the predictor is performed without gradient computation (i.e., within a ``with torch.no_grad()`` context), so it does not contribute to the large computation graph.
Emperically, we observed that the peak memory usage of Transducer baseline is 28GB, while MonoAttn-Transducer exhibits a slightly higher peak usage of 32GB when the total number of source frames is fixed at 40,000 on a single Nvidia L40 GPU.


> **Experiments on other streaming generation task (Question from Reviewer uLi8 & 2LWH)**

we additionally conducted experiments on **textless speech-to-speech simultaneous translation**. Speech-to-speech simultaneous translation encompasses three streaming tasks: ASR, MT, and TTS. **This challenging task requires implicitly performing all tasks simultaneously, and also handling the non-monotonic alignments in streaming generation, making it a suitable benchmark to evaluate the capabilities of our MonoAttn-Transducer on streaming speech generation.**

|*FR-> EN Speech-to-Speech*| | | |
 | - | - | - | - |
|Chunk Size|  |   320ms    |    Offline  |
|MonoAttn-Transducer| ASR-BLEU |18.3|19.3|
|                   | AL  (ms)     |118|-|
|                   | LAAL (ms)    |972|-|
|Transducer| ASR-BLEU |17.1|18.0|
|          | AL (ms)      |153|-|
|          | LAAL (ms)    |984|-|


**In speech-to-speech streaming generation, MonoAttn-Transducer also demonstrated a significant improvement in speech generation quality. This indicates that our approach is not limited to streaming text generation but also effective for streaming speech generation.**


--------------------------------
***2024/11/25***


Dear Reviewers,

We have uploaded a revised version of our paper addressing your questions. The updates include:

1. Experiments on more streaming generation tasks, discussed in Section 5.
2. Analysis of training efficiency, discussed in Section 6.


Thank you for your valuable feedback and we sincerely hope you could take a moment to read our rebuttal.   
We would greatly appreciate it if you could reply.

Best regards,

Authors
</content>
</thread_object>

</thread>
<thread>
creation date:
1732641205102

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Kind Reminder After a Week

comment:
**Dear Reviewers,**

**It has been a week since we submitted our rebuttal, and we are very hopeful to receive a response.**

**We would greatly appreciate any discussions or feedback.**

**Best regards,**

**Authors**
</content>
</thread_object>

</thread>
<thread>
creation date:
1732793502455

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Very Frustrating Reviewing Process

comment:
**We appreciate the time and effort of everyone involved in reviewing the paper. However, the process makes us strongly feeling the reviewing is not being respected.**

**1. We used GPT-ZERO to evaluate the official review from Reviewer zyS6. We found that the summary section is highly likely to be generated by AI, and the weakness part lacks basic coherence and clarity.**

**2. After waiting a long time since submitting our rebuttal, we sadly found that no reviewer was willing to engage in a discussion. 
It seems all reviewers are even unwilling to share whether their concerns have been addressed or to specify any details they are still worried about. The only comment received is on the overall score. This is TOXIC and disrespects our efforts in the rebuttal period.**

**We are not begging the reviewer to increase their score. Actually, we welcome any discussion or even criticism in responses. The only hope is these discussions will be professional and specific.** **We are not trying to be emotional, but it is hard to believe we are encountering such a situation at a top-tier academic conference.**

**Authors**
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper improves Transducer-based streaming models for tasks that need non-monotonic alignments (like speech-to-text simultaneous translation).  Specifically, the authors integrate a learnable monotonic attention mechanism into a Transducer architecture, leveraging the forward-backward algorithm to compute posterior alignment probabilities between predictor states and input timestamps. Experimental results on EN-DE and EN-ES speech-translation tasks suggest that the proposed MonoAttn-Transducer outperforms a standard Transducer baseline.

**Strengths** (1) The work tackles the known issue of standard Transducers with a principled solution (2) The empirical gains on two benchmarks are solid

**Weaknesses** (1) Some reviewers pointed out the limited scope of the method which only considers STT on closely related languages. More experiments on diverse language pairs and tasks are preferred; (2) some reviewers questioned the novelty compared to existing works.

**Decision**

While the reviewers agree that the paper tackles an important real-world challenge for steaming translation tasks, concerns remain about the novelty and experimental scopes, which could not be resolved during the rebuttal phase. So, I am leaning toward rejection and would like to encourage the authors to include more experiments with clearer comparisons in the future version.

*The review of zyS6 seems low quality (also raised by the authors), and no further engagement happened during the rebuttal phase. Therefore, the judgment is made by discarding the review of zyS6.*