GenAI4ELab/papercli-papers huggingface dataset:
QUESPA Submission for the IWSLT 2023 Dialect and Low-resource Speech Translation Tasks
Authors: John E. Ortega; Rodolfo Zevallos; William Chen
Venue: ACL 2023 (Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023))

Abstract: This article describes the QUESPA team speech translation (ST) submissions for the Quechua to Spanish (QUE–SPA) track featured in the Evaluation Campaign of IWSLT 2023: low-resource and dialect speech translation. Two main submission types were supported in the campaign: constrained and unconstrained. We submitted six total systems of which our best (primary) constrained system consisted of an ST model based on the Fairseq S2T framework where the audio representations were created using log mel-scale filter banks as features and the translations were performed using a transformer. The best (primary) unconstrained system used a pipeline approach which combined automatic speech recognition (ASR) with machine translation (MT). The ASR transcriptions for the best unconstrained system were computed using a pre-trained XLS-R-based model along with a fine-tuned language model. Transcriptions were translated using a MT system based on a fine-tuned, pre-trained language model (PLM). The four other submissions are presented in this article (2 constrained and 2 unconstrained) for comparison because they consist of various architectures. Our results show that direct ST (ASR and MT combined together) can be more effective than a PLM in a low-resource (constrained) setting for Quechua to Spanish. On the other hand, we show that fine-tuning of any type on both the ASR and MT system is worthwhile, resulting in nearly 16 BLEU for the unconstrained task.
GMU Systems for the IWSLT 2023 Dialect and Low-resource Speech Translation Tasks
Authors: Jonathan Mbuya; Antonios Anastasopoulos
Venue: ACL 2023 (Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023))

Abstract: This paper describes the GMU Systems for the IWSLT 2023 Dialect and Low-resource Speech Translation Tasks. We submitted systems for five low-resource tasks and the dialectal task. In this work, we explored self-supervised pre-trained speech models and finetuned them on speech translation downstream tasks. We use the Wav2vec 2.0, XLSR-53, and Hubert as self-supervised models. Unlike Hubert, Wav2vec 2.0 and XLSR-53 achieve the best results when we remove the top three layers. Our results show that Wav2vec 2.0 and Hubert perform similarly with their relative best configuration. In addition, we found that Wav2vec 2.0 pre-trained on audio data of the same language as the source language of a speech translation model achieves better results. For the low-resource setting, the best results are achieved using either the Wav2vec 2.0 or Hubert models, while XLSR-53 achieves the best results for the dialectal transfer task. We find that XLSR-53 does not perform well for low-resource tasks. Using Wav2vec 2.0, we report close to 2 BLEU point improvements on the test set for the Tamasheq-French compared to the baseline system at the IWSLT 2022.
The HW-TSC’s Speech-to-Speech Translation System for IWSLT 2023
Authors: Minghan Wang; Yinglu Li; Jiaxin Guo; Zongyao Li; Hengchao Shang; Daimeng Wei; Min Zhang; Shimin Tao; Hao Yang
Venue: ACL 2023 (Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023))

Abstract: This paper describes our work on the IWSLT2023 Speech-to-Speech task. Our proposed cascaded system consists of an ensemble of Conformer and S2T-Transformer-based ASR models, a Transformer-based MT model, and a Diffusion-based TTS model. Our primary focus in this competition was to investigate the modeling ability of the Diffusion model for TTS tasks in high-resource scenarios and the role of TTS in the overall S2S task. To this end, we proposed DTS, an end-to-end diffusion-based TTS model that takes raw text as input and generates waveform by iteratively denoising on pure Gaussian noise. Compared to previous TTS models, the speech generated by DTS is more natural and performs better in code-switching scenarios. As the training process is end-to-end, it is relatively straightforward. Our experiments demonstrate that DTS outperforms other TTS models on the GigaS2S benchmark, and also brings positive gains for the entire S2S system.
JHU IWSLT 2023 Dialect Speech Translation System Description
Authors: Amir Hussein; Cihan Xiao; Neha Verma; Thomas Thebaud; Matthew Wiesner; Sanjeev Khudanpur
Venue: ACL 2023 (Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023))

Abstract: This paper presents JHU’s submissions to the IWSLT 2023 dialectal and low-resource track of Tunisian Arabic to English speech translation. The Tunisian dialect lacks formal orthography and abundant training data, making it challenging to develop effective speech translation (ST) systems. To address these challenges, we explore the integration of large pre-trained machine translation (MT) models, such as mBART and NLLB-200 in both end-to-end (E2E) and cascaded speech translation (ST) systems. We also improve the performance of automatic speech recognition (ASR) through the use of pseudo-labeling data augmentation and channel matching on telephone data. Finally, we combine our E2E and cascaded ST systems with Minimum Bayes-Risk decoding. Our combined system achieves a BLEU score of 21.6 and 19.1 on test2 and test3, respectively.
Learning Nearest Neighbour Informed Latent Word Embeddings to Improve Zero-Shot Machine Translation
Authors: Nishant Kambhatla; Logan Born; Anoop Sarkar
Venue: ACL 2023 (Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023))

Abstract: Multilingual neural translation models exploit cross-lingual transfer to perform zero-shot translation between unseen language pairs. Past efforts to improve cross-lingual transfer have focused on aligning contextual sentence-level representations. This paper introduces three novel contributions to allow exploiting nearest neighbours at the token level during training, including: (i) an efficient, gradient-friendly way to share representations between neighboring tokens; (ii) an attentional semantic layer which extracts latent features from shared embeddings; and (iii) an agreement loss to harmonize predictions across different sentence representations. Experiments on two multilingual datasets demonstrate consistent gains in zero shot translation over strong baselines.
JHU IWSLT 2023 Multilingual Speech Translation System Description
Authors: Henry Li Xinyuan; Neha Verma; Bismarck Bamfo Odoom; Ujvala Pradeep; Matthew Wiesner; Sanjeev Khudanpur
Venue: ACL 2023 (Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023))

Abstract: We describe the Johns Hopkins ACL 60-60 Speech Translation systems submitted to the IWSLT 2023 Multilingual track, where we were tasked to translate ACL presentations from English into 10 languages. We developed cascaded speech translation systems for both the constrained and unconstrained subtracks. Our systems make use of pre-trained models as well as domain-specific corpora for this highly technical evaluation-only task. We find that the specific technical domain which ACL presentations fall into presents a unique challenge for both ASR and MT, and we present an error analysis and an ACL-specific corpus we produced to enable further work in this area.
The NPU-MSXF Speech-to-Speech Translation System for IWSLT 2023 Speech-to-Speech Translation Task
Authors: Kun Song; Yi Lei; Peikun Chen; Yiqing Cao; Kun Wei; Yongmao Zhang; Lei Xie; Ning Jiang; Guoqing Zhao
Venue: ACL 2023 (Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023))

Abstract: This paper describes the NPU-MSXF system for the IWSLT 2023 speech-to-speech translation (S2ST) task which aims to translate from English speech of multi-source to Chinese speech. The system is built in a cascaded manner consisting of automatic speech recognition (ASR), machine translation (MT), and text-to-speech (TTS). We make tremendous efforts to handle the challenging multi-source input. Specifically, to improve the robustness to multi-source speech input, we adopt various data augmentation strategies and a ROVER-based score fusion on multiple ASR model outputs. To better handle the noisy ASR transcripts, we introduce a three-stage fine-tuning strategy to improve translation accuracy. Finally, we build a TTS model with high naturalness and sound quality, which leverages a two-stage framework, using network bottleneck features as a robust intermediate representation for speaker timbre and linguistic content disentanglement. Based on the two-stage framework, pre-trained speaker embedding is leveraged as a condition to transfer the speaker timbre in the source English speech to the translated Chinese speech. Experimental results show that our system has high translation accuracy, speech naturalness, sound quality, and speaker similarity. Moreover, it shows good robustness to multi-source data.
Low-Resource Formality Controlled NMT Using Pre-trained LM
Authors: Priyesh Vakharia; Shree Vignesh S; Pranjali Basmatkar
Venue: ACL 2023 (Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023))

Abstract: This paper describes the UCSC’s submission to the shared task on formality control for spoken language translation at IWSLT 2023. For this task, we explored the use of ‘additive style intervention’ using a pre-trained multilingual translation model, namely mBART. Compared to prior approaches where a single style-vector was added to all tokens in the encoder output, we explored an alternative approach in which we learn a unique style-vector for each input token. We believe this approach, which we call ‘style embedding intervention,’ is better suited for formality control as it can potentially learn which specific input tokens to modify during decoding. While the proposed approach obtained similar performance to ‘additive style intervention’ for the supervised English-to-Vietnamese task, it performed significantly better for English-to-Korean, in which it achieved an average matched accuracy of 90.6 compared to 85.2 for the baseline. When we constrained the model further to only perform style intervention on the <bos> (beginning of sentence) token, the average matched accuracy improved further to 92.0, indicating that the model could learn to control the formality of the translation output based solely on the embedding of the <bos> token.
NAIST Simultaneous Speech-to-speech Translation System for IWSLT 2023
Authors: Ryo Fukuda; Yuta Nishikawa; Yasumasa Kano; Yuka Ko; Tomoya Yanagita; Kosuke Doi; Mana Makinae; Sakriani Sakti; Katsuhito Sudoh; Satoshi Nakamura
Venue: ACL 2023 (Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023))

Abstract: This paper describes NAIST’s submission to the IWSLT 2023 Simultaneous Speech Translation task: English-to-German, Japanese, Chinese speech-to-text translation and English-to-Japanese speech-to-speech translation. Our speech-to-text system uses an end-to-end multilingual speech translation model based on large-scale pre-trained speech and text models. We add Inter-connections into the model to incorporate the outputs from intermediate layers of the pre-trained speech model and augment prefix-to-prefix text data using Bilingual Prefix Alignment to enhance the simultaneity of the offline speech translation model. Our speech-to-speech system employs an incremental text-to-speech module that consists of a Japanese pronunciation estimation model, an acoustic model, and a neural vocoder.
Language Model Based Target Token Importance Rescaling for Simultaneous Neural Machine Translation
Authors: Aditi Jain; Nishant Kambhatla; Anoop Sarkar
Venue: ACL 2023 (Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023))

Abstract: The decoder in simultaneous neural machine translation receives limited information from the source while having to balance the opposing requirements of latency versus translation quality. In this paper, we use an auxiliary target-side language model to augment the training of the decoder model. Under this notion of target adaptive training, generating rare or difficult tokens is rewarded which improves the translation quality while reducing latency. The predictions made by a language model in the decoder are combined with the traditional cross entropy loss which frees up the focus on the source side context. Our experimental results over multiple language pairs show that compared to previous state of the art methods in simultaneous translation, we can use an augmented target side context to improve BLEU scores significantly. We show improvements over the state of the art in the low latency range with lower average lagging values (faster output).
The Kyoto Speech-to-Speech Translation System for IWSLT 2023
Authors: Zhengdong Yang; Shuichiro Shimizu; Wangjin Zhou; Sheng Li; Chenhui Chu
Venue: ACL 2023 (Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023))

Abstract: This paper describes the Kyoto speech-to-speech translation system for IWSLT 2023. Our system is a combination of speech-to-text translation and text-to-speech synthesis. For the speech-to-text translation model, we used the dual-decoderTransformer model. For text-to-speech synthesis model, we took a cascade approach of an acoustic model and a vocoder.
Tagged End-to-End Simultaneous Speech Translation Training Using Simultaneous Interpretation Data
Authors: Yuka Ko; Ryo Fukuda; Yuta Nishikawa; Yasumasa Kano; Katsuhito Sudoh; Satoshi Nakamura
Venue: ACL 2023 (Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023))

Abstract: Simultaneous speech translation (SimulST) translates partial speech inputs incrementally. Although the monotonic correspondence between input and output is preferable for smaller latency, it is not the case for distant language pairs such as English and Japanese. A prospective approach to this problem is to mimic simultaneous interpretation (SI) using SI data to train a SimulST model. However, the size of such SI data is limited, so the SI data should be used together with ordinary bilingual data whose translations are given in offline. In this paper, we propose an effective way to train a SimulST model using mixed data of SI and offline. The proposed method trains a single model using the mixed data with style tags that tell the model to generate SI- or offline-style outputs. Experiment results show improvements of BLEURT in different latency ranges, and our analyses revealed the proposed model generates SI-style outputs more than the baseline.
The HW-TSC’s Simultaneous Speech-to-Text Translation System for IWSLT 2023 Evaluation
Authors: Jiaxin Guo; Daimeng Wei; Zhanglin Wu; Zongyao Li; Zhiqiang Rao; Minghan Wang; Hengchao Shang; Xiaoyu Chen; Zhengzhe Yu; Shaojun Li; Yuhao Xie; Lizhi Lei; Hao Yang
Venue: ACL 2023 (Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023))

Abstract: In this paper, we present our submission to the IWSLT 2023 Simultaneous Speech-to-Text Translation competition. Our participation involves three language directions: English-German, English-Chinese, and English-Japanese. Our proposed solution is a cascaded incremental decoding system that comprises an ASR model and an MT model. The ASR model is based on the U2++ architecture and can handle both streaming and offline speech scenarios with ease. Meanwhile, the MT model adopts the Deep-Transformer architecture. To improve performance, we explore methods to generate a confident partial target text output that guides the next MT incremental decoding process. In our experiments, we demonstrate that our simultaneous strategies achieve low latency while maintaining a loss of no more than 2 BLEU points when compared to offline systems.
The HW-TSC’s Simultaneous Speech-to-Speech Translation System for IWSLT 2023 Evaluation
Authors: Hengchao Shang; Zhiqiang Rao; Zongyao Li; Zhanglin Wu; Jiaxin Guo; Minghan Wang; Daimeng Wei; Shaojun Li; Zhengzhe Yu; Xiaoyu Chen; Lizhi Lei; Hao Yang
Venue: ACL 2023 (Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023))

Abstract: In this paper, we present our submission to the IWSLT 2023 Simultaneous Speech-to-Speech Translation competition. Our participation involves three language directions: English-German, English-Chinese, and English-Japanese. Our solution is a cascaded incremental decoding system, consisting of an ASR model, an MT model, and a TTS model. By adopting the strategies used in the Speech-to-Text track, we have managed to generate a more confident target text for each audio segment input, which can guide the next MT incremental decoding process. Additionally, we have integrated the TTS model to seamlessly reproduce audio files from the translation hypothesis. To enhance the effectiveness of our experiment, we have utilized a range of methods to reduce error conditions in the TTS input text and improve the smoothness of the TTS output audio.
Towards Efficient Simultaneous Speech Translation: CUNI-KIT System for Simultaneous Track at IWSLT 2023
Authors: Peter Polák; Danni Liu; Ngoc-Quan Pham; Jan Niehues; Alexander Waibel; Ondřej Bojar
Venue: ACL 2023 (Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023))

Abstract: In this paper, we describe our submission to the Simultaneous Track at IWSLT 2023. This year, we continue with the successful setup from the last year, however, we adopt the latest methods that further improve the translation quality. Additionally, we propose a novel online policy for attentional encoder-decoder models. The policy prevents the model to generate translation beyond the current speech input by using an auxiliary CTC output layer. We show that the proposed simultaneous policy can be applied to both streaming blockwise models and offline encoder-decoder models. We observe significant improvements in quality (up to 1.1 BLEU) and the computational footprint (up to 45% relative RTF).
Speech Translation with Foundation Models and Optimal Transport: UPC at IWSLT23
Authors: Ioannis Tsiamas; Gerard I. Gállego; José A. R. Fonollosa; Marta R. Costa-jussá
Venue: ACL 2023 (Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023))

Abstract: This paper describes the submission of the UPC Machine Translation group to the IWSLT 2023 Offline Speech Translation task. Our Speech Translation systems utilize foundation models for speech (wav2vec 2.0) and text (mBART50). We incorporate a Siamese pretraining step of the speech and text encoders with CTC and Optimal Transport, to adapt the speech representations to the space of the text model, thus maximizing transfer learning from MT. After this pretraining, we fine-tune our system end-to-end on ST, with Cross Entropy and Knowledge Distillation. Apart from the available ST corpora, we create synthetic data with SegAugment to better adapt our models to the custom segmentations of the IWSLT test sets. Our best single model obtains 31.2 BLEU points on MuST-C tst-COMMON, 29.8 points on IWLST.tst2020 and 33.4 points on the newly released IWSLT.ACLdev2023.
The Xiaomi AI Lab’s Speech Translation Systems for IWSLT 2023 Offline Task, Simultaneous Task and Speech-to-Speech Task
Authors: Wuwei Huang; Mengge Liu; Xiang Li; Yanzhi Tian; Fengyu Yang; Wen Zhang; Jian Luan; Bin Wang; Yuhang Guo; Jinsong Su
Venue: ACL 2023 (Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023))

Abstract: This system description paper introduces the systems submitted by Xiaomi AI Lab to the three tracks of the IWSLT 2023 Evaluation Campaign, namely the offline speech translation (Offline-ST) track, the offline speech-to-speech translation (Offline-S2ST) track, and the simultaneous speech translation (Simul-ST) track. All our submissions for these three tracks only involve the English-Chinese language direction. Our English-Chinese speech translation systems are constructed using large-scale pre-trained models as the foundation. Specifically, we fine-tune these models’ corresponding components for various downstream speech translation tasks. Moreover, we implement several popular techniques, such as data filtering, data augmentation, speech segmentation, and model ensemble, to improve the system’s overall performance. Extensive experiments show that our systems achieve a significant improvement over the strong baseline systems in terms of the automatic evaluation metric.
Improving Formality-Sensitive Machine Translation Using Data-Centric Approaches and Prompt Engineering
Authors: Seungjun Lee; Hyeonseok Moon; Chanjun Park; Heuiseok Lim
Venue: ACL 2023 (Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023))

Abstract: In this paper, we present the KU x Upstage team’s submission for the Special Task on Formality Control on Spoken Language Translation, which involves translating English into four languages with diverse grammatical formality markers. Our methodology comprises two primary components: 1) a language-specific data-driven approach, and 2) the generation of synthetic data through the employment of large-scale language models and empirically-grounded prompt engineering. By adapting methodologies and models to accommodate the unique linguistic properties of each language, we observe a notable enhancement in performance relative to the baseline, substantiating the heightened efficacy of data-driven approaches. Moreover, our devised prompt engineering strategy yields superior synthetic translation instances.
UM-DFKI Maltese Speech Translation
Authors: Aiden Williams; Kurt Abela; Rishu Kumar; Martin Bär; Hannah Billinghurst; Kurt Micallef; Ahnaf Mozib Samin; Andrea DeMarco; Lonneke van der Plas; Claudia Borg
Venue: ACL 2023 (Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023))

Abstract: For the 2023 IWSLT Maltese Speech Translation Task, UM-DFKI jointly presents a cascade solution which achieves 0.6 BLEU. While this is the first time that a Maltese speech translation task has been released by IWSLT, this paper explores previous solutions for other speech translation tasks, focusing primarily on low-resource scenarios. Moreover, we present our method of fine-tuning XLS-R models for Maltese ASR using a collection of multi-lingual speech corpora as well as the fine-tuning of the mBART model for Maltese to English machine translation.
NVIDIA NeMo Offline Speech Translation Systems for IWSLT 2023
Authors: Oleksii Hrinchuk; Vladimir Bataev; Evelina Bakhturina; Boris Ginsburg
Venue: ACL 2023 (Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023))

Abstract: This paper provides an overview of NVIDIA NeMo’s speech translation systems for the IWSLT 2023 Offline Speech Translation Task. This year, we focused on end-to-end system which capitalizes on pre-trained models and synthetic data to mitigate the problem of direct speech translation data scarcity. When trained on IWSLT 2022 constrained data, our best En->De end-to-end model achieves the average score of 31 BLEU on 7 test sets from IWSLT 2010-2020 which improves over our last year cascade (28.4) and end-to-end (25.7) submissions. When trained on IWSLT 2023 constrained data, the average score drops to 29.5 BLEU.