GenAI4ELab/papercli-papers huggingface dataset:
HW-TSC’s Submissions To the IWSLT2024 Low-resource Speech Translation Tasks
Authors: Zheng Jiawei; Hengchao Shang; Zongyao Li; Zhanglin Wu; Daimeng Wei; Zhiqiang Rao; Shaojun Li; Jiaxin Guo; Bin Wei; Yuanchang Luo; Hao Yang
Venue: ACL 2024 (Proceedings of the 21st International Conference on Spoken Language Translation (IWSLT 2024))

Abstract: In this work, we submitted our systems to the low-resource track of the IWSLT 2024 Speech Translation Campaign. Our systems tackled the unconstrained condition of the Dialectal Arabic North Levantine (ISO-3 code: apc) to English language pair. We proposed a cascaded solution consisting of an automatic speech recognition (ASR) model and a machine translation (MT) model. It was noted that the ASR model employed the pre-trained Whisper-large-v3 model to process the speech data, while the MT model adopted the Transformer architecture. To improve the quality of the MT model, it was stated that our system utilized not only the data provided by the competition but also an additional 54 million parallel sentences. Ultimately, we reported that our final system achieved a BLEU score of 24.7 for apc-to-English translation.
CMU’s IWSLT 2024 Offline Speech Translation System: A Cascaded Approach For Long-Form Robustness
Authors: Brian Yan; Patrick Fernandes; Jinchuan Tian; Siqi Ouyang; William Chen; Karen Livescu; Lei Li; Graham Neubig; Shinji Watanabe
Venue: ACL 2024 (Proceedings of the 21st International Conference on Spoken Language Translation (IWSLT 2024))

Abstract: This work describes CMU’s submission to the IWSLT 2024 Offline Speech Translation (ST) Shared Task for translating English speech to German, Chinese, and Japanese text. We are the first participants to employ a long-form strategy which directly processes unsegmented recordings without the need for a separate voice-activity detection stage (VAD). We show that the Whisper automatic speech recognition (ASR) model has a hallucination problem when applied out-of-the-box to recordings containing non-speech noises, but a simple noisy fine-tuning approach can greatly enhance Whisper’s long-form robustness across multiple domains. Then, we feed English ASR outputs into fine-tuned NLLB machine translation (MT) models which are decoded using COMET-based Minimum Bayes Risk. Our VAD-free ASR+MT cascade is tested on TED talks, TV series, and workout videos and shown to outperform prior winning IWSLT submissions and large open-source models.
NAIST Simultaneous Speech Translation System for IWSLT 2024
Authors: Yuka Ko; Ryo Fukuda; Yuta Nishikawa; Yasumasa Kano; Tomoya Yanagita; Kosuke Doi; Mana Makinae; Haotian Tan; Makoto Sakai; Sakriani Sakti; Katsuhito Sudoh; Satoshi Nakamura
Venue: ACL 2024 (Proceedings of the 21st International Conference on Spoken Language Translation (IWSLT 2024))

Abstract: This paper describes NAIST’s submission to the simultaneous track of the IWSLT 2024 Evaluation Campaign: English-to-German, Japanese, Chinese speech-to-text translation and English-to-Japanese speech-to-speech translation. We develop a multilingual end-to-end speech-to-text translation model combining two pre-trained language models, HuBERT and mBART. We trained this model with two decoding policies, Local Agreement (LA) and AlignAtt. The submitted models employ the LA policy because it outperformed the AlignAtt policy in previous models. Our speech-to-speech translation method is a cascade of the above speech-to-text model and an incremental text-to-speech (TTS) module that incorporates a phoneme estimation model, a parallel acoustic model, and a parallel WaveGAN vocoder. We improved our incremental TTS by applying the Transformer architecture with the AlignAtt policy for the estimation model. The results show that our upgraded TTS module contributed to improving the system performance.
Blending LLMs into Cascaded Speech Translation: KIT’s Offline Speech Translation System for IWSLT 2024
Authors: Sai Koneru; Thai Binh Nguyen; Ngoc-Quan Pham; Danni Liu; Zhaolin Li; Alexander Waibel; Jan Niehues
Venue: ACL 2024 (Proceedings of the 21st International Conference on Spoken Language Translation (IWSLT 2024))

Abstract: Large Language Models (LLMs) are currently under exploration for various tasks, including Automatic Speech Recognition (ASR), Machine Translation (MT), and even End-to-End Speech Translation (ST). In this paper, we present KIT’s offline submission in the constrained + LLM track by incorporating recently proposed techniques that can be added to any cascaded speech translation. Specifically, we integrate Mistral-7B into our system to enhance it in two ways. Firstly, we refine the ASR outputs by utilizing the N-best lists generated by our system and fine-tuning the LLM to predict the transcript accurately. Secondly, we refine the MT outputs at the document level by fine-tuning the LLM, leveraging both ASR and MT predictions to improve translation quality. We find that integrating the LLM into the ASR and MT systems results in an absolute improvement of 0.3% in Word Error Rate and 0.65% in COMET for tst2019 test set. In challenging test sets with overlapping speakers and background noise, we find that integrating LLM is not beneficial due to poor ASR performance. Here, we use ASR with chunked long-form decoding to improve context usage that may be unavailable when transcribing with Voice Activity Detection segmentation alone.
ALADAN at IWSLT24 Low-resource Arabic Dialectal Speech Translation Task
Authors: Waad Ben Kheder; Josef Jon; André Beyer; Abdel Messaoudi; Rabea Affan; Claude Barras; Maxim Tychonov; Jean-Luc Gauvain
Venue: ACL 2024 (Proceedings of the 21st International Conference on Spoken Language Translation (IWSLT 2024))

Abstract: This paper presents ALADAN’s approach to the IWSLT 2024 Dialectal and Low-resource shared task, focusing on Levantine Arabic (apc) and Tunisian Arabic (aeb) to English speech translation (ST). Addressing challenges such as the lack of standardized orthography and limited training data, we propose a solution for data normalization in Dialectal Arabic, employing a modified Levenshtein distance and Word2vec models to find orthographic variants of the same word. Our system consists of a cascade ST system integrating two ASR systems (TDNN-F and Zipformer) and two NMT modules derived from pre-trained models (NLLB-200 1.3B distilled model and CohereAI’s Command-R). Additionally, we explore the integration of unsupervised textual and audio data, highlighting the importance of multi-dialectal datasets for both ASR and NMT tasks. Our system achieves BLEU score of 31.5 for Levantine Arabic on the official validation set.
Enhancing Translation Accuracy of Large Language Models through Continual Pre-Training on Parallel Data
Authors: Minato Kondo; Takehito Utsuro; Masaaki Nagata
Venue: ACL 2024 (Proceedings of the 21st International Conference on Spoken Language Translation (IWSLT 2024))

Abstract: In this paper, we propose a two-phase training approach where pre-trained large language models are continually pre-trained on parallel data and then supervised fine-tuned with a small amount of high-quality parallel data. To investigate the effectiveness of our proposed approach, we conducted continual pre-training with a 3.8B-parameter model and parallel data across eight different formats. We evaluate these methods on thirteen test sets for Japanese-to-English and English-to-Japanese translation. The results demonstrate that when utilizing parallel data in continual pre-training, it is essential to alternate between source and target sentences. Additionally, we demonstrated that the translation accuracy improves only for translation directions where the order of source and target sentences aligns between continual pre-training data and inference. In addition, we demonstrate that the LLM-based translation model is more robust in translating spoken language and achieves higher accuracy with less training data compared to supervised encoder-decoder models. We also show that the highest accuracy is achieved when the data for continual pre-training consists of interleaved source and target sentences and when tags are added to the source sentences.
The KIT Speech Translation Systems for IWSLT 2024 Dialectal and Low-resource Track
Authors: Zhaolin Li; Enes Yavuz Ugan; Danni Liu; Carlos Mullov; Tu Anh Dinh; Sai Koneru; Alexander Waibel; Jan Niehues
Venue: ACL 2024 (Proceedings of the 21st International Conference on Spoken Language Translation (IWSLT 2024))

Abstract: This paper presents KIT’s submissions to the IWSLT 2024 dialectal and low-resource track. In this work, we build systems for translating into English from speech in Maltese, Bemba, and two Arabic dialects Tunisian and North Levantine. Under the unconstrained condition, we leverage the pre-trained multilingual models by fine-tuning them for the target language pairs to address data scarcity problems in this track. We build cascaded and end-to-end speech translation systems for different language pairs and show the cascaded system brings slightly better overall performance. Besides, we find utilizing additional data resources boosts speech recognition performance but slightly harms machine translation performance in cascaded systems. Lastly, we show that Minimum Bayes Risk is effective in improving speech translation performance by combining the cascaded and end-to-end systems, bringing a consistent improvement of around 1 BLUE point.
Empowering Low-Resource Language Translation: Methodologies for Bhojpuri-Hindi and Marathi-Hindi ASR and MT
Authors: Harpreet Singh Anand; Amulya Ratna Dash; Yashvardhan Sharma
Venue: ACL 2024 (Proceedings of the 21st International Conference on Spoken Language Translation (IWSLT 2024))

Abstract: The paper describes our submission for the unconstrained track of ‘Dialectal and Low-Resource Task’ proposed in IWSLT-2024. We designed cascaded Speech Translation systems for the language pairs Marathi-Hindi and Bhojpuri-Hindi utilising and fine-tuning different pre-trained models for carrying out Automatic Speech Recognition (ASR) and Machine Translation (MT).
Recent Highlights in Multilingual and Multimodal Speech Translation
Authors: Danni Liu; Jan Niehues
Venue: ACL 2024 (Proceedings of the 21st International Conference on Spoken Language Translation (IWSLT 2024))

Abstract: Speech translation has witnessed significant progress driven by advancements in modeling techniques and the growing availability of training data. In this paper, we highlight recent advances in two ongoing research directions in ST: scaling the models to 1) many translation directions (multilingual ST) and 2) beyond the text output modality (multimodal ST). We structure this review by examining the sequential stages of a model’s development lifecycle: determining training resources, selecting model architecture, training procedures, evaluation metrics, and deployment considerations. We aim to highlight recent developments in each stage, with a particular focus on model architectures (dedicated speech translation models and LLM-based general-purpose model) and training procedures (task-specific vs. task-invariant approaches). Based on the reviewed advancements, we identify and discuss ongoing challenges within the field of speech translation.
Word Order in English-Japanese Simultaneous Interpretation: Analyses and Evaluation using Chunk-wise Monotonic Translation
Authors: Kosuke Doi; Yuka Ko; Mana Makinae; Katsuhito Sudoh; Satoshi Nakamura
Venue: ACL 2024 (Proceedings of the 21st International Conference on Spoken Language Translation (IWSLT 2024))

Abstract: This paper analyzes the features of monotonic translations, which follow the word order of the source language, in simultaneous interpreting (SI). Word order differences are one of the biggest challenges in SI, especially for language pairs with significant structural differences like English and Japanese. We analyzed the characteristics of chunk-wise monotonic translation (CMT) sentences using the NAIST English-to-Japanese Chunk-wise Monotonic Translation Evaluation Dataset and identified some grammatical structures that make monotonic translation difficult in English-Japanese SI. We further investigated the features of CMT sentences by evaluating the output from the existing speech translation (ST) and simultaneous speech translation (simulST) models on the NAIST English-to-Japanese Chunk-wise Monotonic Translation Evaluation Dataset as well as on existing test sets. The results indicate the possibility that the existing SI-based test set underestimates the model performance. The results also suggest that using CMT sentences as references gives higher scores to simulST models than ST models, and that using an offline-based test set to evaluate the simulST models underestimates the model performance.
Leveraging Synthetic Audio Data for End-to-End Low-Resource Speech Translation
Authors: Yasmin Moslem
Venue: ACL 2024 (Proceedings of the 21st International Conference on Spoken Language Translation (IWSLT 2024))

Abstract: This paper describes our system submission to the International Conference on Spoken Language Translation (IWSLT 2024) for Irish-to-English speech translation. We built end-to-end systems based on Whisper, and employed a number of data augmentation techniques, such as speech back-translation and noise augmentation. We investigate the effect of using synthetic audio data and discuss several methods for enriching signal diversity.
HW-TSC’s Simultaneous Speech Translation System for IWSLT 2024
Authors: Shaojun Li; Zhiqiang Rao; Bin Wei; Yuanchang Luo; Zhanglin Wu; Zongyao Li; Hengchao Shang; Jiaxin Guo; Daimeng Wei; Hao Yang
Venue: ACL 2024 (Proceedings of the 21st International Conference on Spoken Language Translation (IWSLT 2024))

Abstract: This paper outlines our submission for the IWSLT 2024 Simultaneous Speech-to-Text (SimulS2T) and Speech-to-Speech (SimulS2S) Translation competition. We have engaged in all four language directions and both the SimulS2T and SimulS2S tracks: English-German (EN-DE), English-Chinese (EN-ZH), English-Japanese (EN-JA), and Czech-English (CS-EN). For the S2T track, we have built upon our previous year’s system and further honed the cascade system composed of ASR model and MT model. Concurrently, we have introduced an end-to-end system specifically for the CS-EN direction. This end-to-end (E2E) system primarily employs the pre-trained seamlessM4T model. In relation to the SimulS2S track, we have integrated a novel TTS model into our SimulS2T system. The final submission for the S2T directions of EN-DE, EN-ZH, and EN-JA has been refined over our championship system from last year. Building upon this foundation, the incorporation of the new TTS into our SimulS2S system has resulted in the ASR-BLEU surpassing last year’s best score.
UoM-DFKI submission to the low resource shared task
Authors: Kumar Rishu; Aiden Williams; Claudia Borg; Simon Ostermann
Venue: ACL 2024 (Proceedings of the 21st International Conference on Spoken Language Translation (IWSLT 2024))

Abstract: This system description paper presents the details of our primary and contrastive approaches to translating Maltese into English for IWSLT 24. The Maltese language shares a large vocabulary with Arabic and Italian languages, thus making it an ideal candidate to test the cross-lingual capabilities of recent state-of-the-art models. We experiment with two end-to-end approaches for our submissions: the Whisper and wav2vec 2.0 models. Our primary system gets a BLEU score of 35.1 on the combined data, whereas our contrastive approach gets 18.5. We also provide a manual analysis of our contrastive approach to identify some pitfalls that may have caused this difference.
HW-TSC’s submission to the IWSLT 2024 Subtitling track
Authors: Yuhao Xie; Yuanchang Luo; Zongyao Li; Zhanglin Wu; Xiaoyu Chen; Zhiqiang Rao; Shaojun Li; Hengchao Shang; Jiaxin Guo; Daimeng Wei; Hao Yang
Venue: ACL 2024 (Proceedings of the 21st International Conference on Spoken Language Translation (IWSLT 2024))

Abstract: This paper introduces HW-TSC’s submission to the IWSLT 2024 Subtitling track. For the automatic subtitling track, we use an unconstrained cascaded strategy, with the main steps being: ASR with word-level timestamps, sentence segmentation based on punctuation restoration, further alignment using CTC or using machine translation with length penalty. For the subtitle compression track, we employ a subtitle compression strategy that integrates machine translation models and extensive rewriting models. We acquire the subtitle text requiring revision through the CPS index, then utilize a translation model to obtain the English version of this text. Following this, we extract the compressed-length subtitle text through controlled decoding. If this method fails to compress the text successfully, we resort to the Llama2 few-shot model for further compression.
Charles Locock, Lowcock or Lockhart? Offline Speech Translation: Test Suite for Named Entities
Authors: Maximilian Awiszus; Jan Niehues; Marco Turchi; Sebastian Stüker; Alex Waibel
Venue: ACL 2024 (Proceedings of the 21st International Conference on Spoken Language Translation (IWSLT 2024))

Abstract: Generating rare words is a challenging task for natural language processing in general and in speech translation (ST) specifically. This paper introduces a test suite prepared for the Offline ST shared task at IWSLT. In the test suite, corresponding rare words (i.e. named entities) were annotated on TED-Talks for English and German and the English side was made available to the participants together with some distractors (irrelevant named entities). Our evaluation checks the capabilities of ST systems to leverage the information in the contextual list of named entities and improve translation quality. Systems are ranked based on the recall and precision of named entities (separately on person, location, and organization names) in the translated texts. Our evaluation shows that using contextual information improves translation quality as well as the recall and precision of NEs. The recall of organization names in all submissions is the lowest of all categories with a maximum of 87.5 % confirming the difficulties of ST systems in dealing with names.
Fixed and Adaptive Simultaneous Machine Translation Strategies Using Adapters
Authors: Abderrahmane Issam; Yusuf Can Semerci; Jan Scholtes; Gerasimos Spanakis
Venue: ACL 2024 (Proceedings of the 21st International Conference on Spoken Language Translation (IWSLT 2024))

Abstract: Simultaneous machine translation aims at solving the task of real-time translation by starting to translate before consuming the full input, which poses challenges in terms of balancing quality and latency of the translation. The wait-k policy offers a solution by starting to translate after consuming words, where the choice of the number k directly affects the latency and quality. In applications where we seek to keep the choice over latency and quality at inference, the wait-k policy obliges us to train more than one model. In this paper, we address the challenge of building one model that can fulfil multiple latency levels and we achieve this by introducing lightweight adapter modules into the decoder. The adapters are trained to be specialized for different wait-k values and compared to other techniques they offer more flexibility to allow for reaping the benefits of parameter sharing and minimizing interference. Additionally, we show that by combining with an adaptive strategy, we can further improve the results. Experiments on two language directions show that our method outperforms or competes with other strong baselines on most latency values.
IWSLT 2024 Indic Track system description paper: Speech-to-Text Translation from English to multiple Low-Resource Indian Languages
Authors: Deepanjali Singh; Ayush Anand; Abhyuday Chaturvedi; Niyati Baliyan
Venue: ACL 2024 (Proceedings of the 21st International Conference on Spoken Language Translation (IWSLT 2024))

Abstract: Our Speech-to-Text (ST) translation system addresses low-resource Indian languages (Hindi, Bengali, Tamil) by combining advanced transcription and translation models for accurate and efficient translations. The key components of the system are: The Audio Processor and Transcription Module which utilizes ResembleAI for noise reduction and OpenAI’s Whisper model for transcription. The Input Module validates and preprocesses audio files. The Translation Modules integrate the Helsinki-NLP model for English to Hindi translation and Facebook’s MBart model for English to Tamil and Bengali translations, fine-tuned for better quality. The Output Module corrects syntax and removes hallucinations, delivering the final translated text. For performance evaluation purpose, SacreBLEU scores were used and attained the following values: English-to-Hindi: 24.21 (baseline: 5.23); English-to-Bengali: 16.18 (baseline: 5.86); English-to-Tamil: 10.79 (baseline: 1.9). The solution streamlines workflow from input validation to output delivery, significantly enhancing communication across different linguistic contexts and achieving substantial improvements in SacreBLEU scores. Through the creation of dedicated datasets and the development of robust models, our aim is to facilitate seamless communication and accessibility across diverse linguistic communities, ultimately promoting inclusivity and empowerment.
Multi-hop Database Reasoning with Virtual Knowledge Graph
Authors: Juhee Son; Yeon Seonwoo; Seunghyun Yoon; James Thorne; Alice Oh
Venue: ACL 2024 (Proceedings of the 1st Workshop on Knowledge Graphs and Large Language Models (KaLLM 2024))

Abstract: Application of LLM to database queries on natural language sentences has demonstrated impressive results in both single and multi-hop scenarios.In the existing methodologies, the requirement to re-encode query vectors at each stage for processing multi-hop queries presents a significant bottleneck to the inference speed.This paper proposes VKGFR (Virtual Knowledge Graph based Fact Retriever) that leverages large language models to extract representations corresponding to a sentence’s knowledge graph, significantly enhancing inference speed for multi-hop reasoning without performance loss.Given that both the queries and natural language database sentences can be structured as a knowledge graph, we suggest extracting a Virtual Knowledge Graph (VKG) representation from sentences with LLM.Over the pre-constructed VKG, our VKGFR conducts retrieval with a tiny model structure, showing performance improvements with higher computational efficiency. We evaluate VKGFR on the WikiNLDB and MetaQA dataset, designed for multi-hop database reasoning over text. The results indicate 13x faster inference speed on the WikiNLDB dataset without performance loss.
Zero- and Few-Shots Knowledge Graph Triplet Extraction with Large Language Models
Authors: Andrea Papaluca; Daniel Krefl; Sergio Rodríguez Méndez; Artem Lensky; Hanna Suominen
Venue: ACL 2024 (Proceedings of the 1st Workshop on Knowledge Graphs and Large Language Models (KaLLM 2024))

Abstract: In this work, we tested the Triplet Extraction (TE) capabilities of a variety of Large Language Models (LLMs) of different sizes in the Zero- and Few-Shots settings. In detail, we proposed a pipeline that dynamically gathers contextual information from a Knowledge Base (KB), both in the form of context triplets and of (sentence, triplets) pairs as examples, and provides it to the LLM through a prompt. The additional context allowed the LLMs to be competitive with all the older fully trained baselines based on the Bidirectional Long Short-Term Memory (BiLSTM) Network architecture. We further conducted a detailed analysis of the quality of the gathered KB context, finding it to be strongly correlated with the final TE performance of the model. In contrast, the size of the model appeared to only logarithmically improve the TE capabilities of the LLMs. We release the code on GitHub for reproducibility.