GenAI4ELab/papercli-papers huggingface dataset:
MoQA: Benchmarking Multi-Type Open-Domain Question Answering
Authors: Howard Yen; Tianyu Gao; Jinhyuk Lee; Danqi Chen
Venue: ACL 2023 (Proceedings of the Third DialDoc Workshop on Document-grounded Dialogue and Conversational Question Answering)

Abstract: Previous research on open-domain question answering (QA) mainly focuses on questions with short answers. However, information-seeking QA often requires various formats of answers depending on the nature of the questions, e.g., why/how questions typically require a long answer. In this paper, we present MoQA, a benchmark for open-domain QA that requires building one system that can provide short, medium, long, and yes/no answers to different questions accordingly. MoQA builds upon Natural Questions with multiple types of questions and additional crowdsourcing efforts to ensure high query quality. We adapt state-of-the-art models, and reveal unique findings in multi-type open-domain QA: (1) For retriever-reader models, training one retriever on all types achieves the overall best performance, but it is challenging to train one reader model to output answers of different formats, or to train a question classifier to distinguish between types; (2) An end-to-end closed-book QA model trained on multiple types struggles with the task across the board; (3) State-of-the-art large language models such as the largest GPT-3 models (Brown et al., 2020; Ouyang et al., 2022) also lag behind open-book QA models. Our benchmark and analysis call for more effort into building versatile open-domain QA models in the future.
Exploration of multilingual prompts in document-grounded dialogue
Authors: Xiaocheng Zhang; Huang Qing; Fu Lin
Venue: ACL 2023 (Proceedings of the Third DialDoc Workshop on Document-grounded Dialogue and Conversational Question Answering)

Abstract: Transferring DGD models from high-resource languages to low-resource languages is a meaningful but challenging task. Being able to provide multilingual responses to multilingual documents further complicates the task. This paper describes our method at DialDoc23 Shared Task (Document-Grounded Dialogue and Conversational Question Answering) for generate responses based on the most relevant passage retrieved. We divide it into three steps of retrieval, re-ranking and generation. Our methods include negative sample augmentation, prompt learning, pseudo-labeling and ensemble. On the submission page, we rank 2nd based on the sum of token-level F1, SacreBleu and Rouge-L scores used for the final evaluation, and get the total score of 210.25.
Position Matters! Empirical Study of Order Effect in Knowledge-grounded Dialogue
Authors: Hsuan Su; Shachi H. Kumar; Sahisnu Mazumder; Wenda Chen; Ramesh Manuvinakurike; Eda Okur; Saurav Sahay; Lama Nachman; Shang-Tse Chen; Hung-yi Lee
Venue: ACL 2023 (Proceedings of the Third DialDoc Workshop on Document-grounded Dialogue and Conversational Question Answering)

Abstract: With the power of large pretrained language models, various research works have integrated knowledge into dialogue systems. The traditional techniques treat knowledge as part of the input sequence for the dialogue system, prepending a set of knowledge statements in front of dialogue history. However, such a mechanism forces knowledge sets to be concatenated in an ordered manner, making models implicitly pay imbalanced attention to the sets during training. In this paper, we first investigate how the order of the knowledge set can influence autoregressive dialogue systems’ responses. We conduct experiments on two commonly used dialogue datasets with two types of transformer-based models and find that models view the input knowledge unequally. To this end, we propose a simple and novel technique to alleviate the order effect by modifying the position embeddings of knowledge input in these models. With the proposed position embedding method, the experimental results show that each knowledge statement is uniformly considered to generate responses.
Enhancing Multilingual Document-Grounded Dialogue Using Cascaded Prompt-Based Post-Training Models
Authors: Jun Liu; Shuang Cheng; Zineng Zhou; Yang Gu; Jian Ye; Haiyong Luo
Venue: ACL 2023 (Proceedings of the Third DialDoc Workshop on Document-grounded Dialogue and Conversational Question Answering)

Abstract: The Dialdoc23 shared task presents a Multilingual Document-Grounded Dialogue Systems (MDGDS) challenge, where system responses are generated in multiple languages using user’s queries, historical dialogue records and relevant passages. A major challenge for this task is the limited training data available in low-resource languages such as French and Vietnamese. In this paper, we propose Cascaded Prompt-based Post-training Models, dividing the task into three subtasks: Retrieval, Reranking and Generation. We conduct post-training on high-resource language such as English and Chinese to enhance performance of low-resource languages by using the similarities of languages. Additionally, we utilize the prompt method to activate model’s ability on diverse languages within the dialogue domain and explore which prompt is a good prompt. Our comprehensive experiments demonstrate the effectiveness of our proposed methods, which achieved the first place on the leaderboard with a total score of 215.40 in token-level F1, SacreBleu, and Rouge-L metrics.
Enhanced Training Methods for Multiple Languages
Authors: Hai Li; Yang Li
Venue: ACL 2023 (Proceedings of the Third DialDoc Workshop on Document-grounded Dialogue and Conversational Question Answering)

Abstract: Document-grounded dialogue generation based on multilingual is a challenging and realistic task. Unlike previous tasks, it need to tackle with multiple high-resource languages facilitating low-resource languages. This paper summarizes our research based on a three-stage pipeline that includes retrieval, re-rank and generation where each component is individually optimized. In different languages with limited data scenarios, we mainly improve the robustness of the pipeline through data augmentation and embedding perturbation with purpose of improving the performance designing three training methods: cross-language enhancement training, weighted training with neighborhood distribution augmentation, and ensemble adversarial training, all of that can be used as plug and play modules. Through experiments with different settings, it has been shown that our methods can effectively improve the generalization performance of pipeline with score ranking 6th among the public submissions on leaderboards.
SLDT: Sequential Latent Document Transformer for Multilingual Document-based Dialogue
Authors: Zhanyu Ma; Zeming Liu; Jian Ye
Venue: ACL 2023 (Proceedings of the Third DialDoc Workshop on Document-grounded Dialogue and Conversational Question Answering)

Abstract: Multilingual document-grounded dialogue, where the system is required to generate responses based on both the conversation Multilingual context and external knowledge sources. Traditional pipeline methods for knowledge identification and response generation, while effective in certain scenarios, suffer from error propagation issues and fail to capture the interdependence between these two sub-tasks. To overcome these challenges, we propose the application of the SLDT method, which treats passage-knowledge selection as a sequential decision process rather than a single-step decision process. We achieved winner 3rd in dialdoc 2023 and we also validated the effectiveness of our method on other datasets. The ablation experiment also shows that our method significantly improves the basic model compared to other methods.
A Dialogue System for Assessing Activities of Daily Living: Improving Consistency with Grounded Knowledge
Authors: Zhecheng Sheng; Raymond Finzel; Michael Lucke; Sheena Dufresne; Maria Gini; Serguei Pakhomov
Venue: ACL 2023 (Proceedings of the Third DialDoc Workshop on Document-grounded Dialogue and Conversational Question Answering)

Abstract: In healthcare, the ability to care for oneself is reflected in the “Activities of Daily Living (ADL),” which serve as a measure of functional ability (functioning). A lack of functioning may lead to poor living conditions requiring personal care and assistance. To accurately identify those in need of support, assistance programs continuously evaluate participants’ functioning across various domains. However, the assessment process may encounter consistency issues when multiple assessors with varying levels of expertise are involved. Novice assessors, in particular, may lack the necessary preparation for real-world interactions with participants. To address this issue, we developed a dialogue system that simulates interactions between assessors and individuals of varying functioning in a natural and reproducible way. The dialogue system consists of two major modules, one for natural language understanding (NLU) and one for natural language generation (NLG), respectively. In order to generate responses consistent with the underlying knowledge base, the dialogue system requires both an understanding of the user’s query and of biographical details of an individual being simulated. To fulfill this requirement, we experimented with query classification and generated responses based on those biographical details using some recently released InstructGPT-like models.
C-PMI: Conditional Pointwise Mutual Information for Turn-level Dialogue Evaluation
Authors: Liliang Ren; Mankeerat Sidhu; Qi Zeng; Revanth Gangi Reddy; Heng Ji; ChengXiang Zhai
Venue: ACL 2023 (Proceedings of the Third DialDoc Workshop on Document-grounded Dialogue and Conversational Question Answering)

Abstract: Existing reference-free turn-level evaluation metrics for chatbots inadequately capture the interaction between the user and the system. Consequently, they often correlate poorly with human evaluations. To address this issue, we propose a novel model-agnostic approach that leverages Conditional Pointwise Mutual Information (C-PMI) to measure the turn-level interaction between the system and the user based on a given evaluation dimension. Experimental results on the widely used FED dialogue evaluation dataset demonstrate that our approach significantly improves the correlation with human judgment compared with existing evaluation systems. By replacing the negative log-likelihood-based scorer with our proposed C-PMI scorer, we achieve a relative 60.5% higher Spearman correlation on average for the FED evaluation metric. Our code is publicly available at https://github.com/renll/C-PMI.
ConvRGX: Recognition, Generation, and Extraction for Self-trained Conversational Question Answering
Authors: Tianhua Zhang; Liping Tang; Wei Fang; Hongyin Luo; Xixin Wu; Helen Meng; James Glass
Venue: ACL 2023 (Proceedings of the Third DialDoc Workshop on Document-grounded Dialogue and Conversational Question Answering)

Abstract: Collecting and constructing human-annotated corpora for training conversational question-answering (CQA) models has recently been shown to be inefficient and costly. To solve this problem, previous works have proposed training QA models with automatically generated QA data. In this work, we extend earlier studies on QA synthesis, and propose an efficient QA data generation algorithm under conversational settings. Our model recognizes potential dialogue topics, generates corresponding questions, and extracts answers from grounding passages. To improve the quality of generated QAs and downstream self-training of CQA models, we propose dropout and agreement-based QA selection methods. We conduct experiments on both data augmentation and domain adaptation settings. Experiments on the QuAC and Doc2Dial tasks show that the proposed method can significantly improve the quality of generated QA data, and also improves the accuracy of self-trained CQA models based on the constructed training corpora.
Language-Agnostic Transformers and Assessing ChatGPT-Based Query Rewriting for Multilingual Document-Grounded QA
Authors: Srinivas Gowriraj; Soham Dinesh Tiwari; Mitali Potnis; Srijan Bansal; Teruko Mitamura; Eric Nyberg
Venue: ACL 2023 (Proceedings of the Third DialDoc Workshop on Document-grounded Dialogue and Conversational Question Answering)

Abstract: The DialDoc 2023 shared task has expanded the document-grounded dialogue task to encompass multiple languages, despite having limited annotated data. This paper assesses the effectiveness of both language-agnostic and language-aware paradigms for multilingual pre-trained transformer models in a bi-encoder-based dense passage retriever (DPR), concluding that the language-agnostic approach is superior. Additionally, the study investigates the impact of query rewriting techniques using large language models, such as ChatGPT, on multilingual, document-grounded question-answering systems. The experiments conducted demonstrate that, for the examples examined, query rewriting does not enhance performance compared to the original queries. This failure is due to topic switching in final dialogue turns and irrelevant topics being considered for query rewriting.
Follow the Knowledge: Structural Biases and Artefacts in Knowledge Grounded Dialog Datasets
Authors: Ehsan Lotfi; Maxime De Bruyn; Jeska Buhmann; Walter Daelemans
Venue: ACL 2023 (Proceedings of the Third DialDoc Workshop on Document-grounded Dialogue and Conversational Question Answering)

Abstract: Crowd-sourcing has been one of the primary ways to curate conversational data, specially for certain scenarios like grounding in knowledge. In this setting, using online platforms like AMT, non-expert participants are hired to converse with each other, following instructions which try to guide the outcome towards the desired format. The resulting data then is used for different parts of dialog modelling like knowledge selection and response selection/generation. In this work, we take a closer look into two of the most popular knowledge grounded dialog (KGD) datasets. Investigating potential biases and artefacts in knowledge selection labels, we observe that in many cases the ‘knowledge selection flow’ simply follows the order of presented knowledge pieces. In Wizard of Wikipedia (the most popular KGD dataset) we use simple content-agnostic models based on this bias to get significant knowledge selection performance. In Topical-Chat we see a similar correlation between the knowledge selection sequence and the order of entities and their segments, as provided to crowd-source workers. We believe that the observed results, question the significance and origin of the presumed dialog-level attributes like ‘knowledge flow’ in these crowd-sourced datasets.
The DISRPT 2023 Shared Task on Elementary Discourse Unit Segmentation, Connective Detection, and Relation Classification
Authors: Chloé Braud; Yang Janet Liu; Eleni Metheniti; Philippe Muller; Laura Rivière; Attapol Rutherford; Amir Zeldes
Venue: ACL 2023 (Proceedings of the 3rd Shared Task on Discourse Relation Parsing and Treebanking (DISRPT 2023))

Abstract: In 2023, the third iteration of the DISRPT Shared Task (Discourse Relation Parsing and Treebanking) was held, dedicated to the underlying units used in discourse parsing across formalisms. Following the success of the 2019and 2021 tasks on Elementary Discourse Unit Segmentation, Connective Detection, and Relation Classification, this iteration has added 10 new corpora, including 2 new languages (Thai and Italian) and 3 discourse treebanks annotated in the discourse dependency representation in addition to the previously included frameworks: RST, SDRT, and PDTB. In this paper, we review the data included in the Shared Task, which covers 26 datasets across 13 languages, survey and compare submitted systems, and report on system performance on each task for both annotated and plain-tokenized versions of the data.
DiscoFlan: Instruction Fine-tuning and Refined Text Generation for Discourse Relation Label Classification
Authors: Kaveri Anuranjana
Venue: ACL 2023 (Proceedings of the 3rd Shared Task on Discourse Relation Parsing and Treebanking (DISRPT 2023))

Abstract: This paper introduces DiscoFlan, a multilingual discourse relation classifier submitted for DISRPT 2023. Our submission represents the first attempt at building a multilingual discourse relation classifier for the DISRPT 2023 shared task. By our model addresses the issue to mismatches caused by hallucination in a seq2seq model by utilizing the label distribution information for label generation. In contrast to the previous state-of-the-art model, our approach eliminates the need for hand-crafted features in computing the discourse relation classes. Furthermore, we propose a novel label generation mechanism that anchors the labels to a fixed set by selectively enhancing training on the decoder model. Our experimental results demonstrate that our model surpasses the current state-of-the-art performance in 11 out of the 26 datasets considered, however the submitted model compatible with provided evaluation scripts is better in 7 out of 26 considered datasets, while demonstrating competitive results in the rest. Overall, DiscoFlan showcases promising advancements in multilingual discourse relation classification for the DISRPT 2023 shared task.
DisCut and DiscReT: MELODI at DISRPT 2023
Authors: Eleni Metheniti; Chloé Braud; Philippe Muller; Laura Rivière
Venue: ACL 2023 (Proceedings of the 3rd Shared Task on Discourse Relation Parsing and Treebanking (DISRPT 2023))

Abstract: This paper presents the results obtained by the MELODI team for the three tasks proposed within the DISRPT 2023 shared task on discourse: segmentation, connective identification, and relation classification. The competition involves corpora in various languages in several underlying frameworks, and proposes two tracks depending on the presence or not of annotations of sentence boundaries and syntactic information. For these three tasks, we rely on a transformer-based architecture, and investigate several optimizations of the models, including hyper-parameter search and layer freezing. For discourse relations, we also explore the use of adapters—a lightweight solution for model fine-tuning—and introduce relation mappings to partially deal with the label set explosion we are facing within the setting of the shared task in a multi-corpus perspective. In the end, we propose one single architecture for segmentation and connectives, based on XLM-RoBERTa large, freezed at lower layers, with new state-of-the-art results for segmentation, and we propose 3 different models for relations, since the task makes it harder to generalize across all corpora.
HITS at DISRPT 2023: Discourse Segmentation, Connective Detection, and Relation Classification
Authors: Wei Liu; Yi Fan; Michael Strube
Venue: ACL 2023 (Proceedings of the 3rd Shared Task on Discourse Relation Parsing and Treebanking (DISRPT 2023))

Abstract: HITS participated in the Discourse Segmentation (DS, Task 1) and Connective Detection (CD, Task 2) tasks at the DISRPT 2023. Task 1 focuses on segmenting the text into discourse units, while Task 2 aims to detect the discourse connectives. We deployed a framework based on different pre-trained models according to the target language for these two tasks.HITS also participated in the Relation Classification track (Task 3). The main task was recognizing the discourse relation between text spans from different languages. We designed a joint model for languages with a small corpus while separate models for large corpora. The adversarial training strategy is applied to enhance the robustness of relation classifiers.
FINDINGS OF THE IWSLT 2023 EVALUATION CAMPAIGN
Authors: Milind Agarwal; Sweta Agrawal; Antonios Anastasopoulos; Luisa Bentivogli; Ondřej Bojar; Claudia Borg; Marine Carpuat; Roldano Cattoni; Mauro Cettolo; Mingda Chen; William Chen; Khalid Choukri; Alexandra Chronopoulou; Anna Currey; Thierry Declerck; Qianqian Dong; Kevin Duh; Yannick Estève; Marcello Federico; Souhir Gahbiche; Barry Haddow; Benjamin Hsu; Phu Mon Htut; Hirofumi Inaguma; Dávid Javorský; John Judge; Yasumasa Kano; Tom Ko; Rishu Kumar; Pengwei Li; Xutai Ma; Prashant Mathur; Evgeny Matusov; Paul McNamee; John P. McCrae; Kenton Murray; Maria Nadejde; Satoshi Nakamura; Matteo Negri; Ha Nguyen; Jan Niehues; Xing Niu; Atul Kr. Ojha; John E. Ortega; Proyag Pal; Juan Pino; Lonneke van der Plas; Peter Polák; Elijah Rippeth; Elizabeth Salesky; Jiatong Shi; Matthias Sperber; Sebastian Stüker; Katsuhito Sudoh; Yun Tang; Brian Thompson; Kevin Tran; Marco Turchi; Alex Waibel; Mingxuan Wang; Shinji Watanabe; Rodolfo Zevallos
Venue: ACL 2023 (Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023))

Abstract: This paper reports on the shared tasks organized by the 20th IWSLT Conference. The shared tasks address 9 scientific challenges in spoken language translation: simultaneous and offline translation, automatic subtitling and dubbing, speech-to-speech translation, multilingual, dialect and low-resource speech translation, and formality control. The shared tasks attracted a total of 38 submissions by 31 teams. The growing interest towards spoken language translation is also witnessed by the constantly increasing number of shared task organizers and contributors to the overview paper, almost evenly distributed across industry and academia.
Evaluating Multilingual Speech Translation under Realistic Conditions with Resegmentation and Terminology
Authors: Elizabeth Salesky; Kareem Darwish; Mohamed Al-Badrashiny; Mona Diab; Jan Niehues
Venue: ACL 2023 (Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023))

Abstract: We present the ACL 60/60 evaluation sets for multilingual translation of ACL 2022 technical presentations into 10 target languages. This dataset enables further research into multilingual speech translation under realistic recording conditions with unsegmented audio and domain-specific terminology, applying NLP tools to text and speech in the technical domain, and evaluating and improving model robustness to diverse speaker demographics.
The MineTrans Systems for IWSLT 2023 Offline Speech Translation and Speech-to-Speech Translation Tasks
Authors: Yichao Du; Guo Zhengsheng; Jinchuan Tian; Zhirui Zhang; Xing Wang; Jianwei Yu; Zhaopeng Tu; Tong Xu; Enhong Chen
Venue: ACL 2023 (Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023))

Abstract: This paper presents the extscMineTrans English-to-Chinese speech translation systems developed for two challenge tracks of IWSLT 2023, i.e., Offline Speech Translation (S2T) and Speech-to-Speech Translation (S2ST). For the S2T track, extscMineTrans employs a practical cascaded system to explore the limits of translation performance in both constrained and unconstrained settings, where the whole system consists of automatic speech recognition (ASR), punctuation recognition (PC), and machine translation (MT) modules. We also investigate the effectiveness of multiple ASR architectures and explore two MT strategies: supervised in-domain fine-tuning and prompt-guided translation using a large language model. For the S2ST track, we explore a speech-to-unit (S2U) framework to build an end-to-end S2ST system. This system encodes the target speech as discrete units via our trained HuBERT. Then it leverages the standard sequence-to-sequence model to directly learn the mapping between source speech and discrete units without any auxiliary recognition tasks (i.e., ASR and MT tasks). Various efforts are made to improve the extscMineTrans’s performance, such as acoustic model pre-training on large-scale data, data filtering, data augmentation, speech segmentation, knowledge distillation, consistency training, model ensembles, etc.