Reviewerly/exHarmony huggingface dataset:
Probing Graph Decomposition for Argument Pair Extraction

Argument pair extraction (APE) aims to extract interactive argument pairs from two passages within a discussion. The key challenge of APE is to effectively capture the complex context-aware interactive relations of arguments between the two passages. In this paper, we elicit relational semantic knowledge from large-scale pre-trained language models (PLMs) via a probing technique. The induced sentence-level relational probing graph can help capture rich explicit interactive relations between argument pairs effectively. Since the relevance score of a sentence pair within a passage is generally larger than that of the sentence pair from different passages, each sentence would prefer to propagate information within the same passage and under-explore the interactive relations between two passages. To tackle this issue, we propose a graph decomposition method to decompose the probing graph into four sub-graphs from intra- and inter-passage perspectives, where the intra-passage graphs can help detect argument spans within each passage and the inter-passage graphs can help identify the argument pairs between the review and rebuttal passages. Experimental results on two benchmark datasets show that our method achieves substantial improvements over strong baselines for APE.
Modular Transformers: Compressing Transformers into Modularized Layers for Flexible Efficient Inference

Pre-trained Transformer models like T5 and BART have advanced the state of the art on a wide range of text generation tasks.Compressing these models into smaller ones has become critically important for practical use.Common neural network compression techniques such as knowledge distillation or quantization are limited to static compression where the compression ratio is fixed.In this paper, we introduce Modular Transformers, a modularized encoderdecoder framework for flexible sequence-tosequence model compression.Modular Transformers trains modularized layers that have the same function of two or more consecutive layers in the original model via module replacing and knowledge distillation.After training, the modularized layers can be flexibly assembled into sequence-to-sequence models that meet different performance-efficiency trade-offs.Experimental results show that after a single training phase, by simply varying the assemble strategy, Modular Transformers can achieve flexible compression ratios from 1.1× to 6× with little to moderate relative performance drop.
Developing finite-state language technology for Maya

We describe a suite of finite-state language technologies for Maya, a Mayan language spoken in Mexico.At the core is a computational model of Maya morphology and phonology using a finite-state transducer. 1 This model results in a morphological analyzer and a morphologically-informed spell-checker.All of these technologies are designed for use as both a pedagogical reading/writing aid for L2 learners and as a general language processing tool capable of supporting much of the natural variation in written Maya.We discuss the relevant features of Maya morphosyntax and orthography, and then outline the implementation details of the analyzer.To conclude, we present a longer-term vision for these tools and their use by both native speakers and learners.
Latent Positional Information is in the Self-Attention Variance of Transformer Language Models Without Positional Embeddings

The use of positional embeddings in transformer language models is widely accepted.However, recent research has called into question the necessity of such embeddings.We further extend this inquiry by demonstrating that a randomly initialized and frozen transformer language model, devoid of positional embeddings, inherently encodes strong positional information through the shrinkage of self-attention variance.To quantify this variance, we derive the underlying distribution of each step within a transformer layer.Through empirical validation using a fully pretrained model, we show that the variance shrinkage effect still persists after extensive gradient updates.Our findings serve to justify the decision to discard positional embeddings and thus facilitate more efficient pretraining of transformer language models.
Abstract then Play: A Skill-centric Reinforcement Learning Framework for Text-based Games

Text-based games present an exciting test-bed for reinforcement learning algorithms in the natural language environment.In these adventure games, an agent must learn to interact with the environment through text in order to accomplish tasks, facing large and combinational action space as well as partial observability issues.However, existing solutions fail to decompose the task and abstract the action autonomously, which either pre-specify the subtasks or pretrain on the human gameplay dataset.In this work, we introduce a novel skill-centric reinforcement learning framework, which is capable of abstracting the action in an end-to-end manner.To learn a more disentangled skill, we focus on the informativeness and distinguishability of the skill in accordance with the information bottleneck principle.Specifically, we introduce a discriminator to enable the skill to reflect the trajectory and push their representations onto the unit hypersphere to distribute uniformly.Moreover, a self-predictive mechanism is employed to learn inverse and forward dynamics, and a self-recovery mechanism is leveraged to refine the action representation, thus resulting in a more comprehensive perception of dynamics and more effective representations of textual state and action.Empirical experiments are carried out on the Jericho environment and the results validate the superiority against state-of-the-art baselines.
MarSan at SemEval-2023 Task 10: Can Adversarial Training with help of a Graph Convolutional Network Detect Explainable Sexism?

This paper describes SemEval-2022's shared task "Explainable Detection of Online Sexism".The fine-grained classification of sexist content plays a major role in building explainable frameworks for online sexism detection.We hypothesize that by encoding dependency information using Graph Convolutional Networks (GCNs) we may capture more stylistic information about sexist contents.Online sexism has the potential to cause significant harm to women who are the targets of such behavior.It not only creates unwelcoming and inaccessible spaces for women online but also perpetuates social asymmetries and injustices.We believed improving the robustness and generalization ability of neural networks during training will allow models to capture different belief distributions for sexism categories.So we proposed adversarial training with GCNs for explainable detection of online sexism.In the end, our proposed method achieved very competitive results in all subtasks and shows that adversarial training of GCNs is a promising method for the explainable detection of online sexism.
Surface-Based Retrieval Reduces Perplexity of Retrieval-Augmented Language Models

Augmenting language models with a retrieval mechanism has been shown to significantly improve their performance while keeping the number of parameters low.Retrieval-augmented models commonly rely on a semantic retrieval mechanism based on the similarity between dense representations of the query chunk and potential neighbors.In this paper, we study the state-of-the-art RETRO model and observe that its performance gain is better explained by surface-level similarities, such as token overlap.Inspired by this, we replace the semantic retrieval in RETRO with a surface-level method based on BM25, obtaining a significant reduction in perplexity.As full BM25 retrieval can be computationally costly for large datasets, we also apply it in a re-ranking scenario, gaining part of the perplexity reduction with minimal computational overhead.
Calvados at MEDIQA-Chat 2023: Improving Clinical Note Generation with Multi-Task Instruction Finetuning

This paper presents our system for the MEDIQA-Chat 2023 shared task on medical conversation summarization.Our approach involves finetuning a LongT5 model on multiple tasks simultaneously, which we demonstrate improves the model's overall performance while reducing the number of factual errors and hallucinations in the generated summary.Furthermore, we investigated the effect of augmenting the data with in-text annotations from a clinical named entity recognition model, finding that this approach decreased summarization quality.Lastly, we explore using different text generation strategies for medical note generation based on the length of the note.Our findings suggest that the application of our proposed approach can be beneficial for improving the accuracy and effectiveness of medical conversation summarization.
Two Birds One Stone: Dynamic Ensemble for OOD Intent Classification

Out-of-domain (OOD) intent classification is an active field of natural language understanding, which is of great practical significance for intelligent devices such as the Task-Oriented Dialogue System.It mainly contains two challenges: it requires the model to know what it knows and what it does not know.This paper investigates "overthinking" in the openworld scenario and its impact on OOD intent classification.Inspired by this, we propose a two-birds-one-stone method, which allows the model to decide whether to make a decision on OOD classification early during inference and can ensure accuracy and accelerate inference.At the same time, to adapt to the behavior of dynamic inference, we also propose a training method based on ensemble methods.In addition to bringing certain theoretical insights, we also conduct detailed experiments on three real-world intent datasets.Compared with the previous baselines, our method can not only improve inference speed, but also achieve significant performance improvements.Code is publicly available.
UMR-Writer 2.0: Incorporating a New Keyboard Interface and Workflow into UMR-Writer

UMR-Writer is a web-based tool for annotating semantic graphs for the Uniform Meaning Representation (UMR) scheme.UMR is a graph-based semantic representation that can be applied cross-linguistically for deep semantic analysis of text.In this work, we implemented a new keyboard interface for UMR-Writer 2.0, which adds to the original clickbased interface to support faster annotation for more experienced annotators.The new interface also addresses some issues with the original click-based interface.Additionally, we demonstrate an efficient workflow for annotation project management in UMR-Writer 2.0, which has been applied to many projects.
A Systematic Study of Knowledge Distillation for Natural Language Generation with Pseudo-Target Training

Modern Natural Language Generation (NLG) models come with massive computational and storage requirements.In this work, we study the potential of compressing them, which is crucial for real-world applications serving millions of users.We focus on Knowledge Distillation (KD) techniques, in which a small student model learns to imitate a large teacher model, allowing to transfer knowledge from the teacher to the student.In contrast to much of the previous work, our goal is to optimize the model for a specific NLG task and a specific dataset.Typically in real-world applications, in addition to labeled data there is abundant unlabeled task-specific data, which is crucial for attaining high compression rates via KD.In this work, we conduct a systematic study of task-specific KD techniques for various NLG tasks under realistic assumptions.We discuss the special characteristics of NLG distillation and particularly the exposure bias problem.Following, we derive a family of Pseudo-Target (PT) augmentation methods, substantially extending prior work on sequence-level KD.We propose the Joint-Teaching method, which applies wordlevel KD to multiple PTs generated by both the teacher and the student.Finally, we validate our findings in an extreme setup with no labeled examples using GPT-4 as the teacher.Our study provides practical model design observations and demonstrates the effectiveness of PT training for task-specific KD in NLG.
UMUTeam at SemEval-2023 Task 12: Ensemble Learning of LLMs applied to Sentiment Analysis for Low-resource African Languages

These working notes summarize the participation of the UMUTeam in the SemEval 2023 shared task: AfriSenti, focused on Sentiment Analysis in several African languages.Two subtasks are proposed, one in which each language is considered separately and another one in which all languages are merged.Our proposal to solve both subtasks is grounded on the combination of features extracted from several multilingual Large Language Models and a subset of language-independent linguistic features.Our best results are achieved with the African languages less represented in the training set: Xitsonga, a Mozambique dialect, with a weighted f1-score of 54.89%; Algerian Arabic, with a weighted f1-score of 68.52%; Swahili, with a weighted f1-score of 60.52%; and Twi, with a weighted f1-score of 71.14%.
A Dataset of Argumentative Dialogues on Scientific Papers

With recent advances in question-answering models, various datasets have been collected to improve and study the effectiveness of these models on scientific texts.Questions and answers in these datasets explore a scientific paper by seeking factual information from the paper's content.However, these datasets do not tackle the argumentative content of scientific papers, which is of huge importance in persuasiveness of a scientific discussion.We introduce ArgSciChat, a dataset of 41 argumentative dialogues between scientists on 20 NLP papers.The unique property of our dataset is that it includes both exploratory and argumentative questions and answers in a dialogue discourse on a scientific paper.Moreover, the size of ArgSciChat demonstrates the difficulties in collecting dialogues for specialized domains.Thus, our dataset is a challenging resource to evaluate dialogue agents in low-resource domains, in which collecting training data is costly.We annotate all sentences of dialogues in ArgSciChat and analyze them extensively.The results confirm that dialogues in ArgSci-Chat include exploratory and argumentative interactions.Furthermore, we use our dataset to fine-tune and evaluate a pre-trained documentgrounded dialogue agent.The agent achieves a low performance on our dataset, motivating a need for dialogue agents with a capability to reason and argue about their answers.We publicly release ArgSciChat 1 .
Exclusive Supermask Subnetwork Training for Continual Learning

Continual Learning (CL) methods focus on accumulating knowledge over time while avoiding catastrophic forgetting.Recently, Wortsman et al. (2020) proposed a CL method, SupSup, which uses a randomly initialized, fixed base network (model) and finds a supermask for each new task that selectively keeps or removes each weight to produce a subnetwork.They prevent forgetting as the network weights are not being updated.Although there is no forgetting, the performance of SupSup is sub-optimal because fixed weights restrict its representational power.Furthermore, there is no accumulation or transfer of knowledge inside the model when new tasks are learned.Hence, we propose EXSSNET (Exclusive Supermask SubNEtwork Training), that performs exclusive and non-overlapping subnetwork weight training.This avoids conflicting updates to the shared weights by subsequent tasks to improve performance while still preventing forgetting.Furthermore, we propose a novel KNN-based Knowledge Transfer (KKT) module that utilizes previously acquired knowledge to learn new tasks better and faster.We demonstrate that EXSSNET outperforms strong previous methods on both NLP and Vision domains while preventing forgetting.Moreover, EXSSNET is particularly advantageous for sparse masks that activate 2-10% of the model parameters, resulting in an average improvement of 8.3% over SupSup.Furthermore, EXSSNET scales to a large number of tasks (100).Our code is available at https://github.com/prateeky2806/exessnet.
Multi-Source (Pre-)Training for Cross-Domain Measurement, Unit and Context Extraction

We present a cross-domain approach for automated measurement and context extraction based on pre-trained language models.We construct a multi-source, multi-domain corpus and train an end-to-end extraction pipeline.We then apply multi-source task-adaptive pre-training and fine-tuning to benchmark the cross-domain generalization capability of our model.Further, we conceptualize and apply a task-specific error analysis and derive insights for future work.Our results suggest that multi-source training leads to the best overall results, while singlesource training yields the best results for the respective individual domain.While our setup is successful at extracting quantity values and units, more research is needed to improve the extraction of contextual entities.We make the cross-domain corpus used in this work available online 1 .
UMUTeam at SemEval-2023 Task 11: Ensemble Learning applied to Binary Supervised Classifiers with disagreements

This paper describes the participation of the UMUTeam in the Learning With Disagreements (Le-Wi-Di) shared task proposed at Se-mEval 2023, which objective is the development of supervised automatic classifiers that consider, during training, the agreements and disagreements among the annotators of the datasets.Specifically, this edition includes a multilingual dataset.Our proposal is grounded on the development of ensemble learning classifiers that combine the outputs of several Large Language Models.Our proposal ranked position 18 of a total of 30 participants.However, our proposal did not incorporate the information about the disagreements.In contrast, we compare the performance of building several classifiers for each dataset separately with a merged dataset.
Self-Distilled Quantization: Achieving High Compression Rates in Transformer-Based Language Models

We investigate the effects of post-training quantization and quantization-aware training on the generalization of Transformer language models.We present a new method called self-distilled quantization (SDQ) that minimizes accumulative quantization errors and outperforms baselines.We apply SDQ to multilingual models XLM-R Base and InfoXLM Base and demonstrate that both models can be reduced from 32-bit floating point weights to 8-bit integer weights while maintaining a high level of performance on the XGLUE benchmark.Our results also highlight the challenges of quantizing multilingual models, which must generalize to languages they were not fine-tuned on.
Prompt Discriminative Language Models for Domain Adaptation

Prompt tuning offers an efficient approach to domain adaptation for pretrained language models, which predominantly focus on masked language modeling or generative objectives.However, the potential of discriminative language models in biomedical tasks remains underexplored.To bridge this gap, we develop BIODLM, a method tailored for biomedical domain adaptation of discriminative language models that incorporates prompt-based continual pretraining and prompt tuning for downstream tasks.BIODLM aims to maximize the potential of discriminative language models in low-resource scenarios by reformulating these tasks as span-level corruption detection, thereby enhancing performance on domainspecific tasks and improving the efficiency of continual pertaining.In this way, BIODLM provides a data-efficient domain adaptation method for discriminative language models, effectively enhancing performance on discriminative tasks within the biomedical domain.
Towards automatically extracting morphosyntactical error patterns from L1-L2 parallel dependency treebanks

L1-L2 parallel dependency treebanks are UDannotated corpora of learner sentences paired with correction hypotheses.Automatic morphosyntactical annotation has the potential to remove the need for explicit manual error tagging and improve interoperability, but makes it more challenging to locate grammatical errors in the resulting datasets.We therefore propose a novel method for automatically extracting morphosyntactical error patterns and perform a preliminary bilingual evaluation of its first implementation through a similar example retrieval task.The resulting pipeline is also available as a prototype CALL application.
MISMATCH: Fine-grained Evaluation of Machine-generated Text with Mismatch Error Types

Keerthiram Murugesan, Sarathkrishna Swaminathan, Soham Dan, Subhajit Chaudhury, Chulaka Gunasekara, Maxwell Crouse, Diwakar Mahajan, Ibrahim Abdelaziz, Achille Fokoue, Pavan Kapanipathi, Salim Roukos, Alexander Gray. Findings of the Association for Computational Linguistics: ACL 2023. 2023.
Deriving Language Models from Masked Language Models

Masked language models (MLM) do not explicitly define a distribution over language, i.e., they are not language models per se.However, recent work has implicitly treated them as such for the purposes of generation and scoring.This paper studies methods for deriving explicit joint distributions from MLMs, focusing on distributions over two tokens, which makes it possible to calculate exact distributional properties.We find that an approach based on identifying joints whose conditionals are closest to those of the MLM works well and outperforms existing Markov random field-based approaches.We further find that this derived model's conditionals can even occasionally outperform the original MLM's conditionals.
A Diverse Set of Freely Available Linguistic Resources for Turkish

This study presents a diverse set of freely available linguistic resources for Turkish natural language processing, including corpora, pretrained models and education material. Although Turkish is spoken by a sizeable population of over 80 million people, Turkish linguistic resources for natural language processing remain scarce. In this study, we provide corpora to allow practitioners to build their own applications and pretrained models that would assist industry researchers in creating quick prototypes. The provided corpora include named entity recognition datasets of diverse genres, including Wikipedia articles and supplement products customer reviews. In addition, crawling e-commerce and movie reviews websites, we compiled several sentiment analysis datasets of different genres. Our linguistic resources for Turkish also include pretrained spaCy language models. To the best of our knowledge, our models are the first spaCy models trained for the Turkish language. Finally, we provide various types of education material, such as video tutorials and code examples, that can support the interested audience on practicing Turkish NLP. The advantages of our linguistic resources are three-fold: they are freely available, they are first of their kind, and they are easy to use in a broad range of implementations. Along with a thorough description of the resource creation process, we also explain the position of our resources in the Turkish NLP world.
Submission of USTC’s System for the IWSLT 2023 - Offline Speech Translation Track

This paper describes the submissions of the research group USTC-NELSLIP to the 2023 IWSLT Offline Speech Translation competition, which involves translating spoken English into written Chinese.We utilize both cascaded models and end-to-end models for this task.To improve the performance of the cascaded models, we introduce Whisper to reduce errors in the intermediate source language text, achieving a significant improvement in ASR recognition performance.For end-to-end models, we propose Stacked Acoustic-and-Textual Encoding extension (SATE-ex), which feeds the output of the acoustic decoder into the textual decoder for information fusion and to prevent error propagation.Additionally, we improve the performance of the end-to-end system in translating speech by combining the SATE-ex model with the encoder-decoder model through ensembling.