Reviewerly/exHarmony huggingface dataset:
WSPAlign: Word Alignment Pre-training via Large-Scale Weakly Supervised Span Prediction

Most existing word alignment methods rely on manual alignment datasets or parallel corpora, which limits their usefulness.Here, to mitigate the dependence on manual data, we broaden the source of supervision by relaxing the requirement for correct, fully-aligned, and parallel sentences.Specifically, we make noisy, partially aligned, and non-parallel paragraphs.We then use such a large-scale weakly-supervised dataset for word alignment pre-training via span prediction.Extensive experiments with various settings empirically demonstrate that our approach, which is named WSPAlign, is an effective and scalable way to pre-train word aligners without manual data.When fine-tuned on standard benchmarks, WSPAlign has set a new state of the art by improving upon the best supervised baseline by 3.3~6.1 points in F1 and 1.5~6.1 points in AER .Furthermore, WSPAlign also achieves competitive performance compared with the corresponding baselines in few-shot, zero-shot and cross-lingual tests, which demonstrates that WSPAlign is potentially more practical for low-resource languages than existing methods. 1 (1) Data Collection and Annotation (2) Pre-training for word alignment Transformer Encoder Ashikaga Yoshimitsu was the third SEP …
Bridging the Gap between Decision and Logits in Decision-based Knowledge Distillation for Pre-trained Language Models

Conventional knowledge distillation (KD) methods require access to the internal information of teachers, e.g., logits.However, such information may not always be accessible for large pre-trained language models (PLMs).In this work, we focus on decision-based KD for PLMs, where only teacher decisions (i.e., top-1 labels) are accessible.Considering the information gap between logits and decisions, we propose a novel method to estimate logits from the decision distributions.Specifically, decision distributions can be both derived as a function of logits theoretically and estimated with test-time data augmentation empirically.By combining the theoretical and empirical estimations of the decision distributions together, the estimation of logits can be successfully reduced to a simple root-finding problem.Extensive experiments show that our method significantly outperforms strong baselines on both natural language understanding and machine reading comprehension datasets. 1
Differentiable Instruction Optimization for Cross-Task Generalization

Instruction tuning has been attracting much attention to achieve generalization ability across a wide variety of tasks.Although various types of instructions have been manually created for instruction tuning, it is still unclear what kind of instruction is optimal to obtain cross-task generalization ability.This work presents instruction optimization, which optimizes training instructions with respect to generalization ability.Rather than manually tuning instructions, we introduce learnable instructions and optimize them with gradient descent by leveraging bilevel optimization.Experimental results show that the learned instruction enhances the diversity of instructions and improves the generalization ability compared to using only manually created instructions.
HausaNLP at SemEval-2023 Task 12: Leveraging African Low Resource TweetData for Sentiment Analysis

We present the findings of SemEval-2023 Task 12, a shared task on sentiment analysis for low-resource African languages using Twitter dataset.The task featured three subtasks; subtask A is monolingual sentiment classification with 12 tracks which are all monolingual languages, subtask B is multilingual sentiment classification using the tracks in subtask A and subtask C is a zero-shot sentiment classification.We present the results and findings of subtask A, subtask B and subtask C. We also release the code on github.Our goal is to leverage low-resource tweet data using pretrained Afro-xlmr-large, AfriBERTa-Large, Bert-base-arabic-camelbert-da-sentiment (Arabic-camelbert), Multilingual-BERT (mBERT) and BERT models for sentiment analysis of 14 African languages.The datasets for these subtasks consists of a gold standard multi-class labeled Twitter datasets from these languages.Our results demonstrate that Afroxlmr-large model performed better compared to the other models in most of the languages datasets.Similarly, Nigerian languages: Hausa, Igbo, and Yoruba achieved better performance compared to other languages and this can be attributed to the higher volume of data present in the languages.
PEER: Pre-training ELECTRA Extended by Ranking

The BERT model and its variants have made great achievements in many downstream natural language processing tasks.The achievements of these models, however, demand highly expensive pre-training computation cost.To address this pre-training efficiency issue, the ELECTRA model is proposed to use a discriminator to perform replaced token detection (RTD) task, that is, to classify whether each input token is original or replaced by a generator.The RTD task performed by the ELEC-TRA accelerates pre-training so substantially, such that it is very challenging to further improve the pre-training efficiency established by the ELECTRA by using or adding other pre-training tasks, as the recent comprehensive study of Bajaj et al. (2022) summarizes.To further advance this pre-training efficiency frontier, in this paper we propose to extend the RTD task into a task of ranking input tokens according to K different quality levels.Essentially, we generalize the binary classifier in the ELEC-TRA into a K-level ranker to undertake a more precise task with negligible additional computation cost.Our extensive experiments show that our proposed method is able to outperform the state-of-the-art pre-training efficient models including ELECTRA in downstream GLUE tasks given the same computation cost.
SEAG: Structure-Aware Event Causality Generation

Extracting event causality underlies a broad spectrum of natural language processing applications.Cutting-edge methods break this task into Event Detection and Event Causality Identification.Although the pipelined solutions succeed in achieving acceptable results, the inherent nature of separating the task incurs limitations.On the one hand, it suffers from the lack of cross-task dependencies and may cause error propagation.On the other hand, it predicts events and relations separately, undermining the integrity of the event causality graph (ECG).To address such issues, in this paper, we propose an approach for Structure-Aware Event Causality Generation (SEAG).With a graph linearization module, we generate the ECG structure in a way of text2text generation based on a pre-trained language model.To foster the structural representation of the ECG, we introduce the novel Causality Structural Discrimination training paradigm in which we perform structural discriminative training alongside auto-regressive generation enabling the model to distinguish from constructed incorrect ECGs.We conduct experiments on three datasets.The experimental results demonstrate the effectiveness of structural event causality generation and the causality structural discrimination training.
Disentangling Reasoning Capabilities from Language Models with Compositional Reasoning Transformers

This paper presents ReasonFormer, a unified reasoning framework for mirroring the modular and compositional reasoning process of humans in complex decision-making.Inspired by dual-process theory in cognitive science, the representation module (automatic thinking) and reasoning modules (controlled thinking) are decoupled to capture different levels of cognition.Upon the top of the representation module, the pre-trained reasoning modules are modular and professional in specific and fundamental reasoning skills (e.g., logic, simple QA, etc).To mimic the controlled compositional thinking process, different reasoning modules are dynamically activated and composed in both parallel and cascaded manners to control what reasoning skills are activated and how deep the reasoning process will be reached to solve the current problems.The unified reasoning framework solves multiple tasks with a single model, and is trained and inferred in an end-to-end manner.Evaluated on 11 datasets requiring different reasoning skills and complexity, Reason-Former demonstrates substantial performance boosts, revealing the compositional reasoning ability.Few-shot experiments exhibit better generalization ability by learning to compose pre-trained skills for new tasks with limited data, and decoupling the representation module and the reasoning modules.Further analysis shows the modularity of reasoning modules as different tasks activate distinct reasoning skills at different reasoning depths.
An Extensible Plug-and-Play Method for Multi-Aspect Controllable Text Generation

Recently, multi-aspect controllable text generation that controls the generated text in multiple aspects (e.g., sentiment, topic, and keywords) has attracted increasing attention.Although methods based on parameter efficient tuning like prefix-tuning could achieve multi-aspect controlling in a plug-and-play way, the mutual interference of multiple prefixes leads to significant degeneration of constraints and limits their extensibility to training-time unseen aspect combinations.In this work, we provide a theoretical lower bound for the interference and empirically found that the interference grows with the number of layers where prefixes are inserted.Based on these analyses, we propose using trainable gates to normalize the intervention of prefixes to restrain the growing interference.As a result, controlling training-time unseen combinations of aspects can be realized by simply concatenating corresponding plugins such that new constraints can be extended at a lower cost.In addition, we propose a unified way to process both categorical and free-form constraints.Experiments on text generation and machine translation demonstrate the superiority of our approach over baselines on constraint accuracy, text quality, and extensibility. 1
Leveraging Natural Language Processing and Clinical Notes for Dementia Detection

Early detection and automated classification of dementia has recently gained considerable attention using neuroimaging data and spontaneous speech.In this paper, we explore the problem of dementia detection with in-hospital clinical notes.We collected 954 patients' clinical notes from a local hospital in Melbourne and assign dementia/non-dementia labels to those patients based on clinical assessment and telephone interview.Given the labeled dementia data sets, we fine tune a ClinicalBioBERT using filtered clinical notes and conducted experiments on both binary and three class dementia classification.Our experiment results show that the fine tuned ClinicalBioBERT achieved satisfied performance on binary classification but performed poorly on three class dementia classification.We explore the difficulties we encountered applying ClinicalBioBERT to hospital text.Further analysis suggests that more human prior knowledge should be considered. ModelsAccuracy Precision Recall F1 Keyword-based (binary) 0.453 (0.021) 0.469 (0.023) 0.482 (0.025) 0.475 (0.021) Keyword-based (coarse) 0.398 (0.051) 0.382 (0.052) 0.393 (0.043) 0.387 (0.052) ClinicalBioBERT (binary) 0.810 (0.041) 0.832 (0.051) 0.801 (0.052) 0.814 (0.050) ClinicalBioBERT (coarse) 0.458 (0.045) 0.449 (0.043) 0.406 (0.046) 0.381 (0.045)
LaTeX2Solver: a Hierarchical Semantic Parsing of LaTeX Document into Code for an Assistive Optimization Modeling Application

We demonstrate an interactive system to help operations research (OR) practitioners convert the mathematical formulation of optimization problems from LaTeX document format into the solver modeling language.In practice, a manual translation is cumbersome and timeconsuming.Moreover, it requires an in-depth understanding of the problem description and a technical expertise to produce the modeling code.Thus, our proposed system LATEX-SOLVER helps partially automate this conversion and help the users build optimization models more efficiently.In this paper, we describe its interface and the components of the hierarchical parsing system.A video demo walkthrough is available online.
LAIT: Efficient Multi-Segment Encoding in Transformers with Layer-Adjustable Interaction

Transformer encoders contextualize token representations by attending to all other tokens at each layer, leading to quadratic increase in compute effort with the input length.In practice, however, the input text of many NLP tasks can be seen as a sequence of related segments (e.g., the sequence of sentences within a passage, or the hypothesis and premise in NLI).While attending across these segments is highly beneficial for many tasks, we hypothesize that this interaction can be delayed until later encoding stages.To this end, we introduce Layer-Adjustable Interactions in Transformers (LAIT).Within LAIT, segmented inputs are first encoded independently, and then jointly.This partial two-tower architecture bridges the gap between a Dual Encoder's ability to precompute representations for segments and a fully self-attentive Transformer's capacity to model cross-segment attention.The LAIT framework effectively leverages existing pretrained Transformers and converts them into the hybrid of the two aforementioned architectures, allowing for easy and intuitive control over the performance-efficiency tradeoff.Experimenting on a wide range of NLP tasks, we find LAIT able to reduce 30-50% of the attention FLOPs on many tasks, while preserving high accuracy; in some practical settings, LAIT could reduce actual latency by orders of magnitude.
Synthesize, Prompt and Transfer: Zero-shot Conversational Question Generation with Pre-trained Language Model

Conversational question generation aims to generate questions that depend on both context and conversation history.Conventional works utilizing deep learning have shown promising results, but heavily rely on the availability of large-scale annotated conversations.In this paper, we introduce a more realistic and less explored setting, Zero-shot Conversational Question Generation (ZeroCQG), which requires no human-labeled conversations for training.To solve ZeroCQG, we propose a multi-stage knowledge transfer framework, Synthesize, Prompt and trAnsfer with pRe-Trained lAnguage model (SPARTA) to effectively leverage knowledge from single-turn question generation instances.To validate the zero-shot performance of SPARTA, we conduct extensive experiments on three conversational datasets: CoQA, QuAC, and DoQA by transferring knowledge from three single-turn datasets: MS MARCO, NewsQA, and SQuAD.The experimental results demonstrate the superior performance of our method.Specifically, SPARTA has achieved 14.81 BLEU-4 (88.2% absolute improvement compared to T5) in CoQA with knowledge transferred from SQuAD.
APTSumm at BioLaySumm Task 1: Biomedical Breakdown, Improving Readability by Relevancy Based Selection

In this paper we tackle a lay summarization task which aims to produce lay-summary of biomedical articles.BioLaySumm in the BioNLP Workshop at ACL 2023 (Goldsack et al., 2023), has presented us with this lay summarization task for biomedical articles.Our proposed models provide a three-step abstractive approach for summarizing biomedical articles.Our methodology involves breaking down the original document into distinct sections, generating candidate summaries for each subsection, then finally re-ranking and selecting the topperforming paragraph for each section.We run ablation studies to establish that each step in our pipeline is critical for improvement in the quality of lay summary.This model achieved the second-highest rank in terms of readability scores (Luo et al., 2022).Our work distinguishes itself from previous studies by not only considering the content of the paper but also its structure, resulting in more coherent and comprehensible lay summaries.We hope that our model for generating lay summaries of biomedical articles will be a useful resource for individuals across various domains, including academia, industry, and healthcare, who require rapid comprehension of key scientific research.
CONE: An Efficient COarse-to-fiNE Alignment Framework for Long Video Temporal Grounding

This paper tackles an emerging and challenging problem of long video temporal grounding (VTG) that localizes video moments related to a natural language (NL) query.Compared with short videos, long videos are also highlydemanded but less explored, which brings new challenges in higher inference computation cost and weaker multi-modal alignment.To address these challenges, we propose CONE, an efficient COarse-to-fiNE alignment framework.CONE is a plug-and-play framework on top of existing VTG models to handle long videos through a sliding window mechanism.Specifically, CONE (1) introduces a query-guided window selection strategy to speed up inference, and (2) proposes a coarse-to-fine mechanism via a novel incorporation of contrastive learning to enhance multi-modal alignment for long videos.Extensive experiments on two large-scale long VTG benchmarks consistently show both substantial performance gains (e.g., 3.13% +119% ---→6.87% on MAD) and state-of-theart results.Analyses also reveal higher efficiency as the query-guided window selection mechanism accelerates inference time by 2x on Ego4D-NLQ and 15x on MAD while keeping SOTA results.Codes have been released at https://github.com/houzhijian/CONE.
DARE: Towards Robust Text Explanations in Biomedical and Healthcare Applications

Along with the successful deployment of deep neural networks in several application domains, the need to unravel the black-box nature of these networks has seen a significant increase recently.Several methods have been introduced to provide insight into the inference process of deep neural networks.However, most of these explainability methods have been shown to be brittle in the face of adversarial perturbations of their inputs in the image and generic textual domain.In this work we show that this phenomenon extends to specific and important high stakes domains like biomedical datasets.In particular, we observe that the robustness of explanations should be characterized in terms of the accuracy of the explanation in linking a model's inputs and its decisions -faithfulness -and its relevance from the perspective of domain experts -plausibility.This is crucial to prevent explanations that are inaccurate but still look convincing in the context of the domain at hand.To this end, we show how to adapt current attribution robustness estimation methods to a given domain, so as to take into account domain-specific plausibility.This results in our DOMAINADAPTIVEARESTIMATOR (DARE) attribution robustness estimator, allowing us to properly characterize the domain-specific robustness of faithful explanations.Next, we provide two methods, adversarial training and FAR training, to mitigate the brittleness characterized by DARE, allowing us to train networks that display robust attributions.Finally, we empirically validate our methods with extensive experiments on three established biomedical benchmarks.VANILLA ADVERSARIAL FAR-IG Original 'took zoloft for 5 months.no side effects except sexual dysfunction.i didn't feel much better or happier and it made me feel really drowsy.' F (s, l ="4.0") = 1.0 'took zoloft for 5 months.no side effects except sexual dysfunction.i didn't feel much better or happier and it made me feel really drowsy.' F (s, l ="4.0") = 1.0 'took zoloft for 5 months.no side effects except sexual dysfunction.i didn't feel much better or happier and it made me feel really drowsy.' F (s, l ="4.0") = 1.0 Adversarial 'took zoloft for 5 months.no side effects except sexual dysfunction.i didn't feel much better or anything and it made me feel really drowsy.' F (s adv , l ="4.0") = 1.0 Cos. = -0.32M edST S = 0.99 'took zoloft for 5 months.no side effects except sexual dysfunction.i didn't feel much better or stronger and it made me feel really drowsy.'F (s adv , l ="4.0") = 1.0 Cos. = -0.05M edST S = 0.96 'took zoloft for 5 months.no side effects except nerve dysfunction.i didn't feel much better or happier and it made me feel really drowsy.' F (s adv , l ="4.0") = 1.0 Cos. = 0.79 M edST S = 0.93
Unsupervised Paraphrasing of Multiword Expressions

We propose an unsupervised approach to paraphrasing multiword expressions (MWEs) in context.Our model employs only monolingual corpus data and pre-trained language models (without fine-tuning), and does not make use of any external resources such as dictionaries.We evaluate our method on the SemEval 2022 idiomatic semantic text similarity task, and show that it outperforms all unsupervised systems and rivals supervised systems.
Continual Knowledge Distillation for Neural Machine Translation

While many parallel corpora are not publicly accessible for data copyright, data privacy and competitive differentiation reasons, trained translation models are increasingly available on open platforms.In this work, we propose a method called continual knowledge distillation to take advantage of existing translation models to improve one model of interest.The basic idea is to sequentially transfer knowledge from each trained model to the distilled model.Extensive experiments on Chinese-English and German-English datasets show that our method achieves significant and consistent improvements over strong baselines under both homogeneous and heterogeneous trained model settings and is robust to malicious models. 1
HausaNLP at SemEval-2023 Task 10: Transfer Learning, Synthetic Data and Side-information for Multi-level Sexism Classification

We present the findings of our participation in the SemEval-2023 Task 10: Explainable Detection of Online Sexism (EDOS) task, a shared task on offensive language (sexism) detection on English Gab and Reddit dataset.We investigated the effects of transferring two language models: XLM-T (sentiment classification) and HateBERT (same domain -Reddit) for multilevel classification into Sexist or not Sexist, and other subsequent sub-classifications of the sexist data.We also use synthetic classification of unlabelled dataset and intermediary class information to maximize the performance of our models.We submitted a system in Task A, and it ranked 49 th with F1-score of 0.82.This result showed to be competitive as it only under-performed the best system by 0.052% F1-score.
Towards Open-World Product Attribute Mining: A Lightly-Supervised Approach

We present a new task setting for attribute mining on e-commerce products, serving as a practical solution to extract open-world attributes without extensive human intervention.Our supervision comes from a high-quality seed attribute set bootstrapped from existing resources, and we aim to expand the attribute vocabulary of existing seed types, and also to discover any new attribute types automatically.A new dataset is created to support our setting, and our approach Amacer is proposed specifically to tackle the limited supervision.Especially, given that no direct supervision is available for those unseen new attributes, our novel formulation exploits selfsupervised heuristic and unsupervised latent attributes, which attains implicit semantic signals as additional supervision by leveraging product context.Experiments suggest that our approach surpasses various baselines by 12 F1, expanding attributes of existing types significantly by up to 12 times, and discovering values from 39% new types.Our data and code can be found at https://github.com/lxucs/woam.