Reviewerly/exHarmony huggingface dataset:
LTRC @ Causal News Corpus 2022: Extracting and Identifying Causal Elements using Adapters

Causality detection and identification is centered on identifying semantic and cognitive connections in a sentence.In this paper, we describe the effort of team LTRC for Causal News Corpus -Event Causality Shared Task 2022 at the 5th Workshop on Challenges and Applications of Automated Extraction of Sociopolitical Events from Text (CASE 2022) (Tan et al., 2022a).The shared task consisted of two subtasks: 1) identifying if a sentence contains a causality relation, and 2) identifying spans of text that correspond to cause, effect and signals.We fine-tuned transformer-based models with adapters for both subtasks.Our best-performing models obtained a binary F1 score of 0.853 on held-out data for subtask 1 and a macro F1 score of 0.032 on held-out data for subtask 2. Our approach is ranked third in subtask 1 and fourth in subtask 2. The paper describes our experiments, solutions, and analysis in detail.
ConvTrans: Transforming Web Search Sessions for Conversational Dense Retrieval

Conversational search provides users with natural and convenient new search experience.Recently, conversational dense retrieval has shown to be a promising technique to realize conversational search.However, as conversational search systems have not been widely deployed, it is hard to get large-scale real conversational search sessions and relevance labels to support the training of conversational dense retrieval.To tackle this data scarcity problem, previous methods focus on developing better few-shot learning approaches or generating pseudo relevance labels, but the data they use for training still heavily rely on manual generation.In this paper, we present ConvTrans, a data augmentation method that can automatically transform easily-accessible web search sessions into conversational search sessions to fundamentally alleviate the data scarcity problem for conversational dense retrieval.ConvTrans eliminates the gaps between these two types of sessions in terms of session quality and query form to achieve effective session transformation.Extensive evaluations on two widely used conversational search benchmarks, i.e., , demonstrate that the same model trained on the data generated by Con-vTrans can achieve comparable retrieval performance as it trained on high-quality but expensive artificial conversational search data.
Joint Coreference Resolution for Zeros and non-Zeros in Arabic

Most existing proposals about anaphoric zero pronoun (AZP) resolution regard full mention coreference and AZP resolution as two independent tasks, even though the two tasks are clearly related.The main issues that need tackling to develop a joint model for zero and non-zero mentions are the difference between the two types of arguments (zero pronouns, being null, provide no nominal information) and the lack of annotated datasets of a suitable size in which both types of arguments are annotated for languages other than Chinese and Japanese.In this paper, we introduce two architectures for jointly resolving AZPs and non-AZPs, and evaluate them on Arabic, a language for which, as far as we know, there has been no prior work on joint resolution.Doing this also required creating a new version of the Arabic subset of the standard coreference resolution dataset used for the CoNLL-2012 shared task (Pradhan et al., 2012) in which both zeros and non-zeros are included in a single dataset.
Word Order Matters When You Increase Masking

Word order, an essential property of natural languages, is injected in Transformer-based neural language models using position encoding.However, recent experiments have shown that explicit position encoding is not always useful, since some models without such feature managed to achieve state-of-the art performance on some tasks.To understand better this phenomenon, we examine the effect of removing position encodings on the pre-training objective itself (i.e., masked language modelling), to test whether models can reconstruct position information from co-occurrences alone.We do so by controlling the amount of masked tokens in the input sentence, as a proxy to affect the importance of position information for the task.We find that the necessity of position information increases with the amount of masking, and that masked language models without position encodings are not able to reconstruct this information on the task.These findings point towards a direct relationship between the amount of masking and the ability of Transformers to capture order-sensitive aspects of language using position encoding.
Semantic Framework based Query Generation for Temporal Question Answering over Knowledge Graphs

Answering factual questions with temporal intent over knowledge graphs (temporal KGQA) attracts rising attention in recent years.In the generation of temporal queries, existing KGQA methods ignore the fact that some intrinsic connections between events can make them temporally related, which may limit their capability.We systematically analyze the possible interpretation of temporal constraints and conclude the interpretation structures as the Semantic Framework of Temporal Constraints, SF-TCons.Based on the semantic framework, we propose a temporal question answering method, SF-TQA, which generates query graphs by exploring the relevant facts of mentioned entities, where the exploring process is restricted by SF-TCons.Our evaluations show that SF-TQA significantly outperforms existing methods on two benchmarks over different knowledge graphs.
Learning to Adapt to Low-Resource Paraphrase Generation

Paraphrase generation is a longstanding NLP task and achieves great success with the aid of large corpora.However, transferring a paraphrasing model to another domain encounters the problem of domain shifting especially when the data is sparse.At the same time, widely using large pre-trained language models (PLMs) faces the overfitting problem when training on scarce labeled data.To mitigate these two issues, we propose, LAPA, an effective adapter for PLMs optimized by meta-learning.LAPA has three-stage training on three types of related resources to solve this problem: 1. pre-training PLMs on unsupervised corpora, 2. inserting an adapter layer and meta-training on source domain labeled data, and 3. fine-tuning adapters on a small amount of target domain labeled data.This method enables paraphrase generation models to learn basic language knowledge first, then learn the paraphrasing task itself later, and finally adapt to the target task.Our experimental results demonstrate that LAPA achieves state-of-the-art in supervised, unsupervised, and low-resource settings on three benchmark datasets.With only 2% of trainable parameters and 1% labeled data of the target task, our approach can achieve a competitive performance with previous work.
Global Span Selection for Named Entity Recognition

Named Entity Recognition (NER) is an important task in Natural Language Processing with applications in many domains.In this paper, we describe a novel approach to named entity recognition, in which we output a set of spans (i.e., segmentations) by maximizing a global score.During training, we optimize our model by maximizing the probability of the gold segmentation.During inference, we use dynamic programming to select the best segmentation under a linear time complexity.We prove that our approach outperforms CRF and semi-CRF models for Named Entity Recognition.We make our code publicly available at https://github.com/urchade/global-span-selection.
Data Cartography for Low-Resource Neural Machine Translation

While collecting or generating more parallel data is necessary to improve machine translation (MT) in low-resource settings, we lack an understanding of how the limited amounts of existing data are actually used to help guide the collection of further resources.In this paper, we apply data cartography techniques (Swayamdipta et al., 2020) to characterize the contribution of training samples in two low-resource MT tasks (Swahili-English and Turkish-English) throughout the training of standard neural MT models.Our empirical study shows that, unlike in prior work for classification tasks, most samples contribute to model training in low-resource MT, albeit not uniformly throughout the training process.Furthermore, uni-dimensional characterizations of samples -e.g., based on dual cross-entropy or word frequency -do not suffice to characterize to what degree they are hard or easy to learn.Taken together, our results suggest that data augmentation strategies for low-resource MT would benefit from model-in-the-loop strategies to maximize improvements.
On the Transformation of Latent Space in Fine-Tuned NLP Models

We study the evolution of latent space in finetuned NLP models.Different from the commonly used probing-framework, we opt for an unsupervised method to analyze representations.More specifically, we discover latent concepts in the representational space using hierarchical clustering.We then use an alignment function to gauge the similarity between the latent space of a pre-trained model and its finetuned version.We use traditional linguistic concepts to facilitate our understanding and also study how the model space transforms towards task-specific information.We perform a thorough analysis, comparing pre-trained and finetuned models across three models and three downstream tasks.The notable findings of our work are: i) the latent space of the higher layers evolve towards task-specific concepts, ii) whereas the lower layers retain generic concepts acquired in the pre-trained model, iii) we discovered that some concepts in the higher layers acquire polarity towards the output class, and iv) that these concepts can be used for generating adversarial triggers.
Universal Evasion Attacks on Summarization Scoring

The automatic scoring of summaries is important as it guides the development of summarizers.Scoring is also complex, as it involves multiple aspects such as fluency, grammar, and even textual entailment with the source text.However, summary scoring has not been considered a machine learning task to study its accuracy and robustness.In this study, we place automatic scoring in the context of regression machine learning tasks and perform evasion attacks to explore its robustness.Attack systems predict a non-summary string from each input, and these non-summary strings achieve competitive scores with good summarizers on the most popular metrics: ROUGE, METEOR, and BERTScore.Attack systems also "outperform" state-of-the-art summarization methods on ROUGE-1 and ROUGE-L, and score the second-highest on METEOR.Furthermore, a BERTScore backdoor is observed: a simple trigger can score higher than any automatic summarization method.The evasion attacks in this work indicate the low robustness of current scoring systems at the system level.We hope that our highlighting of these proposed attacks will facilitate the development of summary scores.
Nearest Neighbor Language Models for Stylistic Controllable Generation

Recent language modeling performance has been greatly improved by the use of external memory.This memory encodes the context so that similar contexts can be recalled during decoding.This similarity depends on how the model learns to encode context, which can be altered to include other attributes, such as style.We construct and evaluate an architecture for this purpose, using corpora annotated for politeness, formality, and toxicity.Through extensive experiments and human evaluation we demonstrate the potential of our method to generate text while controlling style.We find that style-specific datastores improve generation performance, though results vary greatly across styles, and the effect of pretraining data and specific styles should be explored in future work.
T5QL: Taming language models for SQL generation

Automatic SQL generation has been an active research area, aiming at streamlining the access to databases by writing natural language with the given intent instead of writing SQL. Current SOTA methods for semantic parsing depend on LLMs to achieve high predictive accuracy on benchmark datasets. This reduces their applicability, since LLMs requires expensive GPUs. Furthermore, SOTA methods are ungrounded and thus not guaranteed to always generate valid SQL. Here we propose T5QL, a new SQL generation method that improves the performance in benchmark datasets when using smaller LMs, namely T5-Base, by 13pp when compared against SOTA methods. Additionally, T5QL is guaranteed to always output valid SQL using a context-free grammar to constrain SQL generation. Finally, we show that dividing semantic parsing in two tasks, candidate SQLs generation and candidate re-ranking, is a promising research avenue that can reduce the need for large LMs.
Multi-Tenant Optimization For Few-Shot Task-Oriented FAQ Retrieval

Business-specific Frequently Asked Questions (FAQ) retrieval in task-oriented dialog systems poses unique challenges vis à vis community based FAQs. Each FAQ question represents an intent which is usually an umbrella term for many related user queries. We evaluate performance for such Business FAQs both with standard FAQ retrieval techniques using query-Question (q-Q) similarity and few-shot intent detection techniques. Implementing a real-world solution for FAQ retrieval in order to support multiple tenants (FAQ sets) entails optimizing speed, accuracy and cost. We propose a novel approach to scale multi-tenant FAQ applications in real-world context by contrastive fine-tuning of the last layer in sentence Bi-Encoders along with tenant-specific weight switching.
System 1 + System 2 = Better World: Neural-Symbolic Chain of Logic Reasoning

Logical reasoning is a challenge for many current NLP neural network models since it requires more than the ability of learning informative representations from data.Inspired by the Dual Process Theory in cognitive sciencewhich proposes that human cognition process involves two stages: an intuitive, unconscious and fast process relying on perception called System 1, and a logical, conscious and slow process performing complex reasoning called System 2 -we leverage neural logic reasoning (System 2) on top of the representation learning models (System 1), which conducts explicit neural-based differentiable logical reasoning on top of the representations learned by the base neural models.Based on experiments on the commonsense knowledge graph completion task, we show that the two-system architecture always improves from its System 1 model alone.Experiments also show that both the rule-driven logical regularizer and the datadriven value regularizer are important and the performance improvement is marginal without the two regularizers, which indicates that learning from both logical prior and training data is important for reasoning tasks.
MatRank: Text Re-ranking by Latent Preference Matrix

Text ranking plays a key role in providing content that best answers user queries.It is usually divided into two sub-tasks to perform efficient information retrieval given a query: text retrieval and text re-ranking.Recent research on pretrained language models (PLM) has demonstrated efficiency and gain on both sub-tasks.However, while existing methods have benefited from pre-trained language models and achieved high recall rates on passage retrieval, the ranking performance still demands further improvement.In this paper, we propose Ma-tRank, which learns to re-rank the text retrieved for a given query by learning to predict the most relevant passage based on a latent preference matrix.Specifically, MatRank uses a PLM to generate an asymmetric latent matrix of relative preference scores between all pairs of retrieved passages.Then, the latent matrix is aggregated row-wise and column-wise to obtain global preferences and predictions of the most relevant passage in two of these directions, respectively.We conduct extensive experiments on MS MACRO, WikiAQ, and Se-mEval datasets.Experimental results show that MatRank has achieved new state-of-the-art results on these datasets, outperforming all prior methods on ranking performance metrics.
Condition-Treatment Relation Extraction on Disease-related Social Media Data

Social media has become a popular platform where people share information about personal healthcare conditions, diagnostic histories, and medical plans.Analyzing posts on social media depicting such realistic information can help improve quality and clinical decision-making; however, the lack of structured resources in this genre limits us to build robust NLP models for meaningful analysis.This paper presents a new corpus annotating relations among many types of conditions, treatments, and their attributes illustrated in social media posts by patients and caregivers.For experiments, a transformer encoder is pretrained on 1M raw posts and used to train several document-level relation extraction models using our corpus.Our best-performing model achieves the F1 scores of 70.9 and 51.7 for Entity Recognition and Relation Extraction, respectively.These results are encouraging as it is the first neural model extracting complex relations of this kind on social media data.
Break it Down into BTS: Basic, Tiniest Subword Units for Korean

We introduce Basic, Tiniest Subword (BTS) units for the Korean language, which are inspired by the invention principle of Hangeul, the Korean writing system.Instead of relying on 51 Korean consonant and vowel letters, we form the letters from BTS units by adding strokes or combining them.To examine the impact of BTS units on Korean language processing, we develop a novel BTSbased word embedding framework that is readily applicable to various models.Our experiments reveal that BTS units significantly improve the performance of Korean word embedding on all intrinsic and extrinsic tasks in our evaluation.In particular, BTS-based word embedding outperforms the state-of-theart Korean word embedding by 11.8% in word analogy.We further investigate the unique advantages provided by BTS units through indepth analysis.
Named Entity Recognition as Structured Span Prediction

Named Entity Recognition (NER) is an important task in Natural Language Processing with applications in many domains.While the dominant paradigm of NER is sequence labelling, span-based approaches have become very popular in recent times but are less well understood.In this work, we study different aspects of span-based NER, namely the span representation, learning strategy, and decoding algorithms to avoid span overlap.We also propose an exact algorithm that efficiently finds the set of non-overlapping spans that maximizes a global score, given a list of candidate spans.We performed our study on three benchmark NER datasets from different domains.We make our code publicly available at https://github.com/urchade/span-structured-prediction.
Joint Multilingual Knowledge Graph Completion and Alignment

Knowledge graph (KG) alignment and completion are usually treated as two independent tasks.While recent work has leveraged entity and relation alignments from multiple KGs, such as alignments between multilingual KGs with common entities and relations, a deeper understanding of the ways in which multilingual KG completion (MKGC) can aid the creation of multilingual KG alignments (MKGA) is still limited.Motivated by the observation that structural inconsistencies -the main challenge for MKGA models -can be mitigated through KG completion methods, we propose a novel model for jointly completing and aligning knowledge graphs.The proposed model combines two components that jointly accomplish KG completion and alignment.These two components employ relation-aware graph neural networks that we propose to encode multihop neighborhood structures into entity and relation representations.Moreover, we also propose (i) a structural inconsistency reduction mechanism to incorporate information from the completion into the alignment component, and (ii) an alignment seed enlargement and triple transferring mechanism to enlarge alignment seeds and transfer triples during KGs alignment.Extensive experiments on a public multilingual benchmark show that our proposed model outperforms existing competitive baselines, obtaining new state-of-the-art results on both MKGC and MKGA tasks.
Lexical Entailment with Hierarchy Representations by Deep Metric Learning

In this paper, we introduce a novel method for lexical entailment tasks, which detects a hyponym-hypernym relation among words.Existing lexical entailment studies are lacking in generalization performance, as they cannot be applied to words that are not included in the training dataset.Moreover, existing work evaluates the performance by using the dataset that contains words used for training.This study proposes a method that learns a mapping from word embeddings to the hierarchical embeddings in order to predict the hypernymy relations of any input words.To validate the generalization performance, we conduct experiments using a train dataset that does not overlap with the evaluation dataset.As a result, our method achieved state-of-the-art performance and showed robustness for unknown words.
Assessing the Linguistic Knowledge in Arabic Pre-trained Language Models Using Minimal Pairs

Despite the noticeable progress that we recently witnessed in Arabic pre-trained language models (PLMs), the linguistic knowledge captured by these models remains unclear.In this paper, we conducted a study to evaluate available Arabic PLMs in terms of their linguistic knowledge.BERT-based language models (LMs) are evaluated using Minimum Pairs (MP), where each pair represents a grammatical sentence and its contradictory counterpart.MPs isolate specific linguistic knowledge to test the model's sensitivity in understanding a specific linguistic phenomenon.We cover nine major Arabic phenomena from: Verbal sentences, Nominal sentences, Adjective Modification, and Idafa construction.The experiments compared the results of fifteen Arabic BERT-based PLMs.Overall, among all tested models, CAMeL-CA and GigaBERT outperformed the other PLMs by achieving the highest overall accuracy.
Cross-domain Generalization for AMR Parsing

Meaning Representation (AMR) parsing aims to predict an AMR graph from textual input.Recently, there has been notable growth in AMR parsing performance.However, most existing work focuses on improving the performance in the specific domain, ignoring the potential domain dependence of AMR parsing systems.To address this, we extensively evaluate five representative AMR parsers on five domains and analyze challenges to cross-domain AMR parsing.We observe that challenges to cross-domain AMR parsing mainly arise from the distribution shift of words and AMR concepts.Based on our observation, we investigate two approaches to reduce the domain distribution divergence of text and AMR features, respectively.Experimental results on two out-of-domain test sets show the superiority of our method.