Reviewerly/exHarmony huggingface dataset:
Compositional Generalization without Trees using Multiset Tagging and Latent Permutations

Seq2seq models have been shown to struggle with compositional generalization in semantic parsing, i.e. generalizing to unseen compositions of phenomena that the model handles correctly in isolation.We phrase semantic parsing as a two-step process: we first tag each input token with a multiset of output tokens.Then we arrange the tokens into an output sequence using a new way of parameterizing and predicting permutations.We formulate predicting a permutation as solving a regularized linear program and we backpropagate through the solver.In contrast to prior work, our approach does not place a priori restrictions on possible permutations, making it very expressive.Our model outperforms pretrained seq2seq models and prior work on realistic semantic parsing tasks that require generalization to longer examples.We also outperform non-tree-based models on structural generalization on the COGS benchmark.For the first time, we show that a model without an inductive bias provided by trees achieves high accuracy on generalization to deeper recursion depth. 1
KwikBucks: Correlation Clustering with Cheap-Weak and Expensive-Strong Signals

For text clustering, there is often a dilemma: one can either first embed each examples independently and then compute pair-wise similarities based on the embeddings, or use a crossattention model that takes a pair of examples as input and produces a similarity.The former is more scalable but the similarities often have lower quality, whereas the latter does not scale well but produces higher quality similarities.We address this dilemma by developing a clustering algorithm that leverages the best of both worlds: the scalability of former and the quality of the latter.We formulate the problem of text clustering with embeddingbased and cross-attention models as a novel version of the Budgeted Correlation Clustering problem (BCC) where along with a limited number of queries to an expensive oracle (a cross-attention model in our case), we have unlimited access to a cheaper but less accurate second oracle (embedding similarities in our case).We develop a theoretically motivated algorithm that leverages the cheap oracle to judiciously query the strong oracle while maintaining high clustering quality.We empirically demonstrate gains in query minimization and clustering metrics on a variety of datasets with diverse strong and cheap oracles.
Towards Adaptable and Interactive Image Captioning with Data Augmentation and Episodic Memory

Interactive machine learning (IML) is a beneficial learning paradigm in cases of limited data availability, as human feedback is incrementally integrated into the training process.In this paper, we present an IML pipeline for image captioning which allows us to incrementally adapt a pre-trained image captioning model to a new data distribution based on user input.In order to incorporate user input into the model, we explore the use of a combination of simple data augmentation methods to obtain larger data batches for each newly annotated data instance and implement continual learning methods to prevent catastrophic forgetting from repeated updates.For our experiments, we split a domain-specific image captioning dataset, namely VizWiz, into non-overlapping parts to simulate an incremental input flow for continually adapting the model to new data.We find that, while data augmentation worsens results, even when relatively small amounts of data are available, episodic memory is an effective strategy to retain knowledge from previously seen clusters.
Corpus Complexity Matters in Pretraining Language Models

It is well known that filtering low-quality data before pretraining language models or selecting suitable data from domains similar to downstream task datasets generally leads to improved downstream performance.However, the extent to which the quality of a corpus, in particular its complexity, affects its downstream performance remains less explored.In this work, we address the problem of creating a suitable pretraining corpus given a fixed corpus budget.Using metrics of text complexity we propose a simple yet effective approach for constructing a corpus with rich lexical variation.Our extensive set of empirical analyses reveal that such a diverse and complex corpus yields significant improvements over baselines consisting of less diverse and less complex corpora when evaluated in the context of general language understanding tasks.
Building blocks for complex tasks: Robust generative event extraction for radiology reports under domain shifts

This paper explores methods for extracting information from radiology reports that generalize across exam modalities to reduce requirements for annotated data.We demonstrate that multi-pass T5-based text-to-text generative models exhibit better generalization across exam modalities compared to approaches that employ BERT-based task-specific classification layers.We then develop methods that reduce the inference cost of the model, making largescale corpus processing more feasible for clinical applications.Specifically, we introduce a generative technique that decomposes complex tasks into smaller subtask blocks, which improves a single-pass model when combined with multitask training.In addition, we leverage target-domain contexts during inference to enhance domain adaptation, enabling use of smaller models.Analyses offer insights into the benefits of different cost reduction strategies.
Expand, Rerank, and Retrieve: Query Reranking for Open-Domain Question Answering

We propose EAR, a query Expansion And Reranking approach for improving passage retrieval, with the application to open-domain question answering.EAR first applies a query expansion model to generate a diverse set of queries, and then uses a query reranker to select the ones that could lead to better retrieval results.Motivated by the observation that the best query expansion often is not picked by greedy decoding, EAR trains its reranker to predict the rank orders of the gold passages when issuing the expanded queries to a given retriever.By connecting better the query expansion model and retriever, EAR significantly enhances a traditional sparse retrieval method, BM25.Empirically, EAR improves top-5/20 accuracy by 3-8 and 5-10 points in in-domain and out-of-domain settings, respectively, when compared to a vanilla query expansion model, GAR, and a dense retrieval model, DPR. 1
UM6P at SemEval-2023 Task 12: Out-Of-Distribution Generalization Method for African Languages Sentiment Analysis

This paper presents our submitted system to AfriSenti SemEval-2023 Task 12: Sentiment Analysis for African Languages.The AfriSenti consists of three different tasks, covering monolingual, multilingual, and zero-shot sentiment analysis scenarios for African languages.To improve model generalization, we have explored the following steps: 1) further pretraining of the AfroXLM Pre-trained Language Model (PLM), 2) combining AfroXLM and MARBERT PLMs using a residual layer, and 3) studying the impact of metric learning and two out-of-distribution generalization training objectives.The overall evaluation results show that our system has achieved promising results on several sub-tasks of Task A. For Tasks B and C, our system is ranked among the top six participating systems.
“A Little is Enough”: Few-Shot Quality Estimation based Corpus Filtering improves Machine Translation

Quality Estimation (QE) is the task of evaluating the quality of a translation when reference translation is not available. The goal of QE aligns with the task of corpus filtering, where we assign the quality score to the sentence pairs present in the pseudo-parallel corpus. We propose a Quality Estimation based Filtering approach to extract high-quality parallel data from the pseudo-parallel corpus. To the best of our knowledge, this is a novel adaptation of QE framework to extracting quality parallel corpus from the pseudo-parallel corpus.. By training with this filtered corpus, we observe an improvement in the Machine Translation (MT) system’s performance by up to 1.8 BLEU points, for English-Marathi, Chinese-English, and Hindi-Bengali language pairs, over the baseline model. The baseline model is the one that is trained on the whole pseudo-parallel corpus. Our Few-shot QE model transfer learned from the English-Marathi QE model and fine-tuned on only 500 Hindi-Bengali training instances, shows an improvement of up to 0.6 BLEU points for Hindi-Bengali language pair, compared to the baseline model. This demonstrates the promise of transfer learning in the setting under discussion. QE systems typically require in the order of (7K-25K) of training data. Our Hindi-Bengali QE is trained on only 500 instances of training that is 1/40th of the normal requirement and achieves comparable performance. All the scripts and datasets utilized in this study will be publicly available.
NCUEE-NLP at SemEval-2023 Task 7: Ensemble Biomedical LinkBERT Transformers in Multi-evidence Natural Language Inference for Clinical Trial Data

This study describes the model design of the NCUEE-NLP system for the SemEval-2023 NLI4CT task that focuses on multi-evidence natural language inference for clinical trial data. We use the LinkBERT transformer in the biomedical domain (denoted as BioLinkBERT) as our main system architecture. First, a set of sentences in clinical trial reports is extracted as evidence for premise-statement inference. This identified evidence is then used to determine the inference relation (i.e., entailment or contradiction). Finally, a soft voting ensemble mechanism is applied to enhance the system performance. For Subtask 1 on textual entailment, our best submission had an F1-score of 0.7091, ranking sixth among all 30 participating teams. For Subtask 2 on evidence retrieval, our best result obtained an F1-score of 0.7940, ranking ninth of 19 submissions.
Use of NLP in the Context of Belief states of Ethnic Minorities in Latin America

The major goal of our study is to test methods in NLP in the domain of health care education related to Covid-19 of vulnerable groups such as indigenous people from Latin America.In order to achieve this goal, we asked participants in a survey questionnaire to provide answers about health related topics.We used these answers to measure the health education status of our participants.In this paper, we summarize the results from our NLP-application on the participants' answers.In the first experiment, we use embeddings-based tools to measure the semantic similarity between participants' answers and "expert" or "reference" answers.In the second experiment, we use synonym-based methods to classify answers under topics.We compare the results from both experiments with human annotations.Our results show that the tested NLP-methods reach a significantly lower accuracy score than human annotations in both experiments.We explain this difference by the assumption that human annotators are much better in pragmatic inferencing necessary to classify the semantic similarity and topic classification of answers.
Robustness of Multi-Source MT to Transcription Errors

Automatic speech translation is sensitive to speech recognition errors, but in a multilingual scenario, the same content may be available in various languages via simultaneous interpreting, dubbing or subtitling.In this paper, we hypothesize that leveraging multiple sources will improve translation quality if the sources complement one another in terms of correct information they contain.To this end, we first show that on a 10-hour ESIC corpus, the ASR errors in the original English speech and its simultaneous interpreting into German and Czech are mutually independent.We then use two sources, English and German, in a multi-source setting for translation into Czech to establish its robustness to ASR errors.Furthermore, we observe this robustness when translating both noisy sources together in a simultaneous translation setting.Our results show that multisource neural machine translation has the potential to be useful in a real-time simultaneous translation setting, thereby motivating further investigation in this area.
Deductive Additivity for Planning of Natural Language Proofs

Current natural language systems designed for multi-step claim validation typically operate in two phases: retrieve a set of relevant premise statements using heuristics (planning), then generate novel conclusions from those statements using a large language model (deduction).The planning step often requires expensive Transformer operations and does not scale to arbitrary numbers of premise statements.In this paper, we investigate whether an efficient planning heuristic is possible via embedding spaces compatible with deductive reasoning.Specifically, we evaluate whether embedding spaces exhibit a property we call deductive additivity: the sum of premise statement embeddings should be close to embeddings of conclusions based on those premises.We explore multiple sources of off-the-shelf dense embeddings in addition to fine-tuned embeddings from GPT3 and sparse embeddings from BM25.We study embedding models both intrinsically, evaluating whether the property of deductive additivity holds, and extrinsically, using them to assist planning in natural language proof generation.Lastly, we create a dataset, Single-Step Reasoning Contrast (SSRC), to further probe performance on various reasoning types.Our findings suggest that while standard embedding methods frequently embed conclusions near the sums of their premises, they fall short of being effective heuristics and lack the ability to model certain categories of reasoning.
A smashed glass cannot be full: Generation of Commonsense Explanations through Prompt-based Few-shot Learning

We assume that providing explanations is a process to elicit implicit knowledge in human communication, and propose a general methodology to generate commonsense explanations from pairs of semantically related sentences.We take advantage of both prompting applied to large, encoder-decoder pre-trained language models, and few-shot learning techniques, such as pattern-exploiting training.Experiments run on the e-SNLI dataset show that the proposed method achieves state-of-the-art results on the explanation generation task, with a substantial reduction of labelled data.The obtained results open new perspective on a number of tasks involving the elicitation of implicit knowledge.
Deep Active Learning for Morphophonological Processing

Building a system for morphological processing is a challenging task in morphologically complex languages like Arabic.Although there are some deep learning based models that achieve successful results, these models rely on a large amount of annotated data.Building such datasets, specially for some of the lower-resource Arabic dialects, is very difficult, time-consuming, and expensive.In addition, some parts of the annotated data do not contain useful information for training machine learning models.Active learning strategies allow the learner algorithm to select the most informative samples for annotation.There has been little research that focuses on applying active learning for morphological inflection and morphophonological processing.In this paper, we have proposed a deep active learning method for this task.Our experiments on Egyptian Arabic show that with only about 30% of annotated data, we achieve the same results as does the state-of-the-art model on the whole dataset.
Enhancing Ancient Chinese Understanding with Derived Noisy Syntax Trees

Despite the rapid development of neural-based models, syntax still plays a crucial role in modern natural language processing.However, few studies have incorporated syntactic information into ancient Chinese understanding tasks due to the lack of syntactic annotation.This paper explores the role of syntax in ancient Chinese understanding based on the noisy syntax trees from unsupervised derivation and modern Chinese syntax parsers.On top of that, we propose a novel syntax encoding component -confidence-based syntax encoding network (cSEN) to alleviate the side effects from the existing noise caused by unsupervised syntax derivation and the incompatibility between ancient and modern Chinese.Experiments on two typical ancient Chinese understanding tasks, ancient poetry theme classification and ancient-modern Chinese translation, demonstrate that syntactic information can effectively enhance the understanding of ancient Chinese over strong baselines, and that the proposed cSEN plays an important role in noisy scenarios.
Script Normalization for Unconventional Writing of Under-Resourced Languages in Bilingual Communities

The wide accessibility of social media has provided linguistically under-represented communities with an extraordinary opportunity to create content in their native languages.This, however, comes with certain challenges in script normalization, particularly where the speakers of a language in a bilingual community rely on another script or orthography to write their native language.This paper addresses the problem of script normalization for several such languages that are mainly written in a Perso-Arabic script.Using synthetic data with various levels of noise and a transformerbased model, we demonstrate that the problem can be effectively remediated.We conduct a small-scale evaluation of real data as well.Our experiments indicate that script normalization is also beneficial to improve the performance of downstream tasks such as machine translation and language identification.
Risks and NLP Design: A Case Study on Procedural Document QA

As NLP systems are increasingly deployed at scale, concerns about their potential negative impacts have attracted the attention of the research community, yet discussions of risk have mostly been at an abstract level and focused on generic AI or NLP applications.We argue that clearer assessments of risks and harms to users-and concrete strategies to mitigate them-will be possible when we specialize the analysis to more concrete applications and their plausible users.As an illustration, this paper is grounded in cooking recipe procedural document question answering (ProcDocQA), where there are well-defined risks to users such as injuries or allergic reactions.Our case study shows that an existing language model, applied in "zero-shot" mode, quantitatively answers real-world questions about recipes as well or better than the humans who have answered the questions on the web.Using a novel questionnaire informed by theoretical work on AI risk, we conduct a risk-oriented error analysis that could then inform the design of a future system to be deployed with lower risk of harm and better performance.
A Method for Studying Semantic Construal in Grammatical Constructions with Interpretable Contextual Embedding Spaces

We study semantic construal in grammatical constructions using large language models.First, we project contextual word embeddings into three interpretable semantic spaces, each defined by a different set of psycholinguistic feature norms.We validate these interpretable spaces and then use them to automatically derive semantic characterizations of lexical items in two grammatical constructions: nouns in subject or object position within the same sentence, and the AANN construction (e.g., 'a beautiful three days').We show that a word in subject position is interpreted as more agentive than the very same word in object position, and that the nouns in the AANN construction are interpreted as more measurement-like than when in the canonical alternation.Our method can probe the distributional meaning of syntactic constructions at a templatic level, abstracted away from specific lexemes.
Transfer Learning for Low-Resource Clinical Named Entity Recognition

We propose a transfer learning method that adapts a high-resource English clinical NER model to low-resource languages and domains using only small amounts of in-domain annotated data.Our approach involves translating in-domain datasets to English, fine-tuning the English model on the translated data, and then transferring it to the target language/domain.Experiments on Spanish, French, and conversational clinical text datasets show accuracy gains over models trained on target data alone.Our method achieves state-of-the-art performance and can enable clinical NLP in more languages and modalities with limited resources.
Explicit Syntactic Guidance for Neural Text Generation

Most existing text generation models follow the sequence-to-sequence paradigm.Generative Grammar suggests that humans generate natural language texts by learning language grammar.We propose a syntax-guided generation schema, which generates the sequence guided by a constituency parse tree in a topdown direction.The decoding process can be decomposed into two parts: (1) predicting the infilling texts for each constituent in the lexicalized syntax context given the source sentence;(2) mapping and expanding each constituent to construct the next-level syntax context.Accordingly, we propose a structural beam search method to find possible syntax structures hierarchically.Experiments on paraphrase generation and machine translation show that the proposed method outperforms autoregressive baselines, while also demonstrating effectiveness in terms of interpretability, controllability, and diversity.
LCT-1 at SemEval-2023 Task 10: Pre-training and Multi-task Learning for Sexism Detection and Classification

Misogyny and sexism are growing problems in social media.Advances have been made in online sexism detection but the systems are often uninterpretable.SemEval-2023 Task 10 on Explainable Detection of Online Sexism aims at increasing explainability of the sexism detection, and our team participated in all the proposed subtasks.Our system is based on further domain-adaptive pre-training (Gururangan et al., 2020).Building on the Transformerbased models with the domain adaptation, we compare fine-tuning with multi-task learning and show that each subtask requires a different system configuration.In our experiments, multi-task learning performs on par with standard fine-tuning for sexism detection and noticeably better for coarse-grained sexism classification, while fine-tuning is preferable for fine-grained classification 1 .
PV2TEA: Patching Visual Modality to Textual-Established Information Extraction

Information extraction, e.g., attribute value extraction, has been extensively studied and formulated based only on text.However, many attributes can benefit from image-based extraction, like color, shape, pattern, among others.The visual modality has long been underutilized, mainly due to multimodal annotation difficulty.In this paper, we aim to patch the visual modality to the textual-established attribute information extractor.The cross-modality integration faces several unique challenges: (C1) images and textual descriptions are loosely paired intra-sample and inter-samples; (C2) images usually contain rich backgrounds that can mislead the prediction; (C3) weakly supervised labels from textual-established extractors are biased for multimodal training.We present PV2TEA, an encoder-decoder architecture equipped with three bias reduction schemes: (S1) Augmented label-smoothed contrast to improve the cross-modality alignment for loosely-paired image and text; (S2) Attention-pruning that adaptively distinguishes the visual foreground; (S3) Two-level neighborhood regularization that mitigates the label textual bias via reliability estimation.Empirical results on real-world e-Commerce datasets 1 demonstrate up to 11.74% absolute (20.97% relatively) F 1 increase over unimodal baselines.* Work was done when Hejie was an intern at Amazon. 1 The code and the human-annotated