Reviewerly/exHarmony huggingface dataset:
Text-to-SQL Error Correction with Language Models of Code

Despite recent progress in text-to-SQL parsing, current semantic parsers are still not accurate enough for practical use.In this paper, we investigate how to build automatic text-to-SQL error correction models.Noticing that tokenlevel edits are out of context and sometimes ambiguous, we propose building clause-level edit models instead.Besides, while most language models of code are not specifically pre-trained for SQL, they know common data structures and their operations in programming languages such as Python.Thus, we propose a novel representation for SQL queries and their edits that adheres more closely to the pre-training corpora of language models of code.Our error correction model improves the exact set match accuracy of different parsers by 2.4-6.5 and obtains up to 4.3 point absolute improvement over two strong baselines. 1
Masked Latent Semantic Modeling: an Efficient Pre-training Alternative to Masked Language Modeling

In this paper, we propose an alternative to the classic masked language modeling (MLM) pre-training paradigm, where we modify the objective from the reconstruction of the exact identity of randomly selected masked subwords to the prediction of their latent semantic properties.We coin the proposed pre-training technique masked latent semantic modeling (MLSM for short).In order to make the contextualized determination of the latent semantic properties of the masked subwords possible, we rely on an unsupervised technique using sparse coding.Our experimental results reveal that the fine-tuned performance of those models that we pre-trained via MLSM is consistently and significantly better compared to the use of vanilla MLM pre-training and other strong baselines.
Counterfactual Multihop QA: A Cause-Effect Approach for Reducing Disconnected Reasoning

Multi-hop QA requires reasoning over multiple supporting facts to answer the question.However, the existing QA models always rely on shortcuts, e.g., providing the true answer by only one fact, rather than multi-hop reasoning, which is referred to as disconnected reasoning problem.To alleviate this issue, we propose a novel counterfactual multihop QA, a causaleffect approach that enables to reduce the disconnected reasoning.It builds upon explicitly modeling of causality: 1) the direct causal effects of disconnected reasoning and 2) the causal effect of true multi-hop reasoning from the total causal effect.With the causal graph, a counterfactual inference is proposed to disentangle the disconnected reasoning from the total causal effect, which provides us a new perspective and technology to learn a QA model that exploits the true multi-hop reasoning instead of shortcuts.Extensive experiments have been conducted on the benchmark HotpotQA dataset, which demonstrate that the proposed method can achieve notable improvement on reducing disconnected reasoning.For example, our method achieves 5.8% higher points of its Supp s score on HotpotQA through true multihop reasoning.
KINLP at SemEval-2023 Task 12: Kinyarwanda Tweet Sentiment Analysis

This paper describes the system entered by the author to the SemEval-2023 Task 12: Sentiment analysis for African languages.The system focuses on the Kinyarwanda language and uses a language-specific model.Kinyarwanda morphology is modeled in a two tier transformer architecture and the transformer model is pre-trained on a large text corpus using multi-task masked morphology prediction.The model is deployed on an experimental platform that allows users to experiment with the pretrained language model fine-tuning without the need to write machine learning code.Our final submission to the shared task achieves second ranking out of 34 teams in the competition, achieving 72.50% weighted F1 score.Our analysis of the evaluation results highlights challenges in achieving high accuracy on the task and identifies areas for improvement.* This work is part of an independent research and development effort towards Kinyarwanda language technology.
Breeding Machine Translations: Evolutionary approach to survive and thrive in the world of automated evaluation

We propose a genetic algorithm (GA) based method for modifying n-best lists produced by a machine translation (MT) system.Our method offers an innovative approach to improving MT quality and identifying weaknesses in evaluation metrics.Using common GA operations (mutation and crossover) on a list of hypotheses in combination with a fitness function (an arbitrary MT metric), we obtain novel and diverse outputs with high metric scores.With a combination of multiple MT metrics as the fitness function, the proposed method leads to an increase in translation quality as measured by other held-out automatic metrics.With a single metric (including popular ones such as COMET) as the fitness function, we find blind spots and flaws in the metric.This allows for an automated search for adversarial examples in an arbitrary metric, without prior assumptions on the form of such example.As a demonstration of the method, we create datasets of adversarial examples and use them to show that reference-free COMET is substantially less robust than the reference-based version.
Learning Better Masking for Better Language Model Pre-training

Masked Language Modeling (MLM) has been widely used as the denoising objective in pretraining language models (PrLMs).Existing PrLMs commonly adopt a Random-Token Masking strategy where a fixed masking ratio is applied and different contents are masked by an equal probability throughout the entire training.However, the model may receive a complicated impact from pre-training status, which changes accordingly as training time goes on.In this paper, we show that such time-invariant MLM settings on masking ratio and masked content are unlikely to deliver an optimal outcome, which motivates us to explore the influence of time-variant MLM settings.We propose two scheduled masking approaches that adaptively tune the masking ratio and masked content in different training stages, which improves the pre-training efficiency and effectiveness verified on the downstream tasks.Our work is a pioneer study on time-variant masking strategy on ratio and content and gives a better understanding of how masking ratio and masked content influence the MLM pretraining 1 .
Synthetic Dataset for Evaluating Complex Compositional Knowledge for Natural Language Inference

We introduce a synthetic dataset called Sentences Involving Complex Compositional Knowledge (SICCK) and a novel analysis that investigates the performance of Natural Language Inference (NLI) models to understand compositionality in logic.We produce 1,304 sentence pairs by modifying 15 examples from the SICK dataset (Marelli et al., 2014).To this end, we modify the original texts using a set of phrases -modifiers that correspond to universal quantifiers, existential quantifiers, negation, and other concept modifiers in Natural Logic (NL) (MacCartney, 2009).We use these phrases to modify the subject, verb, and object parts of the premise and hypothesis.Lastly, we annotate these modified texts with the corresponding entailment labels following NL rules.We conduct a preliminary verification of how well the change in the structural and semantic composition is captured by neural NLI models, in both zero-shot and fine-tuned scenarios.We found that the performance of NLI models under the zero-shot setting is poor, especially for modified sentences with negation and existential quantifiers.After fine-tuning this dataset, we observe that models continue to perform poorly over negation, existential and universal modifiers.
Incomplete Utterance Rewriting as Sequential Greedy Tagging

The task of incomplete utterance rewriting has recently gotten much attention.Previous models struggled to extract information from the dialogue context, as evidenced by the low restoration scores.To address this issue, we propose a novel sequence tagging-based model, which is more adept at extracting information from context.Meanwhile, we introduce speakeraware embedding to model speaker variation.Experiments on multiple public datasets show that our model achieves optimal results on all nine restoration scores while having other metric scores comparable to previous state-of-theart models.Furthermore, benefitting from the model's simplicity, our approach outperforms most previous models on inference speed.
Automatic Identification of Code-Switching Functions in Speech Transcripts

Code-switching, or switching between languages, occurs for many reasons and has important linguistic, sociological, and cultural implications.Multilingual speakers code-switch for a variety of communicative functions, such as expressing emotions, borrowing terms, making jokes, introducing a new topic, etc.The function of code-switching may be quite useful for the analysis of linguists, cognitive scientists, speech therapists, and others, but is not readily apparent.To remedy this situation, we annotate and release a new dataset of functions of code-switching in Spanish-English.We build the first system (to our knowledge) to automatically identify a wide range of functions for which speakers code-switch in everyday speech, achieving an accuracy of 75% across all functions.
Bootstrapping Neural Relation and Explanation Classifiers

We introduce a method that self trains (or bootstraps) neural relation and explanation classifiers.Our work expands the supervised approach of (Tang and Surdeanu, 2022), which jointly trains a relation classifier with an explanation classifier that identifies context words important for the relation at hand, to semisupervised scenarios.In particular, our approach iteratively converts the explainable models' outputs to rules and applies them to unlabeled text to produce new annotations.Our evaluation on the TACRED dataset shows that our method outperforms the rule-based model we started from by 15 F1 points, outperforms traditional self-training that relies just on the relation classifier by 5 F1 points, and performs comparatively with the prompt-based approach of Sainz et al. (2021) (without requiring an additional natural language inference component). 1
Reimagining Retrieval Augmented Language Models for Answering Queries

Lifelogs are descriptions of experiences that a person had during their life.Lifelogs are created by fusing data from the multitude of digital services, such as online photos, maps, shopping and content streaming services.Question answering over lifelogs can offer personal assistants a critical resource when they try to provide advice in context.However, obtaining answers to questions over lifelogs is beyond the current state of the art of question answering techniques for a variety of reasons, the most pronounced of which is that lifelogs combine free text with some degree of structure such as temporal and geographical information.We create and publicly release TimelineQA 1 , a benchmark for accelerating progress on querying lifelogs.TimelineQA generates lifelogs of imaginary people.The episodes in the lifelog range from major life episodes such as high school graduation to those that occur on a daily basis such as going for a run.We describe a set of experiments on TimelineQA with several state-of-the-art QA models.Our experiments reveal that for atomic queries, an extractive QA system significantly out-performs a state-of-the-art retrieval-augmented QA system.For multi-hop queries involving aggregates, we show that the best result is obtained with a stateof-the-art table QA technique, assuming the ground truth set of episodes for deriving the answer is available.
Verifying Annotation Agreement without Multiple Experts: A Case Study with Gujarati SNACS

Good datasets are a foundation of NLP research, and form the basis for training and evaluating models of language use.While creating datasets, the standard practice is to verify the annotation consistency using a committee of human annotators.This norm assumes that multiple annotators are available, which is not the case for highly specialized tasks or lowresource languages.In this paper, we ask: Can we evaluate the quality of a dataset constructed by a single human annotator?To address this question, we propose four weak verifiers to help estimate dataset quality, and outline when each may be employed.We instantiate these strategies for the task of semantic analysis of adpositions in Gujarati, a low-resource language, and show that our weak verifiers concur with a double-annotation study.As an added contribution, we also release the first dataset with semantic annotations in Gujarati along with several model baselines.
Lego-MT: Learning Detachable Models for Massively Multilingual Machine Translation

Multilingual neural machine translation (MNMT) aims to build a unified model for many language directions.Existing monolithic models for MNMT encounter two challenges: parameter interference among languages and inefficient inference for large models.In this paper, we revisit the classic multi-way structures and develop a detachable model by assigning each language (or group of languages) to an individual branch that supports plug-and-play training and inference.To address the needs of learning representations for all languages in a unified space, we propose a novel efficient training recipe, upon which we build an effective detachable model, Lego-MT.For a fair comparison, we collect data from OPUS and build a translation benchmark covering 433 languages and 1.3B parallel data.Experiments show that Lego-MT with 1.2B parameters brings an average gain of 3.2 spBLEU.It even outperforms M2M-100 with 12B parameters.The proposed training recipe brings a 28.2× speedup over the conventional multi-way training method. 1
Using counterfactual contrast to improve compositional generalization for multi-step quantitative reasoning

In quantitative question answering, compositional generalization is one of the main challenges of state of the art models, especially when longer sequences of reasoning steps are required.In this paper we propose Counter-Comp, a method that uses counterfactual scenarios to generate samples with compositional contrast.Instead of a data augmentation approach, CounterComp is based on metric learning, which allows for direct sampling from the training set and circumvents the need for additional human labels.Our proposed auxiliary metric learning loss improves the performance of three state of the art models on four recently released datasets.We also show how the approach can improve OOD performance on unseen domains, as well as unseen compositions.Lastly, we demonstrate how the method can lead to better compositional attention patterns during training.
Using Learning Analytics for Adaptive Exercise Generation

Single Choice exercises constitute a central exercise type for language learning in a learner's progression from mere implicit exposure through input enhancement to productive language use in open exercises.Distractors that support learning in the individual zone of proximal development should not be derived from static analyses of learner corpora, but rely on dynamic learning analytics based on half-open exercises.We demonstrate how a system's error diagnosis module can be re-used for automatic and dynamic generation and adaptation of distractors, as well as to inform exercise generation in terms of relevant learning goals and reasonable chunking in Jumbled Sentences exercises.
Second Language Acquisition of Neural Language Models

With the success of neural language models (LMs), their language acquisition has gained much attention.This work sheds light on the second language (L2) acquisition of LMs, while previous work has typically explored their first language (L1) acquisition.Specifically, we trained bilingual LMs with a scenario similar to human L2 acquisition and analyzed their cross-lingual transfer from linguistic perspectives.Our exploratory experiments demonstrated that the L1 pretraining accelerated their linguistic generalization in L2, and language transfer configurations (e.g., the L1 choice, and presence of parallel texts) substantially affected their generalizations.These clarify their (non-)human-like L2 acquisition in particular aspects. 1
Leveraging Denoised Abstract Meaning Representation for Grammatical Error Correction

Grammatical Error Correction (GEC) is the task of correcting errorful sentences into grammatically correct, semantically consistent, and coherent sentences.Popular GEC models either use large-scale synthetic corpora or use a large number of human-designed rules.The former is costly to train, while the latter requires quite a lot of human expertise.In recent years, AMR, a semantic representation framework, has been widely used by many natural language tasks due to its completeness and flexibility.A non-negligible concern is that AMRs of grammatically incorrect sentences may not be exactly reliable.In this paper, we propose the AMR-GEC, a seq-to-seq model that incorporates denoised AMR as additional knowledge.Specifically, We design a semantic aggregated GEC model and explore denoising methods to get AMRs more reliable.Experiments on the BEA-2019 shared task and the CoNLL-2014 shared task have shown that AMR-GEC performs comparably to a set of strong baselines with a large number of synthetic data.Compared with the T5 model with synthetic data, AMR-GEC can reduce the training time by 32% while inference time is comparable.To the best of our knowledge, we are the first to incorporate AMR for grammatical error correction.
GIFT: Graph-Induced Fine-Tuning for Multi-Party Conversation Understanding

Addressing the issues of who saying what to whom in multi-party conversations (MPCs) has recently attracted a lot of research attention.However, existing methods on MPC understanding typically embed interlocutors and utterances into sequential information flows, or utilize only the superficial of inherent graph structures in MPCs.To this end, we present a plug-and-play and lightweight method named graph-induced fine-tuning (GIFT) which can adapt various Transformer-based pre-trained language models (PLMs) for universal MPC understanding.In detail, the full and equivalent connections among utterances in regular Transformer ignore the sparse but distinctive dependency of an utterance on another in MPCs.To distinguish different relationships between utterances, four types of edges are designed to integrate graph-induced signals into attention mechanisms to refine PLMs originally designed for processing sequential texts.We evaluate GIFT by implementing it into three PLMs, and test the performance on three downstream tasks including addressee recognition, speaker identification and response selection.Experimental results show that GIFT can significantly improve the performance of three PLMs on three downstream tasks and two benchmarks with only 4 additional parameters per encoding layer, achieving new state-of-theart performance on MPC understanding.
Jus Mundi at SemEval-2023 Task 6: Using a Frustratingly Easy Domain Adaption for a Legal Named Entity Recognition System

In this work, we present a Named Entity Recognition (NER) system that was trained using a Frustratingly Easy Domain Adaptation (FEDA) over multiple legal corpora.The goal was to create a NER capable of detecting 14 types of legal named entities in Indian judgments.Besides the FEDA architecture, we explored a method based on overlapping context and averaging tensors to process long input texts, which can be beneficial when processing legal documents.The proposed NER reached an F1-score of 0.9007 in the sub-task B of Semeval-2023 Task 6, Understanding Legal Texts.
A Simple Concatenation can Effectively Improve Speech Translation

A triple speech translation data comprises speech, transcription, and translation.In the end-to-end paradigm, text machine translation (MT) usually plays the role of a teacher model for the speech translation (ST) via knowledge distillation.Parameter sharing with the teacher is often adopted to construct the ST model architecture, however, the two modalities are independently fed and trained via different losses.This situation does not match ST's properties across two modalities and also limits the upper bound of the performance.Inspired by the works of video Transformer, we propose a simple unified cross-modal ST method, which concatenates speech and text as the input, and builds a teacher that can utilize both cross-modal information simultaneously.Experimental results show that in our unified ST framework, models can effectively utilize the auxiliary information from speech and text, and achieve compelling results on MuST-C datasets.
Better Language Models of Code through Self-Improvement

Pre-trained language models for code (PLMCs) have gained attention in recent research.These models are pre-trained on large-scale datasets using multi-modal objectives.However, finetuning them requires extensive supervision and is limited by the size of the dataset provided.We aim to improve this issue by proposing a data augmentation framework using knowledge distillation.Our framework utilizes knowledge gained during the pre-training and finetuning stage to augment training data, which is then used for the next step.We incorporate this framework into the state-of-the-art language models, such as CodeT5, CodeBERT, and UnixCoder.The results show that our framework significantly improves PLMCs' performance in sequence-generation tasks, such as code summarization and code generation in the CodeXGLUE benchmark.
JCT_DM at SemEval-2023 Task 10: Detection of Online Sexism: from Classical Models to Transformers

This paper presents the experimentation of systems for detecting online sexism relying on classical models, deep learning models, and transformer-based models. The systems aim to provide a comprehensive approach to handling the intricacies of online language, including slang and neologisms. The dataset consists of labeled and unlabeled data from Gab and Reddit, which allows for the development of unsupervised or semi-supervised models. The system utilizes TF-IDF with classical models, bidirectional models with embedding, and pre-trained transformer models. The paper discusses the experimental setup and results, demonstrating the effectiveness of the system in detecting online sexism.
NCUEE-NLP at SemEval-2023 Task 8: Identifying Medical Causal Claims and Extracting PIO Frames Using the Transformer Models

This study describes the model design of the NCUEE-NLP system for the SemEval-2023 Task 8. We use the pre-trained transformer models and fine-tune the task datasets to identify medical causal claims and extract population, intervention, and outcome elements in a Reddit post when a claim is given. Our best system submission for the causal claim identification subtask achieved a F1-score of 70.15%. Our best submission for the PIO frame extraction subtask achieved F1-scores of 37.78% for Population class, 43.58% for Intervention class, and 30.67% for Outcome class, resulting in a macro-averaging F1-score of 37.34%. Our system evaluation results ranked second position among all participating teams.
TohokuNLP at SemEval-2023 Task 5: Clickbait Spoiling via Simple Seq2Seq Generation and Ensembling

This paper describes our system submitted to SemEval-2023 Task 5: Clickbait Spoiling.We work on spoiler generation of the subtask 2 and develop a system which comprises two parts: 1) simple seq2seq spoiler generation and 2) post-hoc model ensembling.Using this simple method, we address the challenge of generating multipart spoiler.In the test set, our submitted system 1 outperformed the baseline by a large margin (approximately 10 points above on the BLEU score) for mixed types of spoilers.We also found that our system successfully handled the challenge of the multipart spoiler, confirming the effectiveness of our approach.