Reviewerly/exHarmony huggingface dataset:
CoLaDa: A Collaborative Label Denoising Framework for Cross-lingual Named Entity Recognition

Cross-lingual named entity recognition (NER) aims to train an NER system that generalizes well to a target language by leveraging labeled data in a given source language.Previous work alleviates the data scarcity problem by translating source-language labeled data or performing knowledge distillation on target-language unlabeled data.However, these methods may suffer from label noise due to the automatic labeling process.In this paper, we propose CoLaDa, a Collaborative Label Denoising Framework, to address this problem.Specifically, we first explore a model-collaboration-based denoising scheme that enables models trained on different data sources to collaboratively denoise pseudo labels used by each other.We then present an instance-collaboration-based strategy that considers the label consistency of each token's neighborhood in the representation space for denoising.Experiments on different benchmark datasets show that the proposed CoLaDa achieves superior results compared to previous methods, especially when generalizing to distant languages.
Imagination is All You Need! Curved Contrastive Learning for Abstract Sequence Modeling Utilized on Long Short-Term Dialogue Planning

Inspired by the curvature of space-time (Einstein, 1921), we introduce Curved Contrastive Learning (CCL), a novel representation learning technique for learning the relative turn distance between utterance pairs in multi-turn dialogues.The resulting bi-encoder models can guide transformers as a response ranking model towards a goal in a zero-shot fashion by projecting the goal utterance and the corresponding reply candidates into a latent space.Here the cosine similarity indicates the distance/reachability of a candidate utterance toward the corresponding goal.Furthermore, we explore how these forward-entailing language representations can be utilized for assessing the likelihood of sequences by the entailment strength i.e. through the cosine similarity of its individual members (encoded separately) as an emergent property in the curved space.These non-local properties allow us to imagine the likelihood of future patterns in dialogues, specifically by ordering/identifying future goal utterances that are multiple turns away, given a dialogue context.As part of our analysis, we investigate characteristics that make conversations (un)plannable and find strong evidence of planning capability over multiple turns (in 61.56% over 3 turns) in conversations from the DailyDialog (Li et al., 2017) dataset.Finally, we show how we achieve higher efficiency in sequence modeling tasks compared to previous work thanks to our relativistic approach, where only the last utterance needs to be encoded and computed during inference.
An Exploratory Study on Model Compression for Text-to-SQL

Text-to-SQL translates user queries into SQL statements that can retrieve relevant answers from relational databases.Recent approaches to Text-to-SQL rely on pre-trained language models that are computationally expensive and technically challenging to deploy in realworld applications that require real-time or on-device processing capabilities.In this paper, we perform a focused study on the feasibility of applying recent model compression techniques to sketch-based and sequence-tosequence Text-to-SQL models.Our results reveal that sketch-based Text-to-SQL models generally have higher inference efficiency and respond better to model compression than sequence-to-sequence models, making them ideal for real-world deployments, especially in use cases with simple SQL statements.
Kanbun-LM: Reading and Translating Classical Chinese in Japanese Methods by Language Models

Recent studies in natural language processing (NLP) have focused on modern languages and achieved state-of-the-art results in many tasks.Meanwhile, little attention has been paid to ancient texts and related tasks.Classical Chinese first came to Japan approximately 2,000 years ago.It was gradually adapted to a Japanese form called Kanbun-Kundoku (Kanbun) in Japanese reading and translating methods, which has significantly impacted Japanese literature.However, compared to the rich resources of ancient texts in mainland China, Kanbun resources remain scarce in Japan.To solve this problem, we construct the first Classical-Chinese-to-Kanbun dataset in the world.Furthermore, we introduce two tasks, character reordering and machine translation, both of which play a significant role in Kanbun comprehension.We also test the current language models on these tasks and discuss the best evaluation method by comparing the results with human scores.We release our code and dataset on GitHub.
A Sequence-to-Sequence&amp;Set Model for Text-to-Table Generation

Recently, the text-to-table generation task has attracted increasing attention due to its wide applications.In this aspect, the dominant model (Wu et al., 2022) formalizes this task as a sequence-to-sequence generation task and serializes each table into a token sequence during training by concatenating all rows in a topdown order.However, it suffers from two serious defects: 1) the predefined order introduces a wrong bias during training, which highly penalizes shifts in the order between rows; 2) the error propagation problem becomes serious when the model outputs a long token sequence.In this paper, we first conduct a preliminary study to demonstrate the generation of most rows is order-insensitive.Furthermore, we propose a novel sequence-to-sequence&set text-to-table generation model.Specifically, in addition to a text encoder encoding the input text, our model is equipped with a table header generator to first output a table header, i.e., the first row of the table, in the manner of sequence generation.Then we use a table body generator with learnable row embeddings and column embeddings to generate a set of table body rows in parallel.Particularly, to deal with the issue that there is no correspondence between each generated table body row and target during training, we propose a target assignment strategy based on the bipartite matching between the first cells of generated table body rows and targets.Experiment results show that our model significantly surpasses the baselines, achieving state-of-the-art performance on commonlyused datasets. 1
Diffusion Theory as a Scalpel: Detecting and Purifying Poisonous Dimensions in Pre-trained Language Models Caused by Backdoor or Bias

Pre-trained Language Models (PLMs) may be poisonous with backdoors or bias injected by the suspicious attacker during the fine-tuning process.A core challenge of purifying potentially poisonous PLMs is precisely finding poisonous dimensions.To settle this issue, we propose the Fine-purifying approach, which utilizes the diffusion theory to study the dynamic process of fine-tuning for finding potentially poisonous dimensions.According to the relationship between parameter drifts and Hessians of different dimensions, we can detect poisonous dimensions with abnormal dynamics, purify them by resetting them to clean pretrained weights, and then fine-tune the purified weights on a small clean dataset.To the best of our knowledge, we are the first to study the dynamics guided by the diffusion theory for safety or defense purposes.Experimental results validate the effectiveness of Fine-purifying even with a small clean dataset.
UM-DFKI Maltese Speech Translation

For the 2023 IWSLT (Agarwal et al., 2023) Maltese Speech Translation Task, UM-DFKI jointly presents a cascade solution which achieves 0.6 BLEU.While this is the first time that a Maltese speech translation task has been released by IWSLT, this paper explores previous solutions for other speech translation tasks, focusing primarily on low-resource scenarios.Moreover, we present our method of fine-tuning XLS-R models for Maltese ASR using a collection of multi-lingual speech corpora as well as the fine-tuning of the mBART model for Maltese to English machine translation.
On “Scientific Debt” in NLP: A Case for More Rigour in Language Model Pre-Training Research

This evidence-based position paper critiques current research practices within the language model pre-training literature.Despite rapid recent progress afforded by increasingly better pre-trained language models (PLMs), current PLM research practices often conflate different possible sources of model improvement, without conducting proper ablation studies and principled comparisons between different models under comparable conditions.These practices (i) leave us ill-equipped to understand which pre-training approaches should be used under what circumstances; (ii) impede reproducibility and credit assignment; and (iii) render it difficult to understand: "How exactly does each factor contribute to the progress that we have today?"We provide a case in point by revisiting the success of BERT over its baselines, ELMo and GPT-1, and demonstrate how -under comparable conditions where the baselines are tuned to a similar extent -these baselines (and even-simpler variants thereof) can, in fact, achieve competitive or better performance than BERT.These findings demonstrate how disentangling different factors of model improvements can lead to valuable new insights.We conclude with recommendations for how to encourage and incentivize this line of work, and accelerate progress towards a better and more systematic understanding of what factors drive the progress of our foundation models today.
Samsung Research China - Beijing at SemEval-2023 Task 2: An AL-R Model for Multilingual Complex Named Entity Recognition

This paper describes our system for SemEval-2023 Task 2 Multilingual Complex Named Entity Recognition (MultiCoNER II).Our team Samsung Research China -Beijing proposes an AL-R (Adjustable Loss RoBERTa) model to boost the performance of recognizing short and complex entities with the challenges of longtail data distribution, out of knowledge base and noise scenarios.We first employ an adjustable dice loss optimization objective to overcome the issue of long-tail data distribution, which is also proved to be noise-robusted, especially in combatting the issue of fine-grained label confusing.Besides, we develop our own knowledge enhancement tool to provide related contexts for the short context setting and address the issue of out of knowledge base.Experiments have verified the validation of our approaches.In the official test result, our system ranked 2nd on the English track in this task.
Prompt-Guided Retrieval Augmentation for Non-Knowledge-Intensive Tasks

Retrieval-augmented methods have received increasing attention to support downstream tasks by leveraging useful information from external resources.Recent studies mainly focus on exploring retrieval to solve knowledgeintensive (KI) tasks.However, the potential of retrieval for most non-knowledge-intensive (NKI) tasks remains under-explored.There are two main challenges to leveraging retrievalaugmented methods for NKI tasks: 1) the demand for diverse relevance score functions and 2) the dilemma between training cost and task performance.To address these challenges, we propose a two-stage framework for NKI tasks, named PGRA.In the first stage, we adopt a task-agnostic retriever to build a shared static index and select candidate evidence efficiently.In the second stage, we design a prompt-guided reranker to rerank the nearest evidence according to task-specific relevance for the reader.Experimental results show that PGRA outperforms other state-ofthe-art retrieval-augmented methods.Our analyses further investigate the influence factors to model performance and demonstrate the generality of PGRA.
LayoutMask: Enhance Text-Layout Interaction in Multi-modal Pre-training for Document Understanding

Visually-rich Document Understanding (VrDU) has attracted much research attention over the past years.Pre-trained models on a large number of document images with transformer-based backbones have led to significant performance gains in this field.The major challenge is how to fusion the different modalities (text, layout, and image) of the documents in a unified model with different pre-training tasks.This paper focuses on improving text-layout interactions and proposes a novel multi-modal pre-training model, LayoutMask.LayoutMask uses local 1D position, instead of global 1D position, as layout input and has two pre-training objectives: (1) Masked Language Modeling: predicting masked tokens with two novel masking strategies; (2) Masked Position Modeling: predicting masked 2D positions to improve layout representation learning.LayoutMask can enhance the interactions between text and layout modalities in a unified model and produce adaptive and robust multimodal representations for downstream tasks.Experimental results show that our proposed method can achieve state-of-the-art results on a wide variety of VrDU problems, including form understanding, receipt understanding, and document image classification.
Context-Aware Query Rewriting for Improving Users’ Search Experience on E-commerce Websites

E-commerce queries are often short and ambiguous.Consequently, query understanding often uses query rewriting to disambiguate userinput queries.While using e-commerce search tools, users tend to enter multiple searches, which we call context, before purchasing.These history searches contain contextual insights about users' true shopping intents.Therefore, modeling such contextual information is critical to a better query rewriting model.However, existing query rewriting models ignore users' history behaviors and consider only the instant search query, which is often a short string offering limited information about the true shopping intent.We propose an end-to-end context-aware query rewriting model to bridge this gap, which takes the search context into account.Specifically, our model builds a session graph using the history search queries and their contained words.We then employ a graph attention mechanism that models cross-query relations and computes contextual information of the session.The model subsequently calculates session representations by combining the contextual information with the instant search query using an aggregation network.The session representations are then decoded to generate rewritten queries.Empirically, we demonstrate the superiority of our method to state-ofthe-art approaches under various metrics.
Team INF-UFRGS at SemEval-2023 Task 7: Supervised Contrastive Learning for Pair-level Sentence Classification and Evidence Retrieval

This paper describes the EvidenceSCL system submitted by our team (INF-UFRGS) to SemEval-2023 Task 7: Multi-Evidence Natural Language Inference for Clinical Trial Data (NLI4CT).NLI4CT is divided into two tasks, one for determining the inference relation between a pair of statements in clinical trials and a second for retrieving a set of supporting facts from the premises necessary to justify the label predicted in the first task.Our approach uses pair-level supervised contrastive learning to classify pairs of sentences.We trained EvidenceSCL on two datasets created from NLI4CT and additional data from other NLI datasets.We show that our approach can address both goals of NLI4CT, and although it reached an intermediate position in the ranking of participating system, there is room for improvement in the technique.
Pay More Attention to Relation Exploration for Knowledge Base Question Answering

Knowledge base question answering (KBQA) is a challenging task that aims to retrieve correct answers from large-scale knowledge bases.Existing attempts primarily focus on entity representation and final answer reasoning, which results in limited supervision for this task.Moreover, the relations, which empirically determine the reasoning path selection, are not fully considered in recent advancements.In this study, we propose a novel framework, RE-KBQA, that utilizes relations in the knowledge base to enhance entity representation and introduce additional supervision.We explore guidance from relations in three aspects, including (1) distinguishing similar entities by employing a variational graph auto-encoder to learn relation importance; (2) exploring extra supervision by predicting relation distributions as soft labels with a multi-task scheme; (3) designing a relation-guided re-ranking algorithm for post-processing.Experimental results on two benchmark datasets demonstrate the effectiveness and superiority of our framework, improving the F1 score by 5.8% from 40.5 to 46.3 on CWQ and 5.7% from 62.8 to 68.5 on WebQSP, better or on par with state-of-the-art methods.
Compositional Data Augmentation for Abstractive Conversation Summarization

Recent abstractive conversation summarization systems generally rely on large-scale datasets with annotated summaries.However, collecting and annotating these conversations can be a time-consuming and labor-intensive task.To address this issue, in this work, we present a sub-structure level compositional data augmentation method, COMPO, for generating diverse and high-quality pairs of conversations and summaries.Specifically, COMPO first extracts conversation structures like topic splits and action triples as basic units.Then we organize these semantically meaningful conversation snippets compositionally to create new training instances.Additionally, we explore noise-tolerant settings in both self-training and joint-training paradigms to make the most of these augmented samples.Our experiments on benchmark datasets, SAMSum and Di-alogSum, show that COMPO substantially outperforms prior baseline methods by achieving a nearly 10% increase of ROUGE scores with limited data.We have publically released our code at https://github.com/ozyyshr/Compo.
Multi-lingual and Multi-cultural Figurative Language Understanding

Figurative language permeates human communication, but at the same time is relatively understudied in NLP.Datasets have been created in English to accelerate progress towards measuring and improving figurative language processing in language models (LMs).However, the use of figurative language is an expression of our cultural and societal experiences, making it difficult for these phrases to be universally applicable.In this work, we create a figurative language inference dataset, MABL, for seven diverse languages associated with a variety of cultures: Hindi, Indonesian, Javanese, Kannada, Sundanese, Swahili and Yoruba.Our dataset reveals that each language relies on cultural and regional concepts for figurative expressions, with the highest overlap between languages originating from the same region.We assess multilingual LMs' abilities to interpret figurative language in zeroshot and few-shot settings.All languages exhibit a significant deficiency compared to English, with variations in performance reflecting the availability of pre-training and fine-tuning data, emphasizing the need for LMs to be exposed to a broader range of linguistic and cultural variation during training.
Aligning Offline Metrics and Human Judgments of Value for Code Generation Models

Large language models have demonstrated great potential to assist programmers in generating code.For such human-AI pair programming scenarios, we empirically demonstrate that while generated code are most often evaluated in terms of their functional correctness (i.e., whether generations pass available unit tests), correctness does not fully capture (e.g., may underestimate) the productivity gains these models may provide.Through a user study with N = 49 experienced programmers, we show that while correctness captures high-value generations, programmers still rate code that fails unit tests as valuable if it reduces the overall effort needed to complete a coding task.Finally, we propose a hybrid metric that combines functional correctness and syntactic similarity and show that it achieves a 14% stronger correlation with value and can therefore better represent real-world gains when evaluating and comparing models.
Aligning Factual Consistency for Clinical Studies Summarization through Reinforcement Learning

In the rapidly evolving landscape of medical research, accurate and concise summarization of clinical studies is crucial to support evidencebased practice.This paper presents a novel approach to clinical studies summarization, leveraging reinforcement learning to enhance factual consistency and align with human annotator preferences.Our work focuses on two tasks: Conclusion Generation and Review Generation.We train a CONFIT summarization model that outperforms GPT-3 and previous state-of-theart models on the same datasets and collects expert and crowd-worker annotations to evaluate the quality and factual consistency of the generated summaries.These annotations enable us to measure the correlation of various automatic metrics, including modern factual evaluation metrics like QAFactEval, with humanassessed factual consistency.By employing top-correlated metrics as objectives for a reinforcement learning model, we demonstrate improved factuality in generated summaries that are preferred by human annotators.
Can Language Models Be Specific? How?

He is a person", "Paris is located on the earth".Both statements are correct but meaningless -due to lack of specificity.In this paper, we propose to measure how specific the language of pre-trained language models (PLMs) is.To achieve this, we introduce a novel approach to build a benchmark for specificity testing by forming masked token prediction tasks with prompts.For instance, given "Toronto is located in [MASK].",we want to test whether a more specific answer will be better filled in by PLMs, e.g., Ontario instead of Canada.From our evaluations, we show that existing PLMs have only a slight preference for more specific answers.We identify underlying factors affecting the specificity and design two prompt-based methods to improve the specificity.Results show that the specificity of the models can be improved by the proposed methods without additional training.We hope this work can bring to awareness the notion of specificity of language models and encourage the research community to further explore this important but understudied problem. 1
Multi-layered Annotation of Conversation-like Narratives in German

This work presents two corpora based on excerpts from two German novels with an informal narration style.We performed fine-grained multi-layer annotations of animate referents, assigning local and global prominence-lending features to the annotated referring expressions.In addition, our corpora include annotations of intra-sentential segments, which can serve as a more reliable unit of length measurement.Furthermore, we present two exemplary studies demonstrating how to use these corpora.
Stanford MLab at SemEval-2023 Task 10: Exploring GloVe- and Transformer-Based Methods for the Explainable Detection of Online Sexism

In this paper, we discuss the methods we applied at SemEval-2023 Task 10: Towards the Explainable Detection of Online Sexism.Given an input text, we perform three classification tasks to predict whether the text is sexist and classify the sexist text into subcategories in order to provide an additional explanation as to why the text is sexist.We explored many different types of models, including GloVe embeddings as the baseline approach, transformer-based deep learning models like BERT, RoBERTa, and DeBERTa, ensemble models, and model blending.We explored various data cleaning and augmentation methods to improve model performance.Pretraining transformer models yielded significant improvements in performance, and ensembles and blending slightly improved robustness in the F1 score.