Reviewerly/exHarmony huggingface dataset:
AlignScore: Evaluating Factual Consistency with A Unified Alignment Function

Many text generation applications require the generated text to be factually consistent with input information.Automatic evaluation of factual consistency is challenging.Previous work has developed various metrics that often depend on specific functions, such as natural language inference (NLI) or question answering (QA), trained on limited data.Those metrics thus can hardly assess diverse factual inconsistencies (e.g., contradictions, hallucinations) that occur in varying inputs/outputs (e.g., sentences, documents) from different tasks.In this paper, we propose ALIGNSCORE, a new holistic metric that applies to a variety of factual inconsistency scenarios as above.ALIGN-SCORE is based on a general function of information alignment between two arbitrary text pieces.Crucially, we develop a unified training framework of the alignment function by integrating a large diversity of data sources, resulting in 4.7M training examples from 7 well-established tasks (NLI, QA, paraphrasing, fact verification, information retrieval, semantic similarity, and summarization).We conduct extensive experiments on large-scale benchmarks including 22 evaluation datasets, where 19 of the datasets were never seen in the alignment training.ALIGNSCORE achieves substantial improvement over a wide range of previous metrics.Moreover, ALIGNSCORE (355M parameters) matches or even outperforms metrics based on ChatGPT and GPT-4 that are orders of magnitude larger. 1
Personality Understanding of Fictional Characters during Book Reading

Comprehending characters' personalities is a crucial aspect of story reading.As readers engage with a story, their understanding of a character evolves based on new events and information; and multiple fine-grained aspects of personalities can be perceived.This leads to a natural problem of situated and fine-grained personality understanding.The problem has not been studied in the NLP field, primarily due to the lack of appropriate datasets mimicking the process of book reading.We present the first labeled dataset PERSONET for this problem.Our novel annotation strategy involves annotating user notes from online reading apps as a proxy for the original books.Experiments and human studies indicate that our dataset construction is both efficient and accurate; and our task heavily relies on long-term context to achieve accurate predictions for both machines and humans. 1
iREL at SemEval-2023 Task 9: Improving understanding of multilingual Tweets using Translation-Based Augmentation and Domain Adapted Pre-Trained Models

This paper describes our system (iREL) for Tweet intimacy analysis shared task of the Se-mEval 2023 workshop at ACL 2023.Our system achieved an overall Pearson's r score of 0.5924 and ranked 10th on the overall leaderboard.For the unseen languages, we ranked third on the leaderboard and achieved a Pearson's r score of 0.485.We used a single multilingual model for all languages, as discussed in this paper.We provide a detailed description of our pipeline along with multiple ablation experiments to further analyse each component of the pipeline.We demonstrate how translation-based augmentation, domainspecific features, and domain-adapted pretrained models improve the understanding of intimacy in tweets.The code can be found at https://github.com/bhavyajeet/Multilingualtweet-intimacy
Decoding Symbolism in Language Models

This work explores the feasibility of eliciting knowledge from language models (LMs) to decode symbolism, recognizing something (e.g., roses) as a stand-in for another (e.g., love).We present our evaluative framework, Symbolism Analysis (SymbA), which compares LMs (e.g., RoBERTa, GPT-J) on different types of symbolism and analyzes the outcomes along multiple metrics.Our findings suggest that conventional symbols are more reliably elicited from LMs while situated symbols are more challenging.Results also reveal the negative impact of the bias in pre-trained corpora.We further demonstrate that a simple re-ranking strategy can mitigate the bias and significantly improve model performances to be on par with human performances in some cases.
SKAM at SemEval-2023 Task 10: Linguistic Feature Integration and Continuous Pretraining for Online Sexism Detection and Classification

Sexism has been prevalent online.In this paper, we explored the effect of explicit linguistic features and continuous pretraining on the performance of pretrained language models in sexism detection.While adding linguistic features did not improve the performance of the model, continuous pretraining did slightly boost the performance of the model in Task B from a mean macro-F1 1 score of 0.6156 to 0.6246.The best mean macro-F1 score in Task A was achieved by a finetuned HateBERT model using regular pretraining (0.8331).We observed that the linguistic features did not improve the model's performance.At the same time, continuous pretraining proved beneficial only for nuanced downstream tasks like Task-B.
Similarizing the Influence of Words with Contrastive Learning to Defend Word-level Adversarial Text Attack

Neural language models are vulnerable to word-level adversarial text attacks, which generate adversarial examples by directly substituting discrete input words.Previous search methods for word-level attacks assume that the information in the important words is more influential on prediction than unimportant words.In this paper, motivated by this assumption, we propose a self-supervised regularization method for Similarizing the Influence of Words with Contrastive Learning (SIWCon) that encourages the model to learn sentence representations in which words of varying importance have a more uniform influence on prediction.Experiments show that SIWCon is compatible with various training methods and effectively improves model robustness against various unforeseen adversarial attacks.The effectiveness of SIWCon is also intuitively shown through qualitative analysis and visualization of the loss landscape, sentence representation, and changes in model confidence.
How Well Apply Simple MLP to Incomplete Utterance Rewriting?

Incomplete utterance rewriting (IUR) aims to restore the incomplete utterance with sufficient context information for comprehension.This paper introduces a simple yet efficient IUR method.Different from prior studies, we first employ only one-layer MLP architecture to mine latent semantic information between joint utterances for IUR task (MIUR).After that, we conduct a joint feature matrix to predict the token type and thus restore the incomplete utterance.The well-designed network and simple architecture make our method significantly superior to existing methods in terms of quality and inference speed 1 .
Prompt Tuning Pushes Farther, Contrastive Learning Pulls Closer: A Two-Stage Approach to Mitigate Social Biases

As the representation capability of Pre-trained Language Models (PLMs) improve, there is growing concern that they will inherit social biases from unprocessed corpora.Most previous debiasing techniques used Counterfactual Data Augmentation (CDA) to balance the training corpus.However, CDA slightly modifies the original corpus, limiting the representation distance between different demographic groups to a narrow range.As a result, the debiasing model easily fits the differences between counterfactual pairs, which affects its debiasing performance with limited text resources.In this paper, we propose an adversarial training-inspired two-stage debiasing model using Contrastive learning with Continuous Prompt Augmentation (named CCPA) to mitigate social biases in PLMs' encoding.In the first stage, we propose a data augmentation method based on continuous prompt tuning to push farther the representation distance between sample pairs along different demographic groups.In the second stage, we utilize contrastive learning to pull closer the representation distance between the augmented sample pairs and then fine-tune PLMs' parameters to get debiased encoding.Our approach guides the model to achieve stronger debiasing performance by adding difficulty to the training process.Extensive experiments show that CCPA outperforms baselines in terms of debiasing performance.Meanwhile, experimental results on the GLUE benchmark show that CCPA retains the language modeling capability of PLMs.
Span Identification of Epistemic Stance-Taking in Academic Written English

Responding to the increasing need for automated writing evaluation (AWE) systems to assess language use beyond lexis and grammar (Burstein et al., 2016), we introduce a new approach to identify rhetorical features of stance in academic English writing. Drawing on the discourse-analytic framework of engagement in the Appraisal analysis (Martin & White, 2005), we manually annotated 4,688 sentences (126,411 tokens) for eight rhetorical stance categories (e.g., PROCLAIM, ATTRIBUTION) and additional discourse elements. We then report an experiment to train machine learning models to identify and categorize the spans of these stance expressions. The best-performing model (RoBERTa + LSTM) achieved macro-averaged F1 of .7208 in the span identification of stance-taking expressions, slightly outperforming the intercoder reliability estimates before adjudication (F1 = .6629).
CodeNLP at SemEval-2023 Task 2: Data Augmentation for Named Entity Recognition by Combination of Sequence Generation Strategies

In the article, we present the CodeNLP submission to the SemEval-2023 Task 2: Multi-CoNER II Multilingual Complex Named Entity Recognition.Our approach is based on data augmentation by combining various strategies of sequence generation for training.We show that the extended procedure of fine-tuning a pre-trained language model can bring improvements compared to any single strategy.On the development subsets the improvements were 1.7 pp and 3.1 pp of F-measure, for English and multilingual datasets, respectively.On the test subsets our models achieved 63.51% and 73.22% of Macro F1, respectively.
Recursion of Thought: A Divide-and-Conquer Approach to Multi-Context Reasoning with Language Models

Generating intermediate steps, or Chain of Thought (CoT), is an effective way to significantly improve language models' (LM) multistep reasoning capability.However, the CoT lengths can grow rapidly with the problem complexity, easily exceeding the maximum context size.Instead of increasing the context limit, which has already been heavily investigated, we explore an orthogonal direction: making LMs divide a problem into multiple contexts.We propose a new inference framework, called Recursion of Thought (RoT), which introduces several special tokens that the models can output to trigger context-related operations.Extensive experiments with multiple architectures including GPT-3 show that RoT dramatically improves LMs' inference capability to solve problems, whose solution consists of hundreds of thousands of tokens.
Counterfactual Probing for the Influence of Affect and Specificity on Intergroup Bias

While existing work on studying bias in NLP focues on negative or pejorative language use, Govindarajan et al. ( 2023) offer a revised framing of bias in terms of intergroup social context, and its effects on language behavior.In this paper, we investigate if two pragmatic features (specificity and affect) systematically vary in different intergroup contexts -thus connecting this new framing of bias to language output.Preliminary analysis finds modest correlations between specificity and affect of tweets with supervised intergroup relationship (IGR) labels.Counterfactual probing further reveals that while neural models finetuned for predicting IGR reliably use affect in classification, the model's usage of specificity is inconclusive.
AttenWalker: Unsupervised Long-Document Question Answering via Attention-based Graph Walking

Annotating long-document question answering (long-document QA) pairs is time-consuming and expensive.To alleviate the problem, it might be possible to generate long-document QA pairs via unsupervised question answering (UQA) methods.However, existing UQA tasks are based on short documents, and can hardly incorporate long-range information.To tackle the problem, we propose a new task, named unsupervised long-document question answering (ULQA), aiming to generate highquality long-document QA instances in an unsupervised manner.Besides, we propose Atten-Walker, a novel unsupervised method to aggregate and generate answers with long-range dependency so as to construct long-document QA pairs.Specifically, AttenWalker is composed of three modules, i.e., span collector, span linker and answer aggregator.Firstly, the span collector takes advantage of constituent parsing and reconstruction loss to select informative candidate spans for constructing answers.Secondly, by going through the attention graph of a pre-trained long-document model, potentially interrelated text spans (that might be far apart) could be linked together via an attentionwalking algorithm.Thirdly, in the answer aggregator, linked spans are aggregated into the final answer via the mask-filling ability of a pre-trained model.Extensive experiments show that AttenWalker outperforms previous methods on Qasper and NarrativeQA.In addition, AttenWalker also shows strong performance in the few-shot learning setting. 1
Mao-Zedong at SemEval-2023 Task 4: Label Represention Multi-Head Attention Model with Contrastive Learning-Enhanced Nearest Neighbor Mechanism for Multi-Label Text Classification

The study of human values is essential in both practical and theoretical domains.With the development of computational linguistics, the creation of large-scale datasets has made it possible to automatically recognize human values accurately.SemEval 2023 Task 4(Kiesel et al., 2023) provides a set of arguments and 20 types of human values that are implicitly expressed in each argument.In this paper, we present our team's solution.We use the Roberta(Liu et al.) model to obtain the word vector encoding of the document and propose a multi-head attention mechanism to establish connections between specific labels and semantic components.Furthermore, we use a contrastive learning-enhanced K-nearest neighbor mechanism(Su et al.) to leverage existing instance information for prediction.Our approach achieved an F1 score of 0.533 on the test set and ranked fourth on the leaderboard.we make our code publicly available at https://github.com/peterlau0626/semeval2023-task4-HumanValue.
clulab at MEDIQA-Chat 2023: Summarization and classification of medical dialogues

Clinical Natural Language Processing has been an increasingly popular research area in the NLP community.With the rise of large language models (LLMs) and their impressive abilities in NLP tasks, it is crucial to pay attention to their clinical applications.Sequence to sequence generative approaches with LLMs have been widely used in recent years.To be a part of the research in clinical NLP with recent advances in the field, we participated in task A of MEDIQA-Chat at ACL-ClinicalNLP Workshop 2023.In this paper, we explain our methods and findings as well as our comments on our results and limitations.
Post-Abstention: Towards Reliably Re-Attempting the Abstained Instances in QA

Despite remarkable progress made in natural language processing, even the state-of-the-art models often make incorrect predictions.Such predictions hamper the reliability of systems and limit their widespread adoption in realworld applications.Selective prediction partly addresses the above concern by enabling models to abstain from answering when their predictions are likely to be incorrect.While selective prediction is advantageous, it leaves us with a pertinent question 'what to do after abstention'.To this end, we present an explorative study on 'Post-Abstention', a task that allows re-attempting the abstained instances with the aim of increasing coverage of the system without significantly sacrificing its accuracy.We first provide mathematical formulation of this task and then explore several methods to solve it.Comprehensive experiments on 11 QA datasets show that these methods lead to considerable risk improvements -performance metric of the Post-Abstention task-both in the in-domain and the out-of-domain settings.We also conduct a thorough analysis of these results which further leads to several interesting findings.Finally, we believe that our work will encourage and facilitate further research in this important area of addressing the reliability of NLP systems.
Can LMs Store and Retrieve 1-to-N Relational Knowledge?

It has been suggested that pretrained language models can be viewed as knowledge bases.One of the prerequisites for using language models as knowledge bases is how accurately they can store and retrieve world knowledge.It is already revealed that language models can store much 1-to-1 relational knowledge, such as "country and its capital," with high memorization accuracy.On the other hand, world knowledge includes not only 1-to-1 but also 1to-N relational knowledge, such as "parent and children."However, it is not clear how accurately language models can handle 1-to-N relational knowledge.To investigate language models' abilities toward 1-to-N relational knowledge, we start by designing the problem settings.Specifically, we organize the character of 1-to-N relational knowledge and define two essential skills: (i) memorizing multiple objects individually and (ii) retrieving multiple stored objects without excesses or deficiencies at once.We inspect LMs' ability to handle 1-to-N relational knowledge on the controlled synthesized data.As a result, we report that it is possible to memorize multiple objects with high accuracy, but generalizing the retrieval ability (expressly, enumeration) is challenging.
Investigating the Saliency of Sentiment Expressions in Aspect-Based Sentiment Analysis

We examine the behaviour of an aspect-based sentiment classifier built by fine-tuning the BERT BASE model on the SemEval 2016 English dataset. In a set of masking experiments, we examine the extent to which the tokens identified as salient by LIME and a gradient-based method are being used by the classifier. We find that both methods are able to produce faithful rationales, with LIME outperforming the gradient-based method. We also identify a set of manually annotated sentiment expressions for this dataset, and carry out more masking experiments with these as human rationales. The enhanced performance of a classifier that only sees the relevant sentiment expressions suggests that they are not being used to their full potential. A comparison of the LIME and gradient rationales with the sentiment expressions reveals only a moderate level of agreement. Some disagreements are related to the fixed length of the rationales and the tendency of the rationales to contain content words related to the aspect itself.
Injecting Comparison Skills in Task-Oriented Dialogue Systems for Database Search Results Disambiguation

In task-oriented dialogue (TOD) systems designed to aid users accomplish specific goals in one or more domains, the agent retrieves entities that satisfy user constraints from the database.However, when multiple database search results exist, an ambiguity occurs regarding which results to select and present to the user.Existing TOD systems handle this ambiguity by randomly selecting one or few results and presenting their names to the user.However, in a real scenario, users do not always accept a randomly recommended entity, and users should have access to more comprehensive information about the search results.To address this limitation, we propose a novel task called Comparison-Based database search Ambiguity handling (CBA), which handles ambiguity in database search results by comparing the properties of multiple entities to enable users to choose according to their preferences.Accordingly, we introduce a new framework for automatically collecting high-quality dialogue data along with the Disambiguating Schemaguided Dialogue (DSD) dataset, an augmented version of the SGD dataset.Experimental studies on the DSD dataset demonstrate that training baseline models with the dataset effectively address the CBA task.Our dataset and code will be publicized.
Z-ICL: Zero-Shot In-Context Learning with Pseudo-Demonstrations

Although large language models can be prompted for both zero-and few-shot learning, performance drops significantly when no demonstrations are available.In this paper, we introduce Z-ICL, a new zero-shot method that closes the gap by constructing pseudo-demonstrations for a given test input using a raw text corpus.Concretely, pseudodemonstrations are constructed by (1) finding the nearest neighbors to the test input from the corpus and pairing them with random task labels, and (2) applying a set of techniques to reduce the amount of direct copying the model does from the resulting demonstrations.Evaluation on nine classification datasets shows that Z-ICL outperforms previous zero-shot methods by a significant margin, and is on par with incontext learning with few-shot labeled training data.Overall, Z-ICL provides a significantly higher estimate of the zero-shot performance levels of a model, and supports future efforts to develop better pseudo-demonstrations that further improve zero-shot results. 1
FMI-SU at SemEval-2023 Task 7: Two-level Entailment Classification of Clinical Trials Enhanced by Contextual Data Augmentation

The paper presents an approach for solving Se-mEval 2023 Task 7 -identifying the inference relation in a clinical trials dataset.The system has two levels for retrieving relevant clinical trial evidence for a statement and then classifying the inference relation based on the relevant sentences.In the first level, the system classifies the evidence-statement pairs as relevant or not using a BERT-based classifier and contextual data augmentation (subtask 2).Using the relevant parts of the clinical trial from the first level, the system uses an additional BERTbased classifier to determine whether the relation is entailment or contradiction (subtask 1).In both levels, the contextual data augmentation is showing a significant improvement in the F1 score on the test set of 3.7% for subtask 2 and 7.6% for subtask 1, achieving final F1 scores of 82.7% for subtask 2 and 64.4% for subtask 1.
SIMMC-VR: A Task-oriented Multimodal Dialog Dataset with Situated and Immersive VR Streams

Building an AI assistant that can seamlessly converse and instruct humans, in a user-centric situated scenario, requires several essential abilities: (1) spatial and temporal understanding of the situated and real-time user scenes, (2) capability of grounding the actively perceived visuals of users to conversation contexts, and (3) conversational reasoning over past utterances to perform just-in-time assistance.However, we currently lack a large-scale benchmark that captures user↔assistant interactions with all of the aforementioned features.To this end, we propose SIMMC-VR, an extension of the SIMMC 2.0 dataset to a video-grounded taskoriented dialog dataset that captures real-world AI-assisted user scenarios in VR.We propose a novel data collection paradigm that involves (1) generating object-centric multimodal dialog flows with egocentric visual streams and visually-grounded templates, and (2) manually paraphrasing the simulated dialogs for naturalness and diversity while preserving multimodal dependencies.To measure meaningful progress in the field, we propose four tasks to address the new challenges in SIMMC-VR, which require complex spatial-temporal dialog reasoning in active egocentric scenes.We benchmark the proposed tasks with strong multimodal models, and highlight the key capabilities that current models lack for future research directions.