Reviewerly/exHarmony huggingface dataset:
Improving the robustness of NLI models with minimax training

Natural language inference (NLI) models are susceptible to learning shortcuts, i.e. decision rules that spuriously correlate with the label.As a result, they achieve high in-distribution performance, but fail to generalize to out-ofdistribution samples where such correlations do not hold.In this paper, we present a training method to reduce the reliance of NLI models on shortcuts and improve their out-of-distribution performance without assuming prior knowledge of the shortcuts being targeted.To this end, we propose a minimax objective between a learner model being trained for the NLI task, and an auxiliary model aiming to maximize the learner's loss by up-weighting examples from regions of the input space where the learner incurs high losses.This process incentivizes the learner to focus on under-represented "hard" examples with patterns that contradict the shortcuts learned from the prevailing "easy" examples.Experimental results on three NLI datasets demonstrate that our method consistently outperforms other robustness enhancing techniques on out-of-distribution adversarial test sets, while maintaining high in-distribution accuracy.
Feature Interactions Reveal Linguistic Structure in Language Models

We study feature interactions in the context of feature attribution methods for post-hoc interpretability. In interpretability research, getting to grips with feature interactions is increasingly recognised as an important challenge, because interacting features are key to the success of neural networks. Feature interactions allow a model to build up hierarchical representations for its input, and might provide an ideal starting point for the investigation into linguistic structure in language models.However, uncovering the exact role that these interactions play is also difficult, and a diverse range of interaction attribution methods has been proposed.In this paper, we focus on the question which of these methods most faithfully reflects the inner workings of the target models.We work out a grey box methodology, in which we train models to perfection on a formal language classification task, using PCFGs. We show that under specific configurations, some methods are indeed able to uncover the grammatical rules acquired by a model. Based on these findings we extend our evaluation to a case study on language models, providing novel insights into the linguistic structure that these models have acquired.
Improving End-to-End Speech Translation by Imitation-Based Knowledge Distillation with Synthetic Transcripts

End-to-end automatic speech translation (AST) relies on data that combines audio inputs with text translation outputs.Previous work used existing large parallel corpora of transcriptions and translations in a knowledge distillation (KD) setup to distill a neural machine translation (NMT) into an AST student model.While KD allows using larger pretrained models, the reliance of previous KD approaches on manual audio transcripts in the data pipeline restricts the applicability of this framework to AST.We present an imitation learning approach where a teacher NMT system corrects the errors of an AST student without relying on manual transcripts.We show that the NMT teacher can recover from errors in automatic transcriptions and is able to correct erroneous translations of the AST student, leading to improvements of about 4 BLEU points over the standard AST end-to-end baseline on the English-German CoVoST-2 and MuST-C datasets, respectively.Code and data are publicly available.
Triplet-Free Knowledge-Guided Response Generation

Generating vivid and informative responses (e.g., comments for social posts and utterances for dialogues) is challenging without giving relevant knowledge.Prior works focus on constructing the "latent" knowledge first and then learning how to "ground" it based on pseudo (context, knowledge, response) triplets.However, the retrieval between real responses and their latent knowledge is difficult in nature.In this paper, instead of focusing on how to ground knowledge given the responses, we take a different perspective to optimize the final responses for given guided knowledge directly.This allows us to re-formulate the entire problem in a simplified yet more scalable way.Specifically, we pretrain a response language model (LM) to measure the relevance and consistency between any context and response, then use search engines to collect the top-ranked passages to serve as the guiding knowledge without explicitly optimizing the "best" latent knowledge that corresponds to a given response.The final response generation model is trained through reinforcement learning by taking both the response LM prior and knowledge-injection rate as rewards.For better evaluations, we construct a new Chinese benchmark, "IceKC", using fresh multimodal online social posts.Both automatic evaluations and human evaluations show our zero-resource approach performs significantly better than prior works.
Towards Benchmarking and Improving the Temporal Reasoning Capability of Large Language Models

Reasoning about time is of fundamental importance.Many facts are time-dependent.For example, athletes change teams from time to time, and different government officials are elected periodically.Previous time-dependent question answering (QA) datasets tend to be biased in either their coverage of time spans or question types.In this paper, we introduce a comprehensive probing dataset TEMPREASON to evaluate the temporal reasoning capability of large language models.Our dataset includes questions of three temporal reasoning levels.In addition, we also propose a novel learning framework to improve the temporal reasoning capability of large language models, based on temporal span extraction and time-sensitive reinforcement learning.We conducted experiments in closed book QA, open book QA, and reasoning QA settings and demonstrated the effectiveness of our approach 1 .
Lexicools at SemEval-2023 Task 10: Sexism Lexicon Construction via XAI

This paper presents our work on the SemEval-2023 Task 10 Explainable Detection of Online Sexism (EDOS) (Kirk et al., 2023) using lexicon-based models.Our approach consists of three main steps: lexicon construction based on Pointwise Mutual Information (PMI) and Shapley value, lexicon augmentation using an unannotated corpus and Large Language Models (LLMs), and, lastly, lexical incorporation for Bag-of-Word (BoW) logistic regression and fine-tuning LLMs.Our results demonstrate that our Shapley approach effectively produces a high-quality lexicon.We also show that simply counting the presence of certain words in our lexicons and comparing the count can outperform a BoW logistic regression in task B/C and fine-tuning BERT in task C. In the end, our classifier achieved F1-scores of 53.34% and 27.31% on the official blind test sets for tasks B and C, respectively.We, additionally, provide an in-depth analysis highlighting model limitations and bias.We also present our attempts to understand the model's behavior based on our constructed lexicons.Our code and the resulting lexicons are open-sourced in
Improving Knowledge Production Efficiency With Question Answering on Conversation

Through an online customer service application, we have collected many conversations between customer service agents and customers. Building a knowledge production system can help reduce the labor cost of maintaining the FAQ database for the customer service chatbot, whose core module is question answering (QA) on these conversations. However, most existing researches focus on document-based QA tasks, and there is a lack of researches on conversation-based QA and related datasets, especially in Chinese language. The challenges of conversation-based QA include: 1) answers may be scattered among multiple dialogue turns; 2) understanding complex dialogue contexts is more complicated than documents. To address these challenges, we propose a multi-span extraction model on this task and introduce continual pre-training and multi-task learning schemes to further improve model performance. To validate our approach, we construct two Chinese datasets using dialogues as the knowledge source, namely cs-qaconv and kd-qaconv, respectively. Experimental results demonstrate that the proposed model outperforms the baseline on both datasets. The online application also verifies the effectiveness of our method. The dataset kd-qaconv will be released publicly for research purposes.
FiRC at SemEval-2023 Task 10: Fine-grained Classification of Online Sexism Content Using DeBERTa

SemEval 2023 shared task 10 "Explainable Detection of Online Sexism" focuses on detecting and identifying comments and tweets containing sexist expressions and also explaining why it is sexist.This paper describes the system that we used to participate in this shared task.Our model is an ensemble of different variants of fine-tuned DeBERTa models that employs a k-fold cross-validation.We have participated in the three tasks A, B and C. Our model ranked 2 nd position in tasks A, 7 th in task B and 4 th in task C.
Discrete Prompt Optimization via Constrained Generation for Zero-shot Re-ranker

Re-rankers, which order retrieved documents with respect to the relevance score on the given query, have gained attention for the information retrieval (IR) task.Rather than fine-tuning the pre-trained language model (PLM), the large-scale language model (LLM) is utilized as a zero-shot re-ranker with excellent results.While LLM is highly dependent on the prompts, the impact and the optimization of the prompts for the zero-shot re-ranker are not explored yet.Along with highlighting the impact of optimization on the zero-shot re-ranker, we propose a novel discrete prompt optimization method, Constrained Prompt generation (Co-Prompt), with the metric estimating the optimum for reranking.Co-Prompt guides the generated texts from PLM toward optimal prompts based on the metric without parameter update.The experimental results demonstrate that Co-Prompt leads to outstanding re-ranking performance against the baselines.Also, Co-Prompt generates more interpretable prompts for humans against other prompt optimization methods.
Infusing Hierarchical Guidance into Prompt Tuning: A Parameter-Efficient Framework for Multi-level Implicit Discourse Relation Recognition

Multi-level implicit discourse relation recognition (MIDRR) aims at identifying hierarchical discourse relations among arguments.Previous methods achieve the promotion through finetuning PLMs.However, due to the data scarcity and the task gap, the pre-trained feature space cannot be accurately tuned to the task-specific space, which even aggravates the collapse of the vanilla space.Besides, the comprehension of hierarchical semantics for MIDRR makes the conversion much harder.In this paper, we propose a prompt-based Parameter-Efficient Multilevel IDRR (PEMI) framework to solve the above problems.First, we leverage parameterefficient prompt tuning to drive the inputted arguments to match the pre-trained space and realize the approximation with few parameters.Furthermore, we propose a hierarchical label refining (HLR) method for the prompt verbalizer to deeply integrate hierarchical guidance into the prompt tuning.Finally, our model achieves comparable results on PDTB 2.0 and 3.0 using about 0.1% trainable parameters compared with baselines and the visualization demonstrates the effectiveness of our HLR method.* The Corresponding author.Arg1: After the race, Fortune 500 executives drooled like schoolboys over the cars and drivers.Arg2: No dummies, the drivers pointed out they still had space on their machines for another sponsor's name or two.
FTD at SemEval-2023 Task 3: News Genre and Propaganda Detection by Comparing Mono- and Multilingual Models with Fine-tuning on Additional Data

We report our participation in the SemEval-2023 shared task on propaganda detection and describe our solutions with pre-trained models and their ensembles.For Subtask 1 (News Genre Categorisation), we report the impact of several settings, such as the choice of the classification models (monolingual or multilingual or their ensembles), the choice of the training sets (base or additional sources), the impact of detection certainty in making a classification decision as well as the impact of other hyperparameters.In particular, we fine-tune models on additional data for other genre classification tasks, such as FTD.We also try adding texts from genre-homogenous corpora, such as Panorama, Babylon Bee for satire and Giganews for for reporting texts.We also make prepared models for Subtasks 2 and 3 with finetuning the corresponding models first for Subtask 1.The code needed to reproduce the experiments is available. 1
Causal interventions expose implicit situation models for commonsense language understanding

Accounts of human language processing have long appealed to implicit "situation models" that enrich comprehension with relevant but unstated world knowledge.Here, we apply causal intervention techniques to recent transformer models to analyze performance on the Winograd Schema Challenge (WSC), where a single context cue shifts interpretation of an ambiguous pronoun.We identify a relatively small circuit of attention heads that are responsible for propagating information from the context word that guides which of the candidate noun phrases the pronoun ultimately attends to.We then compare how this circuit behaves in a closely matched "syntactic" control where the situation model is not strictly necessary.These analyses suggest distinct pathways through which implicit situation models are constructed to guide pronoun resolution 1 .
Contrastive Learning with Generated Representations for Inductive Knowledge Graph Embedding

With the evolution of Knowledge Graphs (KGs), new entities emerge which are not seen before.Representation learning of KGs in such an inductive setting aims to capture and transfer the structural patterns from existing entities to new entities.However, the performance of existing methods in inductive KGs are limited by sparsity and implicit transfer.In this paper, we propose VMCL, a Contrastive Learning (CL) framework with graph guided Variational autoencoder on Meta-KGs in the inductive setting.We first propose representation generation to capture the encoded and generated representations of entities, where the generated variations can augment the representation space with complementary features.Then, we design two CL objectives that work across entities and meta-KGs to simulate the transfer mode.With extensive experiments we demonstrate that our proposed VMCL can significantly outperform previous state-of-the-art baselines.
DMLM: Descriptive Masked Language Modeling

Over the last few years, Masked Language Modeling (MLM) pre-training has resulted in remarkable advancements in many Natural Language Understanding (NLU) tasks, which has sparked an interest in researching alternatives and extensions to the MLM objective.In this paper, we tackle the absence of explicit semantic grounding in MLM and propose Descriptive Masked Language Modeling (DMLM), a knowledge-enhanced reading comprehension objective, where the model is required to predict the most likely word in a context, being provided with the word's definition.For instance, given the sentence "I was going to the _", if we provided as definition "financial institution", the model would have to predict the word "bank"; if, instead, we provided "sandy seashore", the model should predict "beach".Our evaluation highlights the effectiveness of DMLM in comparison with standard MLM, showing improvements on a number of well-established NLU benchmarks, as well as other semantics-focused tasks, e.g., Semantic Role Labeling.Furthermore, we also demonstrate how it is possible to take full advantage of DMLM to embed explicit semantics in downstream tasks, explore several properties of DMLM-based contextual representations and suggest a number of future directions to investigate.
Regression-Free Model Updates for Spoken Language Understanding

Andrea Caciolai, Verena Weber, Tobias Falke, Alessandro Pedrani, Davide Bernardi. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track). 2023.
No Strong Feelings One Way or Another: Re-operationalizing Neutrality in Natural Language Inference

Natural Language Inference (NLI) has been a cornerstone task in evaluating language models’ inferential reasoning capabilities. However, the standard three-way classification scheme used in NLI has well-known shortcomings in evaluating models’ ability to capture the nuances of natural human reasoning. In this paper, we argue that the operationalization of the neutral label in current NLI datasets has low validity, is interpreted inconsistently, and that at least one important sense of neutrality is often ignored. We uncover the detrimental impact of these shortcomings, which in some cases leads to annotation datasets that actually decrease performance on downstream tasks. We compare approaches of handling annotator disagreement and identify flaws in a recent NLI dataset that designs an annotator study based on a problematic operationalization. Our findings highlight the need for a more refined evaluation framework for NLI, and we hope to spark further discussion and action in the NLP community.
Unsupervised Summarization Re-ranking

With the rise of task-specific pre-training objectives, abstractive summarization models like PEGASUS offer appealing zero-shot performance on downstream summarization tasks.However, the performance of such unsupervised models still lags significantly behind their supervised counterparts.Similarly to the supervised setup, we notice a very high variance in quality among summary candidates from these models while only one candidate is kept as the summary output.In this paper, we propose to re-rank summary candidates in an unsupervised manner, aiming to close the performance gap between unsupervised and supervised models.Our approach improves the unsupervised PEGASUS by up to 7.27% and ChatGPT by up to 6.86% relative mean ROUGE across four widely-adopted summarization benchmarks ; and achieves relative gains of 7.51% (up to 23.73% from XSum to WikiHow) averaged over 30 zero-shot transfer setups (finetuning on a dataset, evaluating on another).
Faithful Question Answering with Monte-Carlo Planning

Although large language models demonstrate remarkable question-answering performances, revealing the intermediate reasoning steps that the models faithfully follow remains challenging.In this paper, we propose FAME (FAithful question answering with MontE-carlo planning) to answer questions based on faithful reasoning steps.The reasoning steps are organized as a structured entailment tree, which shows how premises are used to produce intermediate conclusions that can prove the correctness of the answer.We formulate the task as a discrete decision-making problem and solve it through the interaction of a reasoning environment and a controller.The environment is modular and contains several basic task-oriented modules, while the controller proposes actions to assemble the modules.Since the search space could be large, we introduce a Monte-Carlo planning algorithm to do a look-ahead search and select actions that will eventually lead to highquality steps.FAME achieves advanced performance on the standard benchmark.It can produce valid and faithful reasoning steps compared with large language models with a much smaller model size.
Your spouse needs professional help: Determining the Contextual Appropriateness of Messages through Modeling Social Relationships

Understanding interpersonal communication requires, in part, understanding the social context and norms in which a message is said.However, current methods for identifying offensive content in such communication largely operate independent of context, with only a few approaches considering community norms or prior conversation as context.Here, we introduce a new approach to identifying inappropriate communication by explicitly modeling the social relationship between the individuals.We introduce a new dataset of contextually-situated judgments of appropriateness and show that large language models can readily incorporate relationship information to accurately identify appropriateness in a given context.Using data from online conversations and movie dialogues, we provide insight into how the relationships themselves function as implicit norms and quantify the degree to which context-sensitivity is needed in different conversation settings.Further, we also demonstrate that contextualappropriateness judgments are predictive of other social factors expressed in language such as condescension and politeness.
KWJA: A Unified Japanese Analyzer Based on Foundation Models

We present KWJA, a high-performance unified Japanese text analyzer based on foundation models.KWJA supports a wide range of tasks, including typo correction, word segmentation, word normalization, morphological analysis, named entity recognition, linguistic feature tagging, dependency parsing, PAS analysis, bridging reference resolution, coreference resolution, and discourse relation analysis, making it the most versatile among existing Japanese text analyzers.KWJA solves these tasks in a multi-task manner but still achieves competitive or better performance compared to existing analyzers specialized for each task.
ExplainMeetSum: A Dataset for Explainable Meeting Summarization Aligned with Human Intent

To enhance the explainability of meeting summarization, we construct a new dataset called "ExplainMeetSum," an augmented version of QMSum, by newly annotating evidence sentences that faithfully "explain" a summary.Using ExplainMeetSum, we propose a novel multiple extractor guided summarization, namely Multi-DYLE, which extensively generalizes DYLE to enable using a supervised extractor based on human-aligned extractive oracles.We further present an explainabilityaware task, named "Explainable Evidence Extraction" (E3), which aims to automatically detect all evidence sentences that support a given summary.Experimental results on the QMSum dataset show that the proposed Multi-DYLE outperforms DYLE with gains of up to 3.13 in the ROUGE-1 score.We further present the initial results on the E3 task, under the settings using separate and joint evaluation metrics. 1
UM6P at SemEval-2023 Task 3: News genre classification based on transformers, graph convolution networks and number of sentences

This paper presents our proposed approach for English document genre classification in the context of SemEval-2023 Task 3, Subtask 1.Our method uses an ensemble technique to combine four distinct model predictions: Longformer, RoBERTa, GCN, and a sentence number-based model.Each model is optimized on simple and easy-to-understand objectives.We provide snippets of code that define each model to make the reading experience better.Our method ranked 12th in document genre (subtask 1) classification for English texts.
Chride at SemEval-2023 Task 10: Fine-tuned Deberta-V3 on Detection of Online Sexism with Hierarchical Loss

Sexism is one of the most concerning problems in the internet society.By detecting sexist expressions, we can reduce the offense toward females and provide useful information to understand how sexism occurs.Our work focuses on a newly-published dataset, EDOS, which annotates English sexist expressions from Reddit and categorizes their specific types.Our method is to train a DeBERTa-V3 classifier with all three kinds of labels provided by the dataset, including sexist, category, and granular vectors.Our classifier predicts the probability distribution on vector labels and further applies it to represent category and sexist distributions.Our classifier uses its label and finer-grained labels for each classification to calculate the hierarchical loss for optimization.Our experiments and analyses show that using a combination of loss with finer-grained labels generally achieves better performance on sexism detection and categorization.Codes for our implementation can be found at https://github.com/KomeijiForce/SemEval2023_Task10.