Reviewerly/exHarmony huggingface dataset:
On Improving Summarization Factual Consistency from Natural Language Feedback

Despite the recent progress in language generation models, their outputs may not always meet user expectations.In this work, we study whether informational feedback in natural language can be leveraged to improve generation quality and user preference alignment.To this end, we consider factual consistency in summarization, the quality that the summary should only contain information supported by the input documents, as the user-expected preference.We collect a high-quality dataset, DeFacto, containing human demonstrations and informational natural language feedback consisting of corrective instructions, edited summaries, and explanations with respect to the factual consistency of the summary.Using our dataset, we study three natural language generation tasks:(1) editing a summary by following the human feedback, (2) generating human feedback for editing the original summary, and (3) revising the initial summary to correct factual errors by generating both the human feedback and edited summary.We show that DeFacto can provide factually consistent human-edited summaries and further insights into summarization factual consistency thanks to its informational natural language feedback.We further demonstrate that fine-tuned language models can leverage our dataset to improve the summary factual consistency, while large language models lack the zero-shot learning ability in our proposed tasks that require controllable text generation.
Better Simultaneous Translation with Monotonic Knowledge Distillation

Simultaneous machine translation (SiMT) presents a unique challenge as it requires generating target tokens before the source sentence is fully consumed.This can lead to the hallucination problem, where target tokens are generated without support from the source sentence.The prefix-to-prefix training data used to train SiMT models are not always parallel, due to divergent word order between the source and target languages, and can contribute to the problem.In this paper, we propose a novel approach that leverages traditional translation models as teachers and employs a two-stage beam search algorithm to generate monotonic yet accurate reference translations for sequence-level knowledge distillation.Experimental results demonstrate the significant improvements achieved by our approach over multiple strong SiMT baselines, leading to new state-of-the-art performance across various language pairs.Notably, when evaluated on a monotonic version of the WMT15 De→En test set, which includes references generated in a more monotonic style by professional translators, our approach achieves even more substantial improvement over the baselines.The source code and data are publicly available for further exploration 1 .
UniEvent: Unified Generative Model with Multi-Dimensional Prefix for Zero-Shot Event-Relational Reasoning

Reasoning about events and their relations attracts surging research efforts since it is regarded as an indispensable ability to fulfill various event-centric or common-sense reasoning tasks.However, these tasks often suffer from limited data availability due to the laborintensive nature of their annotations.Consequently, recent studies have explored knowledge transfer approaches within a multi-task learning framework to address this challenge.Although such methods have achieved acceptable results, such brute-force solutions struggle to effectively transfer event-relational knowledge due to the vast array of inter-event relations (e.g.temporal, causal, conditional) and reasoning formulations (e.g.discriminative, abductive, ending prediction).To enhance knowledge transfer and enable zero-shot generalization among various combinations, in this work we propose a novel unified framework, called UNIEVENT.Inspired by prefix-based multitask learning, our approach organizes event relational reasoning tasks into a coordinate system with multiple axes, representing inter-event relations and reasoning formulations.We then train a unified text-to-text generative model that utilizes coordinate-assigning prefixes for each task.By leveraging our adapted prefixes, our unified model achieves state-of-the-art or competitive performance on both zero-shot and supervised reasoning tasks, as demonstrated in extensive experiments.
Grounding the Lexical Substitution Task in Entailment

Existing definitions of lexical substitutes are often vague or inconsistent with the gold annotations.We propose a new definition which is grounded in the relation of entailment; namely, that the sentence that results from the substitution should be in the relation of mutual entailment with the original sentence.We argue that the new definition is well-founded and supported by previous work on lexical entailment.We empirically validate our definition by verifying that it covers the majority of gold substitutes in existing datasets.Based on this definition, we create a new dataset from existing semantic resources.Finally, we propose a novel context augmentation method motivated by the definition, which relates the substitutes to the sense of the target word by incorporating glosses and synonyms directly into the context.Experimental results demonstrate that our augmentation approach improves the performance of lexical substitution systems on the existing benchmarks.Context Ct Substitute w Cw |= Ct Ct |= Cw Meaning preserved I am glad to be out of the favor-trading scene for half a minute moment No No No It didn't seem like we had a lot of holes to fill.It's good, it gives us something we didn't have and we didn't lose much.award No Yes No Walking out of the church , a little gust of cold air caught me by surprise.icy No Yes No "It's a long way
LMCap: Few-shot Multilingual Image Captioning by Retrieval Augmented Language Model Prompting

Multilingual image captioning has recently been tackled by training with large-scale machine translated data, which is an expensive, noisy, and time-consuming process.Without requiring any multilingual caption data, we propose LMCAP, an image-blind few-shot multilingual captioning model that works by prompting a language model with retrieved captions.Specifically, instead of following the standard encoder-decoder paradigm, given an image, LMCAP first retrieves the captions of similar images using a multilingual CLIP encoder.These captions are then combined into a prompt for an XGLM decoder, in order to generate captions in the desired language.In other words, the generation model does not directly process the image, instead processing retrieved captions.Experiments on the XM3600 dataset of geographically diverse images show that our model is competitive with fully-supervised multilingual captioning models, without requiring any supervised training on any captioning data.
Modeling Instance Interactions for Joint Information Extraction with Neural High-Order Conditional Random Field

Prior works on joint Information Extraction (IE) typically model instance (e.g., event triggers, entities, roles, relations) interactions by representation enhancement, type dependencies scoring, or global decoding.We find that the previous models generally consider binary type dependency scoring of a pair of instances, and leverage local search such as beam search to approximate global solutions.To better integrate cross-instance interactions, in this work, we introduce a joint IE framework (CRFIE) that formulates joint IE as a high-order Conditional Random Field.Specifically, we design binary factors and ternary factors to directly model interactions between not only a pair of instances but also triplets.Then, these factors are utilized to jointly predict labels of all instances.To address the intractability problem of exact high-order inference, we incorporate a high-order neural decoder that is unfolded from a mean-field variational inference method, which achieves consistent learning and inference.The experimental results show that our approach achieves consistent improvements on three IE tasks compared with our baseline and prior work.
UMUTeam and SINAI at SemEval-2023 Task 9: Multilingual Tweet Intimacy Analysis using Multilingual Large Language Models and Data Augmentation

This work presents the participation of the UMUTeam and the SINAI research groups in the SemEval-2023 Task 9: Multilingual Tweet Intimacy Analysis.The goal of this task is to predict the intimacy of a set of tweets in 10 languages: English, Spanish, Italian, Portuguese, French, Chinese, Hindi, Arabic, Dutch and Korean, of which, the last 4 are not in the training data.Our approach to address this task is based on data augmentation and the use of three multilingual Large Language Models (multilingual BERT, XLM and mDeBERTA) by ensemble learning.Our team ranked 30th out of 45 participants.Our best results were achieved with two unseen languages: Korean (16th) and Hindi (19th).
From Adversarial Arms Race to Model-centric Evaluation: Motivating a Unified Automatic Robustness Evaluation Framework

Yangyi Chen, Hongcheng Gao, Ganqu Cui, Lifan Yuan, Dehan Kong, Hanlu Wu, Ning Shi, Bo Yuan, Longtao Huang, Hui Xue, Zhiyuan Liu, Maosong Sun, Heng Ji. Findings of the Association for Computational Linguistics: ACL 2023. 2023.
Measuring Progress in Fine-grained Vision-and-Language Understanding

While pretraining on large-scale image-text data from the Web has facilitated rapid progress on many vision-and-language (V&L) tasks, recent work has demonstrated that pretrained models lack "fine-grained" understanding, such as the ability to recognise relationships, verbs, and numbers in images.This has resulted in an increased interest in the community to either develop new benchmarks or models for such capabilities.To better understand and quantify progress in this direction, we investigate four competitive V&L models on four fine-grained benchmarks.Through our analysis, we find that X-VLM (Zeng et al., 2022) consistently outperforms other baselines, and that modelling innovations can impact performance more than scaling Web data, which even degrades performance sometimes.Through a deeper investigation of X-VLM, we highlight the importance of both novel losses and rich data sources for learning fine-grained skills.Finally, we inspect training dynamics, and discover that for some tasks, performance peaks early in training or significantly fluctuates, never converging.
LEDA: a Large-Organization Email-Based Decision-Dialogue-Act Analysis Dataset

Collaboration increasingly happens online.This is especially true for large groups working on global tasks, with collaborators all around the world.The size and distributed nature of such groups make decision-making challenging.This paper proposes a set of dialog acts for the study of decision-making mechanisms in such groups, and provides a new annotated dataset based on real-world data from the public mail-archives of one such organization -the Internet Engineering Task Force (IETF).We provide an initial data analysis showing that this dataset can be used to better understand decision-making in such organizations.Finally, we experiment with a preliminary transformerbased dialog act tagging model.
IXA at SemEval-2023 Task 2: Baseline Xlm-Roberta-base Approach

IXA proposes a Sequence labeling fine-tune approach, which consists of a lightweight baseline (10e), the system takes advantage of transfer learning from pre-trained Named Entity Recognition and cross-lingual knowledge from the LM checkpoint.This technique obtains a drastic reduction in the effective training costs that works as a perfect baseline, future improvements in the baseline approach could fit: 1) Domain adequation, 2) Data augmentation, and 3) Intermediate task learning.
Enhancing Document-level Event Argument Extraction with Contextual Clues and Role Relevance

Document-level event argument extraction poses new challenges of long input and crosssentence inference compared to its sentencelevel counterpart.However, most prior works focus on capturing the relations between candidate arguments and the event trigger in each event, ignoring two crucial points: a) nonargument contextual clue information; b) the relevance among argument roles.In this paper, we propose a SCPRG (Span-triggerbased Contextual Pooling and latent Role Guidance) model, which contains two novel and effective modules for the above problem.The Span-Trigger-based Contextual Pooling (STCP) adaptively selects and aggregates the information of non-argument clue words based on the context attention weights of specific argument-trigger pairs from pre-trained model.The Role-based Latent Information Guidance (RLIG) module constructs latent role representations, makes them interact through roleinteractive encoding to capture semantic relevance, and merges them into candidate arguments.Both STCP and RLIG introduce no more than 1% new parameters compared with the base model and can be easily applied to other event extraction models, which are compact and transplantable.Experiments on two public datasets show that our SCPRG outperforms previous state-of-the-art methods, with 1.13 F1 and 2.64 F1 improvements on RAMS and WikiEvents respectively.Further analyses illustrate the interpretability of our model.
Data Curation Alone Can Stabilize In-context Learning

In-context learning (ICL) enables large language models (LLMs) to perform new tasks by prompting them with a sequence of training examples.However, it is known that ICL is very sensitive to the choice of training examples: randomly sampling examples from a training set leads to high variance in performance.In this paper, we show that carefully curating a subset of training data greatly stabilizes ICL performance without any other changes to the ICL algorithm (e.g., prompt retrieval or calibration).We introduce two methods to choose training subsets-both score training examples individually, then select the highest-scoring ones.CONDACC scores a training example by its average dev-set ICL accuracy when combined with random training examples, while DATAMODELS learns linear regressors that estimate how the presence of each training example influences LLM outputs.Across five tasks and two LLMs, sampling from stable subsets selected by CONDACC and DATAMODELS improves average accuracy over sampling from the entire training set by 7.7% and 6.3%, respectively.Surprisingly, the stable subset examples are not especially diverse in content or low in perplexity, in contrast with other work suggesting that diversity and perplexity are important when prompting LLMs.
The CRINGE Loss: Learning what language not to model

Standard language model training employs gold human documents or human-human interaction data, and treats all training data as positive examples.Growing evidence shows that even with very large amounts of positive training data, issues remain that can be alleviated with relatively small amounts of negative data -examples of what the model should not do.In this work, we propose a novel procedure to train with such data called the CRINGE loss (ContRastive Iterative Negative GEneration).We show the effectiveness of this approach across three different experiments on the tasks of safe generation, contradiction avoidance, and open-domain dialogue.Our models outperform multiple strong baselines and are conceptually simple, easy to train and implement.
Iterative Nearest Neighbour Machine Translation for Unsupervised Domain Adaptation

Unsupervised domain adaptation of machine translation, which adapts a pre-trained translation model to a specific domain without indomain parallel data, has drawn extensive attention in recent years.However, most existing methods focus on the fine-tuning based techniques, which is non-extensible.In this paper, we propose a new method to perform unsupervised domain adaptation in a non-parametric manner.Employing only indomain monolingual data, this method jointly perform nearest neighbour inference on both forward and backward translation directions.The forward translation model creates nearest neighbour datastore for the backward direction, and vice versa, strengthening each other in an iterative style.Experiments on multidomain datasets demonstrate that our method significantly improves the in-domain translation performance and achieves state-of-the-art results among non-parametric methods.
Align-then-Enhance: Multilingual Entailment Graph Enhancement with Soft Predicate Alignment

Entailment graphs (EGs) with predicates as nodes and entailment relations as edges are typically incomplete, while EGs in different languages are often complementary to each other.In this paper, we propose a new task, multilingual entailment graph enhancement, which aims to utilize the entailment information from one EG to enhance another EG in a different language.The ultimate goal is to obtain an enhanced EG containing richer and more accurate entailment information.We present an alignthen-enhance framework (ATE) to achieve accurate multilingual entailment graph enhancement, which first exploits a cross-graph guided interaction mechanism to automatically discover potential equivalent predicates between different EGs and then constructs more accurate enhanced entailment graphs based on soft predicate alignments.Extensive experiments show that ATE achieves better and more robust predicate alignment results between different EGs, and the enhanced entailment graphs generated by ATE outperform the original graphs for entailment detection 1 .
garNER at SemEval-2023: Simplified Knowledge Augmentation for Multilingual Complex Named Entity Recognition

This paper presents our solution, garNER, to the SemEval-2023 MultiConer task.We propose a knowledge augmentation approach by directly querying entities from the Wikipedia API and appending the summaries of the entities to the input sentence.These entities are either retrieved from the labeled training set (Gold Entity) or from off-the-shelf entity taggers (Entity Extractor).Ensemble methods are then applied across multiple models to get the final prediction.Our analysis shows that the added contexts are beneficial only when such contexts are relevant to the target-named entities, but detrimental when the contexts are irrelevant.
GMNLP at SemEval-2023 Task 12: Sentiment Analysis with Phylogeny-Based Adapters

This report describes GMU's sentiment analysis system for the SemEval-2023 shared task AfriSenti-SemEval.We participated in all three sub-tasks: Monolingual, Multilingual, and Zero-Shot.Our approach uses models initialized with AfroXLMR-large, a pre-trained multilingual language model trained on African languages and fine-tuned correspondingly.We also introduce augmented training data along with original training data.Alongside finetuning, we perform phylogeny-based adaptertuning to create several models and ensemble the best models for the final submission.Our system achieves the best F 1 -score on track 5: Amharic, with 6.2 points higher F 1 -score than the second-best performing system on this track.Overall, our system ranks 5 th among the 10 systems participating in all 15 tracks.
ERNIE-Code: Beyond English-Centric Cross-lingual Pretraining for Programming Languages

Software engineers working with the same programming language (PL) may speak different natural languages (NLs) and vice versa, erecting huge barriers to communication and working efficiency. Recent studies have demonstrated the effectiveness of generative pre-training in computer programs, yet they are always English-centric. In this work, we step towards bridging the gap between multilingual NLs and multilingual PLs for large language models (LLMs). We release ERNIE-Code, a unified pre-trained language model for 116 NLs and 6 PLs. We employ two methods for universal cross-lingual pre-training: span-corruption language modeling that learns patterns from monolingual NL or PL; and pivot-based translation language modeling that relies on parallel data of many NLs and PLs. Extensive results show that ERNIE-Code outperforms previous multilingual LLMs for PL or NL across a wide range of end tasks of code intelligence, including multilingual code-to-text, text-to-code, code-to-code, and text-to-text generation. We further show its advantage of zero-shot prompting on multilingual code summarization and text-to-text translation. We release our code and pre-trained checkpoints.
It’s not Sexually Suggestive; It’s Educative | Separating Sex Education from Suggestive Content on TikTok videos

We introduce SexTok, a multi-modal dataset composed of TikTok videos labeled as sexually suggestive (from the annotator's point of view), sex-educational content, or neither.Such a dataset is necessary to address the challenge of distinguishing between sexually suggestive content and virtual sex education videos on TikTok.Children's exposure to sexually suggestive videos has been shown to have adversarial effects on their development (Collins et al., 2017).Meanwhile, virtual sex education, especially on subjects that are more relevant to the LGBTQIA+ community, is very valuable (Mitchell et al., 2014).The platform's current system removes/punishes some of both types of videos, even though they serve different purposes.Our dataset contains video URLs, and it is also audio transcribed.To validate its importance, we explore two transformer-based models for classifying the videos.Our preliminary results suggest that the task of distinguishing between these types of videos is learnable but challenging.These experiments suggest that this dataset is meaningful and invites further study on the subject.
Deeply Coupled Cross-Modal Prompt Learning

Recent advancements in multimodal foundation models (e.g., CLIP) have excelled in zeroshot generalization.Prompt tuning involved in the knowledge transfer from foundation models to downstream tasks has gained significant attention recently.Existing prompttuning methods in cross-modal learning, however, either solely focus on language branch, or learn vision-language interaction in a shallow mechanism.In this context, we propose a Deeply coupled Cross-modal Prompt learning (DCP) method based on CLIP.DCP flexibly accommodates the interplay between vision and language with a Cross-Modal Prompt Attention (CMPA) mechanism, which enables the mutual exchange of respective representation through a well-connected multi-head attention module progressively and strongly.We then conduct comprehensive few-shot learning experiments on 11 image classification datasets and analyze the robustness to domain shift as well.Thorough experimental analysis evidently demonstrates the superb few-shot generalization and compelling domain adaption capacity of a well-executed DCP.The code can be found at https://github.com/GingL/CMPA.
FramingFreaks at SemEval-2023 Task 3: Detecting the Category and the Framing of Texts as Subword Units with Traditional Machine Learning

This paper describes our participation as team FramingFreaks in the SemEval-2023 task 3 "Category and Framing Predictions in online news in a multi-lingual setup."We participated in subtasks 1 and 2. Our approach was to classify texts by splitting them into subwords to reduce the feature set size and then using these tokens as input in Support Vector Machine (SVM) or logistic regression classifiers.Our results are similar to the baseline results.
eevvgg at SemEval-2023 Task 11: Offensive Language Classification with Rater-based Information

A standard majority-based approach to text classification is challenged with an individualised approach in the Semeval-2023 Task 11.Here, disagreements are treated as a useful source of information that could be utilised in the training pipeline.The team proposal makes use of partially disaggregated data and additional information about annotators provided by the organisers to train a BERT-based model for offensive text classification.The approach extends previous studies examining the impact of using raters' demographic features on classification performance (Hovy, 2015) or training machine learning models on disaggregated data (Davani et al., 2022).The proposed approach was ranked 11 across all 4 datasets, scoring best for cases with a large pool of annotators (6th place in the MD-Agreement dataset) utilising features based on raters' annotation behaviour.