Reviewerly/exHarmony huggingface dataset:
Pivotal Role of Language Modeling in Recommender Systems: Enriching Task-specific and Task-agnostic Representation Learning

Recent studies have proposed unified user modeling frameworks that leverage user behavior data from various applications.Many of them benefit from utilizing users' behavior sequences as plain texts, representing rich information in any domain or system without losing generality.Hence, a question arises: Can language modeling for user history corpus help improve recommender systems?While its versatile usability has been widely investigated in many domains, its applications to recommender systems still remain underexplored.We show that language modeling applied directly to taskspecific user histories achieves excellent results on diverse recommendation tasks.Also, leveraging additional task-agnostic user histories delivers significant performance benefits.We further demonstrate that our approach can provide promising transfer learning capabilities for a broad spectrum of real-world recommender systems, even on unseen domains and services.
Class-Incremental Learning based on Label Generation

Despite the great success of pre-trained language models, it is still a challenge to use these models for continual learning, especially for the class-incremental learning (CIL) setting due to catastrophic forgetting (CF).This paper reports our finding that if we formulate CIL as a continual label generation problem, CF is drastically reduced and the generalizable representations of pre-trained models can be better retained.We thus propose a new CIL method (VAG) that also leverages the sparsity of vocabulary to focus the generation and creates pseudo-replay samples by using label semantics.Experimental results show that VAG outperforms baselines by a large margin. 1
Parameter-Efficient Fine-Tuning without Introducing New Latency

Parameter-efficient fine-tuning (PEFT) of pretrained language models has recently demonstrated remarkable achievements, effectively matching the performance of full fine-tuning while utilizing significantly fewer trainable parameters, and consequently addressing the storage and communication constraints.Nonetheless, various PEFT methods are limited by their inherent characteristics.In the case of sparse fine-tuning, which involves modifying only a small subset of the existing parameters, the selection of fine-tuned parameters is task-and domain-specific, making it unsuitable for federated learning.On the other hand, PEFT methods with adding new parameters typically introduce additional inference latency.In this paper, we demonstrate the feasibility of generating a sparse mask in a task-agnostic manner, wherein all downstream tasks share a common mask.Our approach, which relies solely on the magnitude information of pre-trained parameters, surpasses existing methodologies by a significant margin when evaluated on the GLUE benchmark.Additionally, we introduce a novel adapter technique that directly applies the adapter to pre-trained parameters instead of the hidden representation, thereby achieving identical inference speed to that of full finetuning.Through extensive experiments, our proposed method attains a new state-of-the-art outcome in terms of both performance and storage efficiency, storing only 0.03% parameters of full fine-tuning.1
DEplain: A German Parallel Corpus with Intralingual Translations into Plain Language for Sentence and Document Simplification

Text simplification is an intralingual translation task in which documents, or sentences of a complex source text are simplified for a target audience.The success of automatic text simplification systems is highly dependent on the quality of parallel data used for training and evaluation.To advance sentence simplification and document simplification in German, this paper presents DEPLAIN, a new dataset of parallel, professionally written and manually aligned simplifications in plain German ("plain DE" or in German: 'Einfache Sprache").DEPLAIN consists of a newsdomain (approx.500 document pairs, approx.13k sentence pairs) and a web-domain corpus (approx.150 aligned documents, approx.2k aligned sentence pairs).In addition, we are building a web harvester and experimenting with automatic alignment methods to facilitate the integration of non-aligned and to bepublished parallel documents.Using this approach, we are dynamically increasing the webdomain corpus, so it is currently extended to approx.750 document pairs and approx.3.5k aligned sentence pairs.We show that using DEPLAIN to train a transformer-based seq2seq text simplification model can achieve promising results.We make available the corpus, the adapted alignment methods for German, the web harvester and the trained models here: https://github.com/rstodden/DEPlain.
Enhancing Unsupervised Semantic Parsing with Distributed Contextual Representations

We extend a non-parametric Bayesian model of (Titov and Klementiev, 2011) to deal with homonymy and polysemy by leveraging distributed contextual word and phrase representations pre-trained on a large collection of unlabelled texts.Then, unsupervised semantic parsing is performed by decomposing sentences into fragments, clustering the fragments to abstract away syntactic variations of the same meaning, and predicting predicate-argument relations between the fragments.To better model the statistical dependencies between predicates and their arguments, we further conduct a hierarchical Pitman-Yor process.An improved Metropolis-Hastings merge-split sampler is proposed to speed up the mixing and convergence of Markov chains by leveraging pre-trained distributed representations.The experimental results show that the models achieve better accuracy on both question-answering and relation extraction tasks.
USTC-NELSLIP at SemEval-2023 Task 2: Statistical Construction and Dual Adaptation of Gazetteer for Multilingual Complex NER

This paper describes the system developed by the USTC-NELSLIP team for SemEval-2023 Task 2 Multilingual Complex Named Entity Recognition (MultiCoNER II).A method named Statistical Construction and Dual Adaptation of Gazetteer (SCDAG) is proposed for Multilingual Complex NER.The method first utilizes a statistics-based approach to construct a gazetteer.Secondly, the representations of gazetteer networks and language models are adapted by minimizing the KL divergence between them at both the sentence-level and entity-level.Finally, these two networks are then integrated for supervised named entity recognition (NER) training.The proposed method is applied to XLM-R with a gazetteer built from Wikidata, and shows great generalization ability across different tracks.Experimental results and detailed analysis verify the effectiveness of the proposed method.The official results show that our system ranked 1st on one track (Hindi) in this task.
PoSh at SemEval-2023 Task 10: Explainable Detection of Online Sexism

The growing popularity of online platforms and social media has intensified the problem of sexism, making it more prevalent and widespread.Online platforms have provided an anonymous space where individuals can freely express their misogynistic beliefs with discriminatory and abusive language, creating a culture of online harassment and hate speech, mainly targeting women.The dataset used in this task was provided by SemEval-2023 Task 10: Explainable Detection of Online Sexism.To precisely identify the different forms of online sexism, we utilize several sentence transformer models such as ALBERT, BERT, RoBERTa, DistilBERT, and XLNet.By combining the predictions from these models, we can generate a more comprehensive and improved result.Each transformer model is trained after pre-processing the data from the training dataset.Our team obtained macro f1 scores of 0.7937 for subtask A, 0.5284 for subtask B and 0.2674 for subtask C.
Epicurus at SemEval-2023 Task 4: Improving Prediction of Human Values behind Arguments by Leveraging Their Definitions

We describe our experiments for SemEval-2023 Task 4 on the identification of human values behind arguments (ValueEval).Because human values are subjective concepts which require precise definitions, we hypothesize that incorporating the definitions of human values (in the form of annotation instructions and validated survey items) during model training can yield better prediction performance.We explore this idea and show that our proposed models perform better than the challenge organizers' baselines, with improvements in macro F 1 scores of up to 18%.
To Asymmetry and Beyond: Structured Pruning of Sequence to Sequence Models for Improved Inference Efficiency

Sequence-to-sequence language models can be used to produce abstractive summaries which are coherent, relevant, and concise.Still, model sizes can make deployment in latency-sensitive or web-scale implementations difficult.This paper studies the relationship between model size, structured pruning, inference efficiency, and summarization accuracy on widely used summarization datasets.We show that model accuracy is tied to the encoder size while inference efficiency is connected to the decoder.Using asymmetric pruning can lead to nearly 3x improvement in inference latency with 1 point loss in Rouge-2.Moreover, we find both the average degradation and the role of asymmetry to be consistent across model sizes and variations in datasets.We release our code 1 , training regimes, and associated model 2 for broad usage to encourage usage and experimentation.
A Multilingual Evaluation of NER Robustness to Adversarial Inputs

Adversarial evaluations of language models typically focus on English alone.In this paper, we performed a multilingual evaluation of Named Entity Recognition (NER) in terms of its robustness to small perturbations in the input.Our results showed the NER models we explored across three languages (English, German and Hindi) are not very robust to such changes, as indicated by the fluctuations in the overall F1 score as well as in a more finegrained evaluation.With that knowledge, we further explored whether it is possible to improve the existing NER models using a part of the generated adversarial data sets as augmented training data to train a new NER model or as fine-tuning data to adapt an existing NER model.Our results showed that both these approaches improve performance on the original as well as adversarial test sets.While there is no significant difference between the two approaches for English, re-training is significantly better than fine-tuning for German and Hindi.
TeamAmpa at SemEval-2023 Task 3: Exploring Multilabel and Multilingual RoBERTa Models for Persuasion and Framing Detection

This paper describes our submission to the SemEval 2023 Task 3 on two subtasks: detecting persuasion techniques and framing.Both subtasks are multi-label classification problems.We present a set of experiments, exploring how to get robust performance across languages using pre-trained RoBERTa models.We test different oversampling strategies, a strategy of adding textual features from predictions obtained with related models, and present both inconclusive and negative results.We achieve a robust ranking across languages and subtasks with our best ranking being nr. 1 for Subtask 3 on Spanish.
When to Use What: An In-Depth Comparative Empirical Analysis of OpenIE Systems for Downstream Applications

Open Information Extraction (OpenIE) has been used in the pipelines of various NLP tasks.Unfortunately, there is no clear consensus on which models to use for which tasks.Muddying things further is the lack of comparisons that take differing training sets into account.In this paper, we present an application-focused empirical survey of neural OpenIE models, training sets, and benchmarks in an effort to help users choose the most suitable OpenIE systems for their applications.We find that the different assumptions made by different models and datasets have a statistically significant effect on performance, making it important to choose the most appropriate model for one's applications.We demonstrate the applicability of our recommendations on a downstream Complex QA application.
Interpreting Positional Information in Perspective of Word Order

The attention mechanism is a powerful and effective method utilized in natural language processing.However, it has been observed that this method is insensitive to positional information.Although several studies have attempted to improve positional encoding and investigate the influence of word order perturbation, it remains unclear how positional encoding impacts NLP models from the perspective of word order.In this paper, we aim to shed light on this problem by analyzing the working mechanism of the attention module and investigating the root cause of its inability to encode positional information.Our hypothesis is that the insensitivity can be attributed to the weight sum operation utilized in the attention module.To verify this hypothesis, we propose a novel weight concatenation operation and evaluate its efficacy in neural machine translation tasks.Our enhanced experimental results not only reveal that the proposed operation can effectively encode positional information but also confirm our hypothesis.
IITD at SemEval-2023 Task 2: A Multi-Stage Information Retrieval Approach for Fine-Grained Named Entity Recognition

MultiCoNER-II is a fine-grained Named Entity Recognition (NER) task that aims to identify ambiguous and complex named entities in multiple languages, with a small amount of contextual information available.To address this task, we propose a multi-stage information retrieval (IR) pipeline that improves the performance of language models for fine-grained NER.Our approach involves leveraging a combination of a BM25-based IR model and a language model to retrieve relevant passages from a corpus.These passages are then used to train a model that utilizes a weighted average of losses.The prediction is generated by a decoder stack that includes a projection layer and conditional random field.To demonstrate the effectiveness of our approach, we participated in the English track of the MultiCoNER-II competition.Our approach yielded promising results, which we validated through detailed analysis.
How to Plant Trees in Language Models: Data and Architectural Effects on the Emergence of Syntactic Inductive Biases

Accurate syntactic representations are essential for robust generalization in natural language.Recent work has found that pre-training can teach language models to rely on hierarchical syntactic features-as opposed to incorrect linear features-when performing tasks after finetuning.We test what aspects of pre-training are important for endowing encoder-decoder Transformers with an inductive bias that favors hierarchical syntactic generalizations.We focus on architectural features (depth, width, and number of parameters), as well as the genre and size of the pre-training corpus, diagnosing inductive biases using two syntactic transformation tasks: question formation and passivization, both in English.We find that the number of parameters alone does not explain hierarchical generalization: model depth plays greater role than model width.We also find that pre-training on simpler language, such as child-directed speech, induces a hierarchical bias using an order-of-magnitude less data than pre-training on more typical datasets based on web text or Wikipedia; this suggests that in cognitively plausible language acquisition settings, neural language models may be more data-efficient than previously thought.
BenCoref: A Multi-Domain Dataset of Nominal Phrases and Pronominal Reference Annotations

Coreference Resolution is a well studied problem in NLP.While widely studied for English and other resource-rich languages, research on coreference resolution in Bengali largely remains unexplored due to the absence of relevant datasets.Bengali, being a low-resource language, exhibits greater morphological richness compared to English.In this article, we introduce a new dataset, BenCoref, comprising coreference annotations for Bengali texts gathered from four distinct domains.This relatively small dataset contains 5200 mention annotations forming 502 mention clusters within 48,569 tokens.We describe the process of creating this dataset and report performance of multiple models trained using BenCoref.We expect that our work provides some valuable insights on the variations in coreference phenomena across several domains in Bengali and encourages the development of additional resources for Bengali.Furthermore, we found poor crosslingual performance at zero-shot setting from English, highlighting the need for more language-specific resources for this task.The dataset is available at 1 .
An Inner Table Retriever for Robust Table Question Answering

Recent years have witnessed the thriving of pretrained Transformer-based language models for understanding semi-structured tables, with several applications, such as Table Question Answering (TableQA).These models are typically trained on joint tables and surrounding natural language text, by linearizing table content into sequences comprising special tokens and cell information.This yields very long sequences which increase system inefficiency, and moreover, simply truncating long sequences results in information loss for downstream tasks.We propose Inner Table Retriever (ITR), 1 a generalpurpose approach for handling long tables in TableQA that extracts sub-tables to preserve the most relevant information for a question.We show that ITR can be easily integrated into existing systems to improve their accuracy with up to 1.3-4.8%and achieve state-of-the-art results in two benchmarks, i.e., 63.4% in Wik-iTableQuestions and 92.1% in WikiSQL.Additionally, we show that ITR makes TableQA systems more robust to reduced model capacity and to different ordering of columns and rows.
Tab-CoT: Zero-shot Tabular Chain of Thought

The chain-of-though (CoT) prompting methods were successful in various natural language processing (NLP) tasks thanks to their ability to unveil the underlying complex reasoning processes.Such reasoning processes typically exhibit implicitly structured steps.Recent efforts also started investigating methods to encourage more explicitly structured reasoning procedures to be captured (Zhou et al., 2022).In this work, we propose Tab-CoT, a novel tabular-format CoT prompting method, which allows the complex reasoning process to be explicitly modelled in a highly structured manner.Despite its simplicity, we show that our approach is capable of performing reasoning across multiple dimensions (i.e., both rows and columns).We demonstrate our approach's strong zero-shot and few-shot capabilities through extensive experiments on a range of reasoning tasks. 1
Disentangling Aspect and Stance via a Siamese Autoencoder for Aspect Clustering of Vaccination Opinions

Mining public opinions about vaccines from social media has been increasingly relevant to analyse trends in public debates and to provide quick insights to policy-makers.However, the application of existing models has been hindered by the wide variety of users' attitudes and the new aspects continuously arising in the public debate.Existing approaches, frequently framed via well-known tasks, such as aspect classification or text span detection, make direct usage of the supervision information constraining the models to predefined aspect classes, while still not distinguishing those aspects from users' stances.As a result, this has significantly hindered the dynamic integration of new aspects.We thus propose a model, namely Disentangled Opinion Clustering (DOC), for vaccination opinion mining from social media.DOC is able to disentangle users' stances from opinions via a disentangling attention mechanism and a Swapping-Autoencoder, and is designed to process unseen aspect categories via a clustering approach, leveraging clustering-friendly representations induced by out-of-the-box Sentence-BERT encodings and disentangling mechanisms.We conduct a thorough experimental assessment demonstrating the benefit of the disentangling mechanisms and cluster-based approach on both the quality of aspect clusters and the generalization across new aspect categories, outperforming existing methodologies on aspectbased opinion mining.
Query-Efficient Black-Box Red Teaming via Bayesian Optimization

The deployment of large-scale generative models is often restricted by their potential risk of causing harm to users in unpredictable ways.We focus on the problem of black-box red teaming, where a red team generates test cases and interacts with the victim model to discover a diverse set of failures with limited query access.Existing red teaming methods construct test cases based on human supervision or language model (LM) and query all test cases in a brute-force manner without incorporating any information from past evaluations, resulting in a prohibitively large number of queries.To this end, we propose Bayesian red teaming (BRT), novel query-efficient blackbox red teaming methods based on Bayesian optimization, which iteratively identify diverse positive test cases leading to model failures by utilizing the pre-defined user input pool and the past evaluations.Experimental results on various user input pools demonstrate that our method consistently finds a significantly larger number of diverse positive test cases under the limited query budget than the baseline methods.The source code is available at https://github.com/snu-mllab/Bayesian-Red-Teaming.
Quick Dense Retrievers Consume KALE: Post Training KullbackLeibler Alignment of Embeddings for Asymmetrical dual encoders

In this paper, we consider the problem of improving the inference latency of language model-based dense retrieval systems by introducing structural compression and model size asymmetry between the context and query encoders.First, we investigate the impact of pre and post-training compression on the MSMARCO, Natural Questions, TriviaQA, SQUAD, and SCIFACT, finding that asymmetry in the dual-encoders in dense retrieval can lead to improved inference efficiency.Knowing this, we introduce Kullback-Leibler Alignment of Embeddings (KALE), an efficient and accurate method for increasing the inference efficiency of dense retrieval methods by pruning and aligning the query encoder after training.Specifically, KALE extends traditional Knowledge Distillation after bi-encoder training, allowing for effective query encoder compression without full retraining or index generation.Using KALE and asymmetric training, we can generate models which exceed the performance of DistilBERT despite having 3x faster inference.
InfoMetIC: An Informative Metric for Reference-free Image Caption Evaluation

Automatic image captioning evaluation is critical for benchmarking and promoting advances in image captioning research.Existing metrics only provide a single score to measure caption qualities, which are less explainable and informative.Instead, we humans can easily identify the problems of captions in details, e.g., which words are inaccurate and which salient objects are not described, and then rate the caption quality.To support such informative feedback, we propose an Informative Metric for Reference-free Image Caption evaluation (InfoMetIC).Given an image and a caption, InfoMetIC is able to report incorrect words and unmentioned image regions at fine-grained level, and also provide a text precision score, a vision recall score and an overall quality score at coarse-grained level.The coarse-grained score of InfoMetIC achieves significantly better correlation with human judgements than existing metrics on multiple benchmarks.We also construct a token-level evaluation dataset and demonstrate the effectiveness of InfoMetIC in fine-grained evaluation.Our code and datasets are publicly available at https://github.com/HAWLYQ/InfoMetIC.