Reviewerly/exHarmony huggingface dataset:
Bi-Phone: Modeling Inter Language Phonetic Influences in Text

A large number of people are forced to use the Web in a language they have low literacy in due to technology asymmetries.Written text in the second language (L2) from such users often contains a large number of errors that are influenced by their native language (L1).We propose a method to mine phoneme confusions (sounds in L2 that an L1 speaker is likely to conflate) for pairs of L1 and L2.These confusions are then plugged into a generative model (Bi-Phone) for synthetically producing corrupted L2 text.Through human evaluations, we show that Bi-Phone generates plausible corruptions that differ across L1s and also have widespread coverage on the Web.We also corrupt the popular language understanding benchmark SuperGLUE with our technique (FunGLUE for Phonetically Noised GLUE) and show that SoTA language understating models perform poorly.We also introduce a new phoneme prediction pre-training task which helps byte models to recover performance close to SuperGLUE.Finally, we also release the FunGLUE benchmark to promote further research in phonetically robust language models.To the best of our knowledge, FunGLUE is the first benchmark to introduce L1-L2 interactions in text.
HEVS-TUW at SemEval-2023 Task 8: Ensemble of Language Models and Rule-based Classifiers for Claims Identification and PICO Extraction

This paper describes the HEVS-TUW team submission to the SemEval-2023 Task 8: Causal Claims.We participated in two subtasks: (1) causal claims detection and (2) PIO identification.For subtask 1, we experimented with an ensemble of weakly supervised question detection and fine-tuned Transformer-based models.For subtask 2 of PIO frame extraction, we used a combination of deep representation learning and a rule-based approach.Our best model for subtask 1 ranks fourth with an F1-score of 65.77%.It shows moderate benefit from ensembling models pre-trained on independent categories.The results for subtask 2 warrant further investigation for improvement.
Helping a Friend or Supporting a Cause? Disentangling Active and Passive Cosponsorship in the U.S. Congress

In the U.S. Congress, legislators can use active and passive cosponsorship to support bills.We show that these two types of cosponsorship are driven by two different motivations: the backing of political colleagues and the backing of the bill's content.To this end, we develop an Encoder+RGCN based model that learns legislator representations from bill texts and speech transcripts.These representations predict active and passive cosponsorship with an F1-score of 0.88.Applying our representations to predict voting decisions, we show that they are interpretable and generalize to unseen tasks.
Finding the SWEET Spot: Analysis and Improvement of Adaptive Inference in Low Resource Settings

Adaptive inference is a simple method for reducing inference costs.The method works by maintaining multiple classifiers of different capacities, and allocating resources to each test instance according to its difficulty.In this work, we compare the two main approaches for adaptive inference, Early-Exit and Multi-Model, when training data is limited.First, we observe that for models with the same architecture and size, individual Multi-Model classifiers outperform their Early-Exit counterparts by an average of 2.3%.We show that this gap is caused by Early-Exit classifiers sharing model parameters during training, resulting in conflicting gradient updates of model weights.We find that despite this gap, Early-Exit still provides a better speed-accuracy trade-off due to the overhead of the Multi-Model approach.To address these issues, we propose SWEET, 1 an Early-Exit fine-tuning method that assigns each classifier its own set of unique model weights, not updated by other classifiers.We compare SWEET's speed-accuracy curve to standard Early-Exit and Multi-Model baselines and find that it outperforms both methods at fast speeds while maintaining comparable scores to Early-Exit at slow speeds.Moreover, SWEET individual classifiers outperform Early-Exit ones by 1.1% on average.SWEET enjoys the benefits of both methods, paving the way for further reduction of inference costs in NLP.We publicly release our code. 2C -1 C -2 C -3 C -4 C-1 C-2 C-3 C-4 1.0 -0.01 -0.02 -0.01 -0.01 1.0 0.47 0.36 -0.02 0.47 1.0 0.53 -0.01 0.36 0.53 1.0 Layer 1 C -2 C -3 C -4 C-2 C-3 C-4 1.0 0.03 0.02 0.03 1.0 0.33 0.02 0.33 1.
The Web Can Be Your Oyster for Improving Language Models

Pretrained language models (PLMs) encode a large amount of world knowledge.However, as such knowledge is frozen at the time of model training, the models become static and limited by the training data at that time.In order to further improve the capacity of PLMs for knowledge-intensive tasks, we consider augmenting PLMs with the large-scale web using search engine.Unlike previous augmentation sources (e.g., Wikipedia data dump), the web provides broader, more comprehensive and constantly updated information.In this paper, we present a web-augmented PLM -UNIWEB, which is trained over 16 knowledge-intensive tasks in a unified text-to-text format.Instead of simply using the retrieved contents from web, our approach has made two major improvements.Firstly, we propose an adaptive search engine assisted learning method that can self-evaluate the confidence level of PLM's predictions, and adaptively determine when to refer to the web for more data, which can avoid useless or noisy augmentation from web.Secondly, we design a pretraining task, i.e., continual knowledge learning, based on salient spans prediction, to reduce the discrepancy between the encoded and retrieved knowledge.Experiments on a wide range of knowledge-intensive tasks show that our model significantly outperforms previous retrieval-augmented methods.Our code and
Multijugate Dual Learning for Low-Resource Task-Oriented Dialogue System

Dialogue data in real scenarios tend to be sparsely available, rendering data-starved endto-end dialogue systems trained inadequately.We discover that data utilization efficiency in low-resource scenarios can be enhanced by mining alignment information uncertain utterance and deterministic dialogue state.Therefore, we innovatively implement dual learning in task-oriented dialogues to exploit the correlation of heterogeneous data.In addition, the one-to-one duality is converted into a multijugate duality to reduce the influence of spurious correlations in dual training for generalization.Without introducing additional parameters, our method could be implemented in arbitrary networks.Extensive empirical analyses demonstrate that our proposed method improves the effectiveness of end-to-end task-oriented dialogue systems under multiple benchmarks and obtains state-of-the-art results in low-resource scenarios.
SafeWebUH at SemEval-2023 Task 11: Learning Annotator Disagreement in Derogatory Text: Comparison of Direct Training vs Aggregation

Subjectivity and difference of opinion are key social phenomena, and it is crucial to take these into account in the annotation and detection process of derogatory textual content.In this paper, we use four datasets provided by SemEval-2023 Task 11 and fine-tune a BERT model to capture the disagreement in the annotation.We find individual annotator modeling and aggregation lowers the Cross-Entropy score by an average of 0.21, compared to the direct training on the soft labels.Our findings further demonstrate that annotator metadata contributes to the average 0.029 reduction in the Cross-Entropy score.
Domain-specific Attention with Distributional Signatures for Multi-Domain End-to-end Task-Oriented Dialogue

The end-to-end task-oriented dialogue system has achieved great success in recent years.Most of these dialogue systems need to accommodate multi-domain dialogue in real-world scenarios.However, due to the high cost of dialogue data annotation and the scarcity of labeled dialogue data, existing methods are difficult to extend to new domains.Therefore, it is essential to use limited data to construct multi-domain dialogue systems.To solve this problem, we propose a novel domain attention module.It uses distributional signatures to construct a multi-domain dialogue system effectively with limited data, which has strong extensibility.We also define an adjacent n-gram pattern to explore potential patterns for dialogue entities.Experimental results show that our approach outperforms the baseline models on most metrics.In the few-shot scenario, we show our method gets a great improvement compared with previous methods while keeping a smaller model scale.
Estimating the Uncertainty in Emotion Attributes using Deep Evidential Regression

In automatic emotion recognition (AER), labels assigned by different human annotators to the same utterance are often inconsistent due to the inherent complexity of emotion and the subjectivity of perception.Though deterministic labels generated by averaging or voting are often used as the ground truth, it ignores the intrinsic uncertainty revealed by the inconsistent labels.This paper proposes a Bayesian approach, deep evidential emotion regression (DEER), to estimate the uncertainty in emotion attributes.Treating the emotion attribute labels of an utterance as samples drawn from an unknown Gaussian distribution, DEER places an utterance-specific normal-inverse gamma prior over the Gaussian likelihood and predicts its hyper-parameters using a deep neural network model.It enables a joint estimation of emotion attributes along with the aleatoric and epistemic uncertainties.AER experiments on the widely used MSP-Podcast and IEMOCAP datasets showed DEER produced state-of-theart results for both the mean values and the distribution of emotion attributes 1 .
Visually-augmented pretrained language models for NLP tasks without images

Although pre-trained language models (PLMs) have shown impressive performance by textonly self-supervised training, they are found lack of visual semantics or commonsense.Existing solutions often rely on explicit images for visual knowledge augmentation (requiring time-consuming retrieval or generation), and they also conduct the augmentation for the whole input text, without considering whether it is actually needed in specific inputs or tasks.To address these issues, we propose a novel Visually-Augmented fine-tuning approach that can be generally applied to various PLMs or NLP tasks, Without using any retrieved or generated Images, namely VAWI.Experimental results show that our approach can consistently improve the performance of BERT, RoBERTa, BART, and T5 at different scales, and outperform several competitive baselines on ten tasks.Our codes and data are publicly available at https://github.com/RUCAIBox/VAWI.
Binary and Ternary Natural Language Generation

Ternary and binary neural networks enable multiplication-free computation and promise multiple orders of magnitude efficiency gains over full-precision networks if implemented on specialized hardware.However, since both the parameter and the output space are highly discretized, such networks have proven very difficult to optimize.The difficulties are compounded for the class of transformer text generation models due to the sensitivity of the attention operation to quantization and the noise-compounding effects of autoregressive decoding in the high-cardinality output space.We approach the problem with a mix of statistics-based quantization for the weights and elastic quantization of the activations and demonstrate the first ternary and binary transformer models on the downstream tasks of summarization and machine translation.Our ternary BART base achieves an R1 score of 41 on the CNN/DailyMail benchmark, which is merely 3.9 points behind the full model while being 16x more efficient.Our binary model, while less accurate, achieves a highly nontrivial score of 35.6.For machine translation, we achieved BLEU scores of 21.7 and 17.6 on the WMT16 En-Ro benchmark, compared with a full precision mBART model score of 26.8.We also compare our approach in the 8-bit activation setting, where our ternary and even binary weight models can match or outperform the best existing 8-bit weight models in the literature.Our code and models are available at:
Measuring Lexico-Semantic Alignment in Debates with Contextualized Word Representations

Dialog participants sometimes align their linguistic styles, e.g., they use the same words and syntactic constructions as their interlocutors.We propose to investigate the notion of lexico-semantic alignment: to what extent do speakers convey the same meaning when they use the same words?We design measures of lexico-semantic alignment relying on contextualized word representations.We show that they reflect interesting semantic differences between the two sides of a debate and that they can assist in the task of debate's winner prediction.
SSP: Self-Supervised Post-training for Conversational Search

Conversational search has been regarded as the next-generation search paradigm.Constrained by data scarcity, most existing methods distill the well-trained ad-hoc retriever to the conversational retriever.However, these methods, which usually initialize parameters by query reformulation to discover contextualized dependency, have trouble in understanding the dialogue structure information and struggle with contextual semantic vanishing.In this paper, we propose Self-Supervised Posttraining (SSP) which is a new post-training paradigm with three self-supervised tasks to efficiently initialize the conversational search model to enhance the dialogue structure and contextual semantic understanding.Furthermore, the SSP can be plugged into most of the existing conversational models to boost their performance.To verify the effectiveness of our proposed method, we apply the conversational encoder post-trained by SSP on the conversational search task using two benchmark datasets: CAsT-19 and CAsT-20.Extensive experiments that our SSP can boost the performance of several existing conversational search methods.Our source code is available at https://github.com/morecry/SSP.
Cross-lingual transfer for low-resource Arabic language understanding

This paper explores cross-lingual transfer learning in natural language understanding (NLU), with the focus on bootstrapping Arabic from high-resource English and French languages for domain classification, intent classification, and named entity recognition tasks.We adopt a BERTbased architecture and pretrain three models using open-source Wikipedia data and large-scale commercial datasets: monolingual:Arabic, bilingual:Arabic-English, and trilingual:Arabic-English-French models.Additionally, we use off-the-shelf machine translator to translate internal data from source English language to the target Arabic language, in an effort to enhance transfer learning through translation.We conduct experiments that finetune the three models for NLU tasks and evaluate them on a large internal dataset.Despite the morphological, orthographical, and grammatical differences between Arabic and the source languages, transfer learning performance gains from source languages and through machine translation are achieved on a real-world Arabic test dataset in both a zero-shot setting and in a setting when the models are further finetuned on labeled data from the target language.
Identifying the Source of Vulnerability in Explanation Discrepancy: A Case Study in Neural Text Classification

Some recent works observed the instability of post-hoc explanations when input side perturbations are applied to the model.This raises the interest and concern in the stability of post-hoc explanations.However, the remaining question is: is the instability caused by the neural network model or the post-hoc explanation method?This work explores the potential source that leads to unstable post-hoc explanations.To separate the influence from the model, we propose a simple output probability perturbation method.Compared to prior input side perturbation methods, the output probability perturbation method can circumvent the neural model's potential effect on the explanations and allow the analysis on the explanation method.We evaluate the proposed method with three widely-used post-hoc explanation methods (LIME (Ribeiro et al., 2016), Kernel Shapley (Lundberg and Lee, 2017a), and Sample Shapley (Strumbelj and Kononenko, 2010)).The results demonstrate that the post-hoc methods are stable, barely producing discrepant explanations under output probability perturbations.The observation suggests that neural network models may be the primary source of fragile explanations.
Dial2vec: Self-Guided Contrastive Learning of Unsupervised Dialogue Embeddings

In this paper, we introduce the task of learning unsupervised dialogue embeddings.Trivial approaches such as combining pre-trained word or sentence embeddings and encoding through pre-trained language models (PLMs) have been shown to be feasible for this task.However, these approaches typically ignore the conversational interactions between interlocutors, resulting in poor performance.To address this issue, we proposed a selfguided contrastive learning approach named dial2vec.Dial2vec considers a dialogue as an information exchange process.It captures the conversational interaction patterns between interlocutors and leverages them to guide the learning of the embeddings corresponding to each interlocutor.The dialogue embedding is obtained by an aggregation of the embeddings from all interlocutors.To verify our approach, we establish a comprehensive benchmark consisting of six widely-used dialogue datasets.We consider three evaluation tasks: domain categorization, semantic relatedness, and dialogue retrieval.Dial2vec achieves on average 8.7, 9.0, and 13.8 points absolute improvements in terms of purity, Spearman's correlation, and mean average precision (MAP) over the strongest baseline on the three tasks respectively.Further analysis shows that dial2vec obtains informative and discriminative embeddings for both interlocutors under the guidance of the conversational interactions and achieves the best performance when aggregating them through the interlocutor-level pooling strategy.
Self-training with Two-phase Self-augmentation for Few-shot Dialogue Generation

In task-oriented dialogue systems, response generation from meaning representations (MRs) often suffers from limited training examples, due to the high cost of annotating MRto-Text pairs.Previous works on self-training leverage fine-tuned conversational models to automatically generate pseudo-labeled MR-to-Text pairs for further fine-tuning.However, some self-augmented data may be noisy or uninformative for the model to learn from.In this work, we propose a two-phase selfaugmentation procedure to generate highquality pseudo-labeled MR-to-Text pairs: the first phase selects the most informative MRs based on model's prediction uncertainty; with the selected MRs, the second phase generates accurate responses by aggregating multiple perturbed latent representations from each MR.Empirical experiments on two benchmark datasets, FEWSHOTWOZ and FEWSHOTSGD, show that our method generally outperforms existing self-training methods on both automatic and human evaluations. 1
Probing Relational Knowledge in Language Models via Word Analogies

Understanding relational knowledge plays an integral part in natural language understanding.When it comes to pre-trained language models (PLMs), prior work has been focusing on probing relational knowledge by filling the blanks in pre-defined prompts such as "The capital of France is -".However, these probes may be affected by the co-occurrence of target relation words and entities (e.g."capital", "France" and "Paris") in the pre-training corpus.In this work, we extend these probing methodologies leveraging analogical proportions as a proxy to probe relational knowledge in transformerbased PLMs without directly presenting the desired relation.In particular, we analysed the ability of PLMs to understand (1) the directionality of a given relation (e.g.Paris-France is not the same as France-Paris); (2) the ability to distinguish types on a given relation (both France and Japan are countries); and (3) the relation itself (Paris is the capital of France, but not Rome).Our results show how PLMs are extremely accurate at (1) and ( 2), but have room for improvement for (3).To better understand the reasons behind this behaviour and the types of mistake made by PLMs, we provide an extended quantitative analysis.
HARALD: Augmenting Hate Speech Data Sets with Real Data

The successful completion of the hate speech detection task hinges upon the availability of rich and variable labeled data, which is hard to obtain.In this work, we present a new approach for data augmentation that uses as input real unlabelled data, which is carefully selected from online platforms where invited hate speech is abundant.We show that by harvesting and processing this data (in an automatic manner), one can augment existing manually-labeled datasets to improve the classification performance of hate speech classification models.We observed an improvement in F1-score ranging from 2.7% and up to 9.5%, depending on the task (in-or cross-domain) and the model used.
Probing for targeted syntactic knowledge through grammatical error detection

Targeted studies testing knowledge of subjectverb agreement (SVA) indicate that pre-trained language models encode syntactic information.We assert that if models robustly encode subject-verb agreement, they should be able to identify when agreement is correct and when it is incorrect.To that end, we propose grammatical error detection as a diagnostic probe to evaluate token-level contextual representations for their knowledge of SVA.We evaluate contextual representations at each layer from five pre-trained English language models: BERT, XLNET, GPT-2, ROBERTA, and ELEC-TRA.We leverage public annotated training data from both English second language learners and Wikipedia edits, and report results on manually crafted stimuli for subject-verb agreement.We find that masked language models linearly encode information relevant to the detection of SVA errors, while the autoregressive models perform on par with our baseline.However, we also observe a divergence in performance when probes are trained on different training sets, and when they are evaluated on different syntactic constructions, suggesting the information pertaining to SVA error detection is not robustly encoded.
Bridging the Training-Inference Gap for Dense Phrase Retrieval

Building dense retrievers requires a series of standard procedures, including training and validating neural models and creating indexes for efficient search.However, these procedures are often misaligned in that training objectives do not exactly reflect the retrieval scenario at inference time.In this paper, we explore how the gap between training and inference in dense retrieval can be reduced, focusing on dense phrase retrieval (Lee et al., 2021a) where billions of representations are indexed at inference.Since validating every dense retriever with a large-scale index is practically infeasible, we propose an efficient way of validating dense retrievers using a small subset of the entire corpus.This allows us to validate various training strategies, including unifying contrastive loss terms and using hard negatives for phrase retrieval, which largely reduces the training-inference discrepancy.As a result, we improve top-1 phrase retrieval accuracy by 2∼3 points and top-20 passage retrieval accuracy by 2∼4 points for open-domain question answering.Our work urges modeling dense retrievers with careful consideration of training and inference via efficient validation while advancing phrase retrieval as a general solution for dense retrieval.