GenAI4ELab/papercli-papers huggingface dataset:
Contrastive Bootstrapping for Label Refinement
Authors: Shudi Hou; Yu Xia; Muhao Chen; Sujian Li
Venue: ACL 2023 (Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers))

Abstract: Traditional text classification typically categorizes texts into pre-defined coarse-grained classes, from which the produced models cannot handle the real-world scenario where finer categories emerge periodically for accurate services. In this work, we investigate the setting where fine-grained classification is done only using the annotation of coarse-grained categories and the coarse-to-fine mapping. We propose a lightweight contrastive clustering-based bootstrapping method to iteratively refine the labels of passages. During clustering, it pulls away negative passage-prototype pairs under the guidance of the mapping from both global and local perspectives. Experiments on NYT and 20News show that our method outperforms the state-of-the-art methods by a large margin.
NollySenti: Leveraging Transfer Learning and Machine Translation for Nigerian Movie Sentiment Classification
Authors: Iyanuoluwa Shode; David Ifeoluwa Adelani; Jing Peng; Anna Feldman
Venue: ACL 2023 (Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers))

Abstract: Africa has over 2000 indigenous languages but they are under-represented in NLP research due to lack of datasets. In recent years, there have been progress in developing labelled corpora for African languages. However, they are often available in a single domain and may not generalize to other domains. In this paper, we focus on the task of sentiment classification for cross-domain adaptation. We create a new dataset, Nollywood movie reviews for five languages widely spoken in Nigeria (English, Hausa, Igbo, Nigerian Pidgin, and Yoruba). We provide an extensive empirical evaluation using classical machine learning methods and pre-trained language models. By leveraging transfer learning, we compare the performance of cross-domain adaptation from Twitter domain, and cross-lingual adaptation from English language. Our evaluation shows that transfer from English in the same target domain leads to more than 5% improvement in accuracy compared to transfer from Twitter in the same language. To further mitigate the domain difference, we leverage machine translation from English to other Nigerian languages, which leads to a further improvement of 7% over cross-lingual evaluation. While machine translation to low-resource languages are often of low quality, our analysis shows that sentiment related words are often preserved.
Trading Syntax Trees for Wordpieces: Target-oriented Opinion Words Extraction with Wordpieces and Aspect Enhancement
Authors: Samuel Mensah; Kai Sun; Nikolaos Aletras
Venue: ACL 2023 (Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers))

Abstract: State-of-the-art target-oriented opinion word extraction (TOWE) models typically use BERT-based text encoders that operate on the word level, along with graph convolutional networks (GCNs) that incorporate syntactic information extracted from syntax trees. These methods achieve limited gains with GCNs and have difficulty using BERT wordpieces. Meanwhile, BERT wordpieces are known to be effective at representing rare words or words with insufficient context information. To address this issue, this work trades syntax trees for BERT wordpieces by entirely removing the GCN component from the methods’ architectures. To enhance TOWE performance, we tackle the issue of aspect representation loss during encoding. Instead of solely utilizing a sentence as the input, we use a sentence-aspect pair. Our relatively simple approach achieves state-of-the-art results on benchmark datasets and should serve as a strong baseline for further research.
An (unhelpful) guide to selecting the best ASR architecture for your under-resourced language
Authors: Robert Jimerson; Zoey Liu; Emily Prud’hommeaux
Venue: ACL 2023 (Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers))

Abstract: Advances in deep neural models for automatic speech recognition (ASR) have yielded dramatic improvements in ASR quality for resource-rich languages, with English ASR now achieving word error rates comparable to that of human transcribers. The vast majority of the world’s languages, however, lack the quantity of data necessary to approach this level of accuracy. In this paper we use four of the most popular ASR toolkits to train ASR models for eleven languages with limited ASR training resources: eleven widely spoken languages of Africa, Asia, and South America, one endangered language of Central America, and three critically endangered languages of North America. We find that no single architecture consistently outperforms any other. These differences in performance so far do not appear to be related to any particular feature of the datasets or characteristics of the languages. These findings have important implications for future research in ASR for under-resourced languages. ASR systems for languages with abundant existing media and available speakers may derive the most benefit simply by collecting large amounts of additional acoustic and textual training data. Communities using ASR to support endangered language documentation efforts, who cannot easily collect more data, might instead focus on exploring multiple architectures and hyperparameterizations to optimize performance within the constraints of their available data and resources.
The Ecological Fallacy in Annotation: Modeling Human Label Variation goes beyond Sociodemographics
Authors: Matthias Orlikowski; Paul Röttger; Philipp Cimiano; Dirk Hovy
Venue: ACL 2023 (Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers))

Abstract: Many NLP tasks exhibit human label variation, where different annotators give different labels to the same texts. This variation is known to depend, at least in part, on the sociodemographics of annotators. Recent research aims to model individual annotator behaviour rather than predicting aggregated labels, and we would expect that sociodemographic information is useful for these models. On the other hand, the ecological fallacy states that aggregate group behaviour, such as the behaviour of the average female annotator, does not necessarily explain individual behaviour. To account for sociodemographics in models of individual annotator behaviour, we introduce group-specific layers to multi-annotator models. In a series of experiments for toxic content detection, we find that explicitly accounting for sociodemographic attributes in this way does not significantly improve model performance. This result shows that individual annotation behaviour depends on much more than just sociodemographics.
Decomposed scoring of CCG dependencies
Authors: Aditya Bhargava; Gerald Penn
Venue: ACL 2023 (Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers))

Abstract: In statistical parsing with CCG, the standard evaluation method is based on predicate-argument structure and evaluates dependencies labelled in part by lexical categories. When a predicate has multiple argument slots that can be filled, the same lexical category is used for the label of multiple dependencies. In this paper, we show that this evaluation can result in disproportionate penalization of supertagging errors and obfuscate the truly erroneous dependencies. Enabled by the compositional nature of CCG lexical categories, we propose decomposed scoring based on subcategorial labels to address this. To evaluate our scoring method, we engage fellow categorial grammar researchers in two English-language judgement tasks: (1) directly ranking the outputs of the standard and experimental scoring methods; and (2) determining which of two sentences has the better parse in cases where the two scoring methods disagree on their ranks. Overall, the judges prefer decomposed scoring in each task; but there is substantial disagreement among the judges in 24% of the given cases, pointing to potential issues with parser evaluations in general.
Do GPTs Produce Less Literal Translations?
Authors: Vikas Raunak; Arul Menezes; Matt Post; Hany Hassan
Venue: ACL 2023 (Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers))

Abstract: Large Language Models (LLMs) such as GPT-3 have emerged as general-purpose language models capable of addressing many natural language generation or understanding tasks. On the task of Machine Translation (MT), multiple works have investigated few-shot prompting mechanisms to elicit better translations from LLMs. However, there has been relatively little investigation on how such translations differ qualitatively from the translations generated by standard Neural Machine Translation (NMT) models. In this work, we investigate these differences in terms of the literalness of translations produced by the two systems. Using literalness measures involving word alignment and monotonicity, we find that translations out of English (E-X) from GPTs tend to be less literal, while exhibiting similar or better scores on MT quality metrics. We demonstrate that this finding is borne out in human evaluations as well. We then show that these differences are especially pronounced when translating sentences that contain idiomatic expressions.
Environmental Claim Detection
Authors: Dominik Stammbach; Nicolas Webersinke; Julia Bingler; Mathias Kraus; Markus Leippold
Venue: ACL 2023 (Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers))

Abstract: To transition to a green economy, environmental claims made by companies must be reliable, comparable, and verifiable. To analyze such claims at scale, automated methods are needed to detect them in the first place. However, there exist no datasets or models for this. Thus, this paper introduces the task of environmental claim detection. To accompany the task, we release an expert-annotated dataset and models trained on this dataset. We preview one potential application of such models: We detect environmental claims made in quarterly earning calls and find that the number of environmental claims has steadily increased since the Paris Agreement in 2015.
Black-box language model explanation by context length probing
Authors: Ondřej Cífka; Antoine Liutkus
Venue: ACL 2023 (Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers))

Abstract: The increasingly widespread adoption of large language models has highlighted the need for improving their explainability. We present *context length probing*, a novel explanation technique for causal language models, based on tracking the predictions of a model as a function of the length of available context, and allowing to assign *differential importance scores* to different contexts. The technique is model-agnostic and does not rely on access to model internals beyond computing token-level probabilities. We apply context length probing to large pre-trained language models and offer some initial analyses and insights, including the potential for studying long-range dependencies. The [source code](https://github.com/cifkao/context-probing/) and an [interactive demo](https://cifkao.github.io/context-probing/) of the method are available.
Let Me Check the Examples: Enhancing Demonstration Learning via Explicit Imitation
Authors: Sirui Wang; Kaiwen Wei; Hongzhi Zhang; Yuntao Li; Wei Wu
Venue: ACL 2023 (Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers))

Abstract: Demonstration learning aims to guide the prompt prediction by providing answered demonstrations in the few shot settings. Despite achieving promising results, existing work only concatenates the answered examples as demonstrations to the prompt template (including the raw context) without any additional operation, neglecting the prompt-demonstration dependencies. Besides, prior research found that randomly replacing the labels of demonstrations marginally hurts performance, illustrating that the model could not properly learn the knowledge brought by the demonstrations. Inspired by the human learning process, in this paper, we introduce Imitation DEMOnstration learning (Imitation-Demo) to strengthen demonstration learning via explicitly imitating human review behaviour, which includes: (1) contrastive learning mechanism to concentrate on similar demonstrations.(2) demonstration-label re-prediction method to consolidate known knowledge. Experiment results show that our proposed method achieves state-of-the-art performance on 5 out of 14 classification corpus. Further studies also prove that Imitation-Demo strengthens the associations between the prompt and demonstrations, which could provide the basis for exploring how demonstration learning works.
The Inside Story: Towards Better Understanding of Machine Translation Neural Evaluation Metrics
Authors: Ricardo Rei; Nuno M. Guerreiro; Marcos Treviso; Luisa Coheur; Alon Lavie; André Martins
Venue: ACL 2023 (Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers))

Abstract: Neural metrics for machine translation evaluation, such as COMET, exhibit significant improvements in their correlation with human judgments, as compared to traditional metrics based on lexical overlap, such as BLEU. Yet, neural metrics are, to a great extent, “black boxes” returning a single sentence-level score without transparency about the decision-making process. In this work, we develop and compare several neural explainability methods and demonstrate their effectiveness for interpreting state-of-the-art fine-tuned neural metrics. Our study reveals that these metrics leverage token-level information that can be directly attributed to translation errors, as assessed through comparison of token-level neural saliency maps with Multidimensional Quality Metrics (MQM) annotations and with synthetically-generated critical translation errors. To ease future research, we release our code at: https://github.com/Unbabel/COMET/tree/explainable-metrics
Typo-Robust Representation Learning for Dense Retrieval
Authors: Panuthep Tasawong; Wuttikorn Ponwitayarat; Peerat Limkonchotiwat; Can Udomcharoenchaikit; Ekapol Chuangsuwanich; Sarana Nutanong
Venue: ACL 2023 (Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers))

Abstract: Dense retrieval is a basic building block of information retrieval applications. One of the main challenges of dense retrieval in real-world settings is the handling of queries containing misspelled words. A popular approach for handling misspelled queries is minimizing the representations discrepancy between misspelled queries and their pristine ones. Unlike the existing approaches, which only focus on the alignment between misspelled and pristine queries, our method also improves the contrast between each misspelled query and its surrounding queries. To assess the effectiveness of our proposed method, we compare it against the existing competitors using two benchmark datasets and two base encoders. Our method outperforms the competitors in all cases with misspelled queries. Our code and models are available at https://github.com/panuthept/DST-DenseRetrieval.
Focused Prefix Tuning for Controllable Text Generation
Authors: Congda Ma; Tianyu Zhao; Makoto Shing; Kei Sawada; Manabu Okumura
Venue: ACL 2023 (Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers))

Abstract: In a controllable text generation dataset, there exist unannotated attributes that could provide irrelevant learning signals to models that use it for training and thus degrade their performance. We propose focused prefix tuning (FPT) to mitigate the problem and to enable the control to focus on the desired attribute. Experimental results show that FPT can achieve better control accuracy and text fluency than baseline models in single-attribute control tasks. In multi-attribute control tasks, FPT achieves comparable control accuracy with the state-of-the-art approach while keeping the flexibility to control new attributes without retraining existing models.
ReAugKD: Retrieval-Augmented Knowledge Distillation For Pre-trained Language Models
Authors: Jianyi Zhang; Aashiq Muhamed; Aditya Anantharaman; Guoyin Wang; Changyou Chen; Kai Zhong; Qingjun Cui; Yi Xu; Belinda Zeng; Trishul Chilimbi; Yiran Chen
Venue: ACL 2023 (Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers))

Abstract: Knowledge Distillation (KD) is one of the most effective approaches to deploying large-scale pre-trained language models in low-latency environments by transferring the knowledge contained in the large-scale models to smaller student models. Prior KD approaches use the soft labels and intermediate activations generated by the teacher to transfer knowledge to the student model parameters alone. In this paper, we show that having access to non-parametric memory in the form of a knowledge base with the teacher’s soft labels and predictions can further improve student generalization. To enable the student to retrieve from the knowledge base effectively, we propose a new framework and loss function that preserves the semantic similarities of teacher and student training examples. We show through extensive experiments that our retrieval mechanism can achieve state-of-the-art performance for task-specific knowledge distillation on the GLUE benchmark.
Debiasing Generative Named Entity Recognition by Calibrating Sequence Likelihood
Authors: Yu Xia; Yongwei Zhao; Wenhao Wu; Sujian Li
Venue: ACL 2023 (Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers))

Abstract: Recognizing flat, overlapped and discontinuous entities uniformly has been paid increasing attention. Among these works, Seq2Seq formulation prevails for its flexibility and effectiveness. It arranges the output entities into a specific target sequence. However, it introduces bias by assigning all the probability mass to the observed sequence. To alleviate the bias, previous works either augment the data with possible sequences or resort to other formulations. In this paper, we stick to the Seq2Seq formulation and propose a reranking-based approach. It redistributes the likelihood among candidate sequences depending on their performance via a contrastive loss. Extensive experiments show that our simple yet effective method consistently boosts the baseline, and yields competitive or better results compared with the state-of-the-art methods on 8 widely-used datasets for Named Entity Recognition.
Deriving Language Models from Masked Language Models
Authors: Lucas Torroba Hennigen; Yoon Kim
Venue: ACL 2023 (Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers))

Abstract: Masked language models (MLM) do not explicitly define a distribution over language, i.e., they are not language models per se. However, recent work has implicitly treated them as such for the purposes of generation and scoring. This paper studies methods for deriving explicit joint distributions from MLMs, focusing on distributions over two tokens, which makes it possible to calculate exact distributional properties. We find that an approach based on identifying joints whose conditionals are closest to those of the MLM works well and outperforms existing Markov random field-based approaches. We further find that this derived model’s conditionals can even occasionally outperform the original MLM’s conditionals.
UniTRec: A Unified Text-to-Text Transformer and Joint Contrastive Learning Framework for Text-based Recommendation
Authors: Zhiming Mao; Huimin Wang; Yiming Du; Kam-Fai Wong
Venue: ACL 2023 (Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers))

Abstract: Prior study has shown that pretrained language models (PLM) can boost the performance of text-based recommendation. In contrast to previous works that either use PLM to encode user history as a whole input text, or impose an additional aggregation network to fuse multi-turn history representations, we propose a unified local- and global-attention Transformer encoder to better model two-level contexts of user history. Moreover, conditioned on user history encoded by Transformer encoders, our framework leverages Transformer decoders to estimate the language perplexity of candidate text items, which can serve as a straightforward yet significant contrastive signal for user-item text matching. Based on this, our framework, UniTRec, unifies the contrastive objectives of discriminative matching scores and candidate text perplexity to jointly enhance text-based recommendation. Extensive evaluation shows that UniTRec delivers SOTA performance on three text-based recommendation tasks.
Reasoning Implicit Sentiment with Chain-of-Thought Prompting
Authors: Hao Fei; Bobo Li; Qian Liu; Lidong Bing; Fei Li; Tat-Seng Chua
Venue: ACL 2023 (Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers))

Abstract: While sentiment analysis systems try to determine the sentiment polarities of given targets based on the key opinion expressions in input texts, in implicit sentiment analysis (ISA) the opinion cues come in an implicit and obscure manner. Thus detecting implicit sentiment requires the common-sense and multi-hop reasoning ability to infer the latent intent of opinion. Inspired by the recent chain-of-thought (CoT) idea, in this work we introduce a Three-hop Reasoning (THOR) CoT framework to mimic the human-like reasoning process for ISA. We design a three-step prompting principle for THOR to step-by-step induce the implicit aspect, opinion, and finally the sentiment polarity. Our THOR+Flan-T5 (11B) pushes the state-of-the-art (SoTA) by over 6% F1 on supervised setup. More strikingly, THOR+GPT3 (175B) boosts the SoTA by over 50% F1 on zero-shot setting.
Latent Positional Information is in the Self-Attention Variance of Transformer Language Models Without Positional Embeddings
Authors: Ta-Chung Chi; Ting-Han Fan; Li-Wei Chen; Alexander Rudnicky; Peter Ramadge
Venue: ACL 2023 (Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers))

Abstract: The use of positional embeddings in transformer language models is widely accepted. However, recent research has called into question the necessity of such embeddings. We further extend this inquiry by demonstrating that a randomly initialized and frozen transformer language model, devoid of positional embeddings, inherently encodes strong positional information through the shrinkage of self-attention variance. To quantify this variance, we derive the underlying distribution of each step within a transformer layer. Through empirical validation using a fully pretrained model, we show that the variance shrinkage effect still persists after extensive gradient updates. Our findings serve to justify the decision to discard positional embeddings and thus facilitate more efficient pretraining of transformer language models.