Reviewerly/exHarmony huggingface dataset:
Sefamerve at SemEval-2023 Task 12: Semantic Evaluation of Rarely Studied Languages

This paper describes our contribution to SemEval-23 Shared Task 12: ArfiSenti.The task consists of several sentiment classification subtasks for rarely studied African languages to predict positive, negative, or neutral classes of a given Twitter dataset.In our system we utilized three different models; FastText, MultiLang Transformers, and Language-Specific Transformers to find the best working model for the classification challenge.We experimented with mentioned models and mostly reached the best prediction scores using the Language Specific Transformers.Our best-submitted result was ranked 3rd among submissions for the Amharic language, obtaining an F1 score of 0.702 behind the second-ranked system.
FactKG: Fact Verification via Reasoning on Knowledge Graphs

In real world applications, knowledge graphs (KG) are widely used in various domains (e.g.medical applications and dialogue agents).However, for fact verification, KGs have not been adequately utilized as a knowledge source.KGs can be a valuable knowledge source in fact verification due to their reliability and broad applicability.A KG consists of nodes and edges which makes it clear how concepts are linked together, allowing machines to reason over chains of topics.However, there are many challenges in understanding how these machine-readable concepts map to information in text.To enable the community to better use KGs, we introduce a new dataset, FACTKG: Fact Verification via Reasoning on Knowledge Graphs.It consists of 108k natural language claims with five types of reasoning: One-hop, Conjunction, Existence, Multi-hop, and Negation.Furthermore, FACTKG contains various linguistic patterns, including colloquial style claims as well as written style claims to increase practicality.Lastly, we develop a baseline approach and analyze FACTKG over these reasoning types.We believe FACTKG can advance both reliability and practicality in KG-based fact verification.
Tracing Linguistic Markers of Influence in a Large Online Organisation

Social science and psycholinguistic research have shown that power and status affect how people use language in a range of domains.Here, we investigate a similar question in a large, distributed, consensus-driven community -the Internet Engineering Task Force (IETF), a collaborative organisation that develops technical standards for the Internet.Our analysis, based on lexical categories (LIWC) and BERT, shows that participants' levels of influence can be predicted from their email text, and identifies key linguistic differences (e.g., certain LIWC categories, such as WE are positively correlated with high-influence).We also identify the differences in language use for the same person before and after becoming influential 1 .
Did You Read the Instructions? Rethinking the Effectiveness of Task Definitions in Instruction Learning

Large language models (LLMs) have shown impressive performance in following natural language instructions to solve unseen tasks.However, it remains unclear whether models truly understand task definitions and whether the human-written definitions are optimal.In this paper, we systematically study the role of task definitions in instruction learning.We first conduct an ablation analysis informed by human annotations to understand which parts of a task definition are most important, and find that model performance only drops substantially when removing contents describing the task output, in particular label information.Next, we propose an automatic algorithm to compress task definitions to a minimal supporting set of tokens, and find that 60% of tokens can be removed while maintaining or even improving model performance.Based on these results, we propose two strategies to help models better leverage task instructions: (1) providing only key information for tasks in a common structured format, and (2) adding a metatuning stage to help the model better understand the definitions.With these two strategies, we achieve a 4.2 Rouge-L improvement over 119 unseen test tasks.
Zero- and Few-Shot Event Detection via Prompt-Based Meta Learning

With emerging online topics as a source for numerous new events, detecting unseen / rare event types presents an elusive challenge for existing event detection methods, where only limited data access is provided for training.To address the data scarcity problem in event detection, we propose MetaEvent, a meta learningbased framework for zero-and few-shot event detection.Specifically, we sample training tasks from existing event types and perform meta training to search for optimal parameters that quickly adapt to unseen tasks.In our framework, we propose to use the clozebased prompt and a trigger-aware soft verbalizer to efficiently project output to unseen event types.Moreover, we design a contrastive meta objective based on maximum mean discrepancy (MMD) to learn class-separating features.As such, the proposed MetaEvent can perform zero-shot event detection by mapping features to event types without any prior knowledge.In our experiments, we demonstrate the effectiveness of MetaEvent in both zero-shot and few-shot scenarios, where the proposed method achieves state-of-the-art performance in extensive experiments on benchmark datasets Few-Event and MAVEN.
Simple Augmentations of Logical Rules for Neuro-Symbolic Knowledge Graph Completion

High-quality and high-coverage rule sets are imperative to the success of Neuro-Symbolic Knowledge Graph Completion (NS-KGC) models, because they form the basis of all symbolic inferences.Recent literature builds neural models for generating rule sets, however, preliminary experiments show that they struggle with maintaining high coverage.In this work, we suggest three simple augmentations to existing rule sets: (1) transforming rules to their abductive forms, (2) generating equivalent rules that use inverse forms of constituent relations and ( 3) random walks that propose new rules.Finally, we prune potentially low quality rules.Experiments over four datasets and five ruleset-baseline settings suggest that these simple augmentations consistently improve results, and obtain up to 7.1 pt MRR and 8.5 pt Hits@1 gains over using rules without augmentations.
Credible without Credit: Domain Experts Assess Generative Language Models

Language models have recently broken into the public consciousness with the release of the wildly popular ChatGPT.Commentators have argued that language models could replace search engines, make college essays obsolete, or even write academic research papers.All of these tasks rely on accuracy of specialized information which can be difficult to assess for non-experts.Using 10 domain experts across science and culture, we provide an initial assessment of the coherence, conciseness, accuracy, and sourcing of two language models across 100 expert-written questions.While we find the results are consistently cohesive and concise, we find that they are mixed in their accuracy.These results raise questions of the role language models should play in generalpurpose and expert knowledge seeking.
SSS at SemEval-2023 Task 10: Explainable Detection of Online Sexism using Majority Voted Fine-Tuned Transformers

This paper describes our submission to Task 10 at SemEval 2023-Explainable Detection of Online Sexism (EDOS) Kirk et al. (2023), divided into three subtasks.The recent rise in social media platforms has seen an increase in disproportionate levels of sexism experienced by women on social media platforms.This has made detecting and explaining online sexist content more important than ever to make social media safer and more accessible for women.Our approach consists of experimenting and finetuning BERT-based models and using a Majority Voting ensemble model that outperforms individual baseline model scores.Our system achieves a macro F1 score of 0.8392 for Task A, 0.6092 for Task B and 0.4319 for Task C and the code is available on GitHub. 1
Chain of Thought Prompting Elicits Knowledge Augmentation

The knowledge-augmented deep learning paradigm refers to a paradigm in which domain knowledge is identified and integrated into deep models.Conventional methods typically employ task-specific approaches to gather external knowledge from various sources.In contrast, large language models are extensively pre-trained and can serve as a comprehensive source of external knowledge.In this paper, we propose CoT-KA, a Chain-of-Thought-based method that augments knowledge for deep learning.CoT-KA avoids the need for additional knowledge retrieval or knowledge reasoning models, as required in conventional augmentation methods.Our results demonstrate that CoT-KA outperforms both pure CoT-based methods and the non-augmented method across the majority of eleven publicly available benchmarks for various reasoning tasks 1 .
Prompt-Based Metric Learning for Few-Shot NER

Few-shot named entity recognition (NER) targets generalizing to unseen labels and/or domains with few labeled examples.Existing metric learning methods compute token-level similarities between query and support sets, but are not able to fully incorporate label semantics into modeling.To address this issue, we propose a simple method to largely improve metric learning for NER: 1) multiple prompt schemas are designed to enhance label semantics; 2) we propose a novel architecture to effectively combine multiple prompt-based representations.Empirically, our method achieves new state-of-the-art (SOTA) results under 16 of the 18 considered settings, substantially outperforming the previous SOTA by an average of 9.12% and a maximum of 34.51% in relative gains of micro F1.Our code is available at https://github.com/AChen-qaq/ProML.
RETUYT-InCo at BEA 2023 Shared Task: Tuning Open-Source LLMs for Generating Teacher Responses

This paper presents the results of our participation in the BEA 2023 shared task, which focuses on generating AI teacher responses in educational dialogues.We conducted experiments using several Open-Source Large Language Models (LLMs) and explored fine-tuning techniques along with prompting strategies, including Few-Shot and Chain-of-Thought approaches.Our best model was ranked 4.5 in the competition with a BertScore F1 of 0.71 and a DialogRPT final (avg) of 0.35.Nevertheless, our internal results did not exactly correlate with those obtained in the competition, which showed the difficulty in evaluating this task.Other challenges we faced were data leakage on the train set and the irregular format of the conversations.
Generate then Select: Open-ended Visual Question Answering Guided by World Knowledge

The open-ended Visual Question Answering (VQA) task requires AI models to jointly reason over visual and natural language inputs using world knowledge.Recently, pre-trained Language Models (PLM) such as GPT-3 have been applied to the task and shown to be powerful world knowledge sources.However, these methods suffer from low knowledge coverage caused by PLM bias -the tendency to generate certain tokens over other tokens regardless of prompt changes, and high dependency on the PLM quality -only models using GPT-3 can achieve the best result.To address the aforementioned challenges, we propose RASO: a new VQA pipeline that deploys a generate-then-select strategy guided by world knowledge for the first time.Rather than following the de facto standard to train a multi-modal model that directly generates the VQA answer, RASO first adopts PLM to generate all the possible answers, and then trains a lightweight answer selection model for the correct answer.As proved in our analysis, RASO expands the knowledge coverage from in-domain training data by a large margin.We provide extensive experimentation and show the effectiveness of our pipeline by advancing the state-of-the-art by +4.1% on OK-VQA, without additional computation cost.Code and models are released at http://cogcomp.org/page/publication_view/1010
Learning Dynamic Contextualised Word Embeddings via Template-based Temporal Adaptation

Dynamic contextualised word embeddings (DCWEs) represent the temporal semantic variations of words.We propose a method for learning DCWEs by time-adapting a pretrained Masked Language Model (MLM) using timesensitive templates.Given two snapshots C 1 and C 2 of a corpus taken respectively at two distinct timestamps T 1 and T 2 , we first propose an unsupervised method to select (a) pivot terms related to both C 1 and C 2 , and (b) anchor terms that are associated with a specific pivot term in each individual snapshot.We then generate prompts by filling manually compiled templates using the extracted pivot and anchor terms.Moreover, we propose an automatic method to learn time-sensitive templates from C 1 and C 2 , without requiring any human supervision.Next, we use the generated prompts to adapt a pretrained MLM to T 2 by fine-tuning using those prompts.Multiple experiments show that our proposed method reduces the perplexity of test sentences in C 2 , outperforming the current state-of-the-art.
History repeats: Overcoming catastrophic forgetting for event-centric temporal knowledge graph completion

Temporal knowledge graph (TKG) completion models typically rely on having access to the entire graph during training.However, in realworld scenarios, TKG data is often received incrementally as events unfold, leading to a dynamic non-stationary data distribution over time.While one could incorporate fine-tuning to existing methods to allow them to adapt to evolving TKG data, this can lead to forgetting previously learned patterns.Alternatively, retraining the model with the entire updated TKG can mitigate forgetting but is computationally burdensome.To address these challenges, we propose a general continual training framework that is applicable to any TKG completion method, and leverages two key ideas: (i) a temporal regularization that encourages repurposing of less important model parameters for learning new knowledge, and (ii) a clustering-based experience replay that reinforces the past knowledge by selectively preserving only a small portion of the past data.Our experimental results on widely used eventcentric TKG datasets demonstrate the effectiveness of our proposed continual training framework in adapting to new events while reducing catastrophic forgetting.Further, we perform ablation studies to show the effectiveness of each component of our proposed framework.Finally, we investigate the relation between the memory dedicated to experience replay and the benefit gained from our clustering-based sampling strategy.
Pathology Dynamics at BioLaySumm: the trade-off between Readability, Relevance, and Factuality in Lay Summarization

Lay summarization aims to simplify complex scientific information for non-expert audiences.This paper investigates the trade-off between readability and relevance in the lay summarization of long biomedical documents.We introduce a two-stage framework that attains the best readability metrics in the first subtask of Bio-LaySumm 2023, with 8.924 Flesch-Kincaid Grade Level and 9.188 Dale-Chall Readability Score.However, this comes at the cost of reduced relevance and factuality, emphasizing the inherent challenges of balancing readability and content preservation in lay summarization.The first stage generates summaries using a large language model, such as BART with LSG attention.The second stage uses a zero-shot sentence simplification method to improve the readability of the summaries.In the second subtask, a hybrid dataset is employed to train a model capable of generating both lay summaries and abstracts.This approach achieves the best readability score and shares the top overall rank with other leading methods.Our study underscores the importance of developing effective methods for creating accessible lay summaries while maintaining information integrity.Future work will integrate simplification and summary generation within a joint optimization framework that generates highquality lay summaries that effectively communicate scientific content to a broader audience.Code: https://github.com/iah3/readability-summarization
Task-Optimized Adapters for an End-to-End Task-Oriented Dialogue System

Task-Oriented Dialogue (TOD) systems are designed to carry out specific tasks by tracking dialogue states and generating appropriate responses to help users achieve defined goals.Recently, end-to-end dialogue models pre-trained based on large datasets have shown promising performance in the conversational system.However, they share the same parameters to train tasks of the dialogue system (NLU, DST, NLG), so debugging each task is challenging.Also, they require a lot of effort to fine-tune large parameters to create a task-oriented chatbot, making it difficult for non-experts to handle.Therefore, we intend to train relatively lightweight and fast models compared to PLM.In this paper, we propose an End-to-end TOD system with Task-Optimized Adapters which learn independently per task, adding only small number of parameters after fixed layers of pretrained network.We also enhance the performance of the DST and NLG modules through reinforcement learning, overcoming the learning curve that has lacked at the adapter learning and enabling the natural and consistent response generation that is appropriate for the goal.Our method is a model-agnostic approach and does not require prompt-tuning as only input data without a prompt.As results of the experiment, our method shows competitive performance on the MultiWOZ benchmark compared to the existing end-to-end models.In particular, we attain state-of-the-art performance on the DST task of 2.2 dataset.
Search Query Spell Correction with Weak Supervision in E-commerce

Misspelled search queries in e-commerce can lead to empty or irrelevant products.Besides inadvertent typing mistakes, most spell mistakes occur because the user does not know the correct spelling, hence typing it as it is pronounced colloquially.This colloquial typing creates countless misspelling patterns for a single correct query.In this paper, we first systematically analyze and group different spell errors into error classes and then leverage the stateof-the-art Transformer model for contextual spell correction.We overcome the constraint of limited human labelled data by proposing novel synthetic data generation techniques for voluminous generation of training pairs needed by data hungry Transformers, without any human intervention.We further utilize weakly supervised data coupled with curriculum learning strategies to improve on tough spell mistakes without regressing on the easier ones.We show significant improvements from our model on human labeled data and online A/B experiments against multiple state-of-art models.
WebCPM: Interactive Web Search for Chinese Long-form Question Answering

Long-form question answering (LFQA) aims at answering complex, open-ended questions with detailed, paragraph-length responses.The de facto paradigm of LFQA necessitates two procedures: information retrieval, which searches for relevant supporting facts, and information synthesis, which integrates these facts into a coherent answer.In this paper, we introduce WebCPM, the first Chinese LFQA dataset.One unique feature of WebCPM is that its information retrieval is based on interactive web search, which engages with a search engine in real time.Following WebGPT (Nakano et al., 2021), we develop a web search interface.We recruit annotators to search for relevant information using our interface and then answer questions.Meanwhile, the web search behaviors of our annotators would be recorded.In total, we collect 5, 500 high-quality questionanswer pairs, together with 15, 372 supporting facts and 125, 954 web search actions.We finetune pre-trained language models to imitate human behaviors for web search and to generate answers based on the collected facts.Our LFQA pipeline, built on these fine-tuned models, generates answers that are no worse than human-written ones in 32.5% and 47.5% of the cases on our dataset and DuReader (He et al., 2018), respectively.The interface, dataset, and codes are publicly available at https: //github.com/thunlp/WebCPM.
EPIC: Multi-Perspective Annotation of a Corpus of Irony

We present EPIC (English Perspectivist Irony Corpus), the first annotated corpus for irony analysis based on the principles of data perspectivism.The corpus contains short conversations from social media in five regional varieties of English, and it is annotated by contributors from five countries corresponding to those varieties.We analyse the resource along the perspectives induced by the diversity of the annotators, in terms of origin, age, and gender, and the relationship between these dimensions, irony, and the topics of conversation.We validate EPIC by creating perspective-aware models that encode the perspectives of annotators grouped according to their demographic characteristics.Firstly, the performance of perspectivist models confirms that different annotators induce very different models.Secondly, in the classification of ironic and non-ironic texts, perspectivist models prove to be generally more confident than the non-perspectivist ones.Furthermore, comparing the performance on a perspective-based test set with those achieved on a gold standard test set, we can observe how perspectivist models tend to detect more precisely the positive class, showing their ability to capture the different perceptions of irony.Thanks to these models, we are moreover able to show interesting insights about the variation in the perception of irony by the different groups of annotators, such as among different generations and nationalities.
Enriching the NArabizi Treebank: A Multifaceted Approach to Supporting an Under-Resourced Language

In this paper we address the scarcity of annotated data for NArabizi, a Romanized form of North African Arabic used mostly on social media, which poses challenges for Natural Language Processing (NLP).We introduce an enriched version of NArabizi Treebank (Seddah et al., 2020) with three main contributions: the addition of two novel annotation layers (named entity recognition and offensive language detection) and a re-annotation of the tokenization, morpho-syntactic and syntactic layers that ensure annotation consistency.Our experimental results, using different tokenization schemes, showcase the value of our contributions and highlight the impact of working with non-gold tokenization for NER and dependency parsing.To facilitate future research, we make these annotations publicly available.Our enhanced NArabizi Treebank paves the way for creating sophisticated language models and NLP tools for this under-represented language.
Nonparametric Decoding for Generative Retrieval

The generative retrieval model depends solely on the information encoded in its model parameters without external memory, its information capacity is limited and fixed. To overcome the limitation, we propose Nonparametric Decoding (Np Decoding) which can be applied to existing generative retrieval models. Np Decoding uses nonparametric contextualized vocab embeddings (external memory) rather than vanilla vocab embeddings as decoder vocab embeddings. By leveraging the contextualized vocab embeddings, the generative retrieval model is able to utilize both the parametric and nonparametric space. Evaluation over 9 datasets (8 single-hop and 1 multi-hop) in the document retrieval task shows that applying Np Decoding to generative retrieval models significantly improves the performance. We also show that Np Decoding is data- and parameter-efficient, and shows high performance in the zero-shot setting.