Reviewerly/exHarmony huggingface dataset:
Language Model Pre-Training with Sparse Latent Typing

Modern large-scale Pre-trained Language Models (PLMs) have achieved tremendous success on a wide range of downstream tasks.However, most of the LM pre-training objectives only focus on text reconstruction, but have not sought to learn latent-level interpretable representations of sentences.In this paper, we manage to push the language models to obtain a deeper understanding of sentences by proposing a new pre-training objective, Sparse Latent Typing, which enables the model to sparsely extract sentence-level keywords with diverse latent types.Experimental results show that our model is able to learn interpretable latent type categories in a self-supervised manner without using any external knowledge.Besides, the language model pre-trained with such an objective also significantly improves Information Extraction related downstream tasks in both supervised and few-shot settings.Our code is publicly available at https://github.com/renll/SparseLT.* Equal contribution.Listing order is random.Liliang proposed and implemented the architecture designs and the training objectives of Sparse Latent Typing (SLT), and he also conducted extensive experiments for pre-training, few-shot evaluation and the analyses.Zixuan designed the language model pre-training pipeline for SLT, built the initial training codebase and conducted experiments for pre-training and supervised evaluation.Both of the authors initially came up with the same project goal of encouraging the model to sparsely select sentence-level key words during pre-training.
PENERAPAN MANAJEMEN SEBAGAI UPAYA PEMBENTUKAN PEMUSTAKA LITERATE DI RUMAH BACA REGENSI PINANG GRAHA RAYA

The main purpose of this community service activity is to form an information literate society and become literate library users. The library management training at Rumah Baca Badut Syari'ah aims to attract community empathy to preserve a positive culture in receiving information so that alliteration does not occur. Rumah Baca Badut Syari'ah is an appropriate facility and has maximum benefit if it is managed professionally. The Reading House is like a library in its growth and its existence in the midst of society is a tool to increase interest and more than that TBM also functions as an effort to carry out educational activities and fulfill the needs of reading citizens in general and community reading gardens are the heart of community education by Various kinds of programs and services in it are expected to be able to motivate and foster interest and penchant for reading for. With the development of interest and fondness for reading, reading is a habit that must be practiced every day as it meets the needs of life. The methods of implementing the activities include lectures, question and answer methods, and simulation / training. The expected result of PKM activities is to shape the community to become information literate and literate library users.
M2D2: A Massively Multi-Domain Language Modeling Dataset

We present M2D2, a fine-grained, massively multi-domain corpus for studying domain adaptation in language models (LMs).M2D2 consists of 8.5B tokens and spans 145 domains extracted from Wikipedia and Semantic Scholar.Using ontologies derived from Wikipedia and ArXiv categories, we organize the domains in each data source into 22 groups.This two-level hierarchy enables the study of relationships between domains and their effects on in-and out-of-domain performance after adaptation.We also present a number of insights into the nature of effective domain adaptation in LMs, as examples of the new types of studies M2D2 enables.To improve in-domain performance, we show the benefits of adapting the LM along a domain hierarchy; adapting to smaller amounts of fine-grained domainspecific data can lead to larger in-domain performance gains than larger amounts of weakly relevant data.We further demonstrate a tradeoff between in-domain specialization and outof-domain generalization within and across ontologies, as well as a strong correlation between out-of-domain performance and lexical overlap between domains.
NSUT-NLP at CASE 2022 Task 1: Multilingual Protest Event Detection using Transformer-based Models

Event detection, specifically in the sociopolitical domain, has posed a long-standing challenge to researchers in the NLP domain.Therefore, the creation of automated techniques that perform classification of the large amounts of accessible data on the Internet becomes imperative.This paper is a summary of the efforts we made in participating in Task 1 of CASE 2022.We use state-of-art multilingual BERT (mBERT) with further fine-tuning to perform document classification in English, Portuguese, Spanish, Urdu, Hindi, Turkish and Mandarin.In the document classification subtask, we were able to achieve F1 scores of 0.8062, 0.6445, 0.7302, 0.5671, 0.6555, 0.7545 and 0.6702 in English, Spanish, Portuguese, Hindi, Urdu, Mandarin and Turkish respectively achieving a rank of 5 in English and 7 on the remaining language tasks.
ClassBases at the CASE-2022 Multilingual Protest Event Detection Task: Multilingual Protest News Detection and Automatically Replicating Manually Created Event Datasets

In this report, we describe our ClassBases submissions to a shared task on multilingual protest event detection. For the multilingual protest news detection, we participated in subtask-1, subtask-2 and subtask-4 which are document classification, sentence classification and token classification. In subtask-1, we compare XLM-RoBERTa-base, mLUKE-base and XLM-RoBERTa-large on finetuning in a sequential classification setting. We always use a combination of the training data from every language provided to train our multilingual models. We found that larger models seem to work better and entity knowledge helps but at a non-negligible cost. For subtask-2, we only submitted an mLUKE-base system for sentence classification. For subtask-4, we only submitted an XLM-RoBERTa-base for token classification system for sequence labeling. For automatically replicating manually created event datasets, we participated in COVID-related protest events from the New York Times news corpus. We created a system to process the crawled data into a dataset of protest events.
SEM-F1: an Automatic Way for Semantic Evaluation of Multi-Narrative Overlap Summaries at Scale

Recent work has introduced an important yet relatively under-explored NLP task called Semantic Overlap Summarization (SOS) that entails generating a summary from multiple alternative narratives which conveys the common information provided by those narratives.Previous work also published a benchmark dataset for this task by collecting 2, 925 alternative narrative pairs from the web and manually annotating 411 different reference summaries by engaging human annotators.In this paper, we exclusively focus on the automated evaluation of the SOS task using the benchmark dataset.More specifically, we first use the popular ROUGE metric from text-summarization literature and conduct a systematic study to evaluate the SOS task.Our experiments discover that ROUGE is not suitable for this novel task and therefore, we propose a new sentencelevel precision-recall style automated evaluation metric, called SEM-F 1 (Semantic F 1 ).It is inspired by the benefits of the sentence-wise annotation technique using overlap labels reported by the previous work.Our experiments show that the proposed SEM-F 1 metric yields a higher correlation with human judgment and higher inter-rater agreement compared to the ROUGE metric.
On The Arabic Dialects’ Identification: Overcoming Challenges of Geographical Similarities Between Arabic dialects and Imbalanced Datasets

Arabic is one of the world's richest languages, with a diverse range of dialects based on geographical origin.In this paper, we present a solution to tackle subtask 1 (Country-level dialect identification) of the Nuanced Arabic Dialect Identification (NADI) shared task 2022 achieving third place with an average macro F1 score between the two test sets of 26.44%.In the preprocessing stage, we removed the most common frequent terms from all sentences across all dialects, and in the modeling step, we employed a hybrid loss function approach that includes Weighted cross entropy loss and Vector Scaling(VS) Loss.On test sets A and B, our model achieved 35.68% and 17.192% Macro F1 scores, respectively.
CEIA-NLP at CASE 2022 Task 1: Protest News Detection for Portuguese

This paper summarizes our work on the document classification subtask of Multilingual protest news detection of the CASE @ ACL-IJCNLP 2022 workshop.In this context, we investigate the performance of monolingual and multilingual transformer-based models in low data resources, taking Portuguese as an example and evaluating language models on document classification.Our approach became the winning solution in Portuguese document classification achieving 0.8007 F1 Score on Test set.The experimental results demonstrate that multilingual models achieves best results than monolingual models in scenarios with few dataset samples of specific language, because we can train models using datasets from other languages of the same task and domain.
Syntactic Multi-view Learning for Open Information Extraction

Open Information Extraction (OpenIE) aims to extract relational tuples from open-domain sentences.Traditional rule-based or statistical models have been developed based on syntactic structures of sentences, identified by syntactic parsers.However, previous neural Ope-nIE models under-explore the useful syntactic information.In this paper, we model both constituency and dependency trees into wordlevel graphs, and enable neural OpenIE to learn from the syntactic structures.To better fuse heterogeneous information from both graphs, we adopt multi-view learning to capture multiple relationships from them.Finally, the finetuned constituency and dependency representations are aggregated with sentential semantic representations for tuple generation.Experiments show that both constituency and dependency information, and the multi-view learning are effective.Our model is publicly available.
Differentially Private Language Models for Secure Data Sharing

To protect the privacy of individuals whose data is being shared, it is of high importance to develop methods allowing researchers and companies to release textual data while providing formal privacy guarantees to its originators.In the field of NLP, substantial efforts have been directed at building mechanisms following the framework of local differential privacy, thereby anonymizing individual text samples before releasing them.In practice, these approaches are often dissatisfying in terms of the quality of their output language due to the strong noise required for local differential privacy.In this paper, we approach the problem at hand using global differential privacy, particularly by training a generative language model in a differentially private manner and consequently sampling data from it.Using natural language prompts and a new prompt-mismatch loss, we are able to create highly accurate and fluent textual datasets taking on specific desired attributes such as sentiment or topic and resembling statistical properties of the training data.We perform thorough experiments indicating that our synthetic datasets do not leak information from our original data and are of high language quality and highly suitable for training models for further analysis on real-world data.Notably, we also demonstrate that training classifiers on private synthetic data outperforms directly training classifiers on real data with DP-SGD.
Improving POS Tagging for Arabic Dialects on Out-of-Domain Texts

We investigate part of speech tagging for four Arabic dialects (Gulf, Levantine, Egyptian, and Maghrebi), in an out-ofdomain setting.More specifically, we look at the effectiveness of 1) upsampling the target dialect in the training data of a joint model, 2) increasing the consistency of the annotations, and 3) using word embeddings pre-trained on a large corpus of dialectal Arabic.We increase the accuracy on average by about 20 percentage points.
Controlled Text Reduction

Producing a reduced version of a source text, as in generic or focused summarization, inherently involves two distinct subtasks: deciding on targeted content and generating a coherent text conveying it.While some popular approaches address summarization as a single end-to-end task, prominent works support decomposed modeling for individual subtasks.Further, semi-automated text reduction is also very appealing, where users may identify targeted content while models would generate a corresponding coherent summary.In this paper, we focus on the second subtask, of generating coherent text given pre-selected content.Concretely, we formalize Controlled Text Reduction as a standalone task, whose input is a source text with marked spans of targeted content ("highlighting").A model then needs to generate a coherent text that includes all and only the target information.We advocate the potential of such models, both for modular fully-automatic summarization, as well as for semi-automated human-in-the-loop use cases.Facilitating proper research, we crowdsource high-quality dev and test datasets for the task.Further, we automatically generate a larger "silver" training dataset from available summarization benchmarks, leveraging a pretrained summary-source alignment model.Finally, employing these datasets, we present a supervised baseline model, showing promising results and insightful analyses. 1
FigurativeQA: A Test Benchmark for Figurativeness Comprehension for Question Answering

Figurative language is widespread in human language (Lakoff and Johnson, 2008), posing potential challenges in NLP applications.In this paper, we investigate the effect of figurative language on the task of question answering (QA).We construct FigurativeQA, a test set of 400 yes-no questions with figurative and nonfigurative contexts, extracted from product reviews and restaurant reviews.We demonstrate that a state-of-the-art RoBERTa QA model has considerably lower performance in question answering when the contexts are figurative rather than literal, indicating a gap in current models.We propose a general method for improving the performance of QA models by converting the figurative contexts into non-figurative by prompting GPT-3, and demonstrate its effectiveness.Our results indicate a need for building QA models infused with figurative language understanding capabilities.
Offline-to-Online Co-Evolutional User Simulator and Dialogue System

Reinforcement learning (RL) has emerged as a promising approach to fine-tune offline pretrained GPT-2 model in task-oriented dialogue (TOD) systems.In order to obtain human-like online interactions while extending the usage of RL, building pretrained user simulators (US) along with dialogue systems (DS) and facilitating jointly fine-tuning via RL becomes prevalent.However, joint training brings distributional shift problem caused by compounding exposure bias.Existing methods usually iterative update US and DS to ameliorate the ensued non-stationarity problem, which could lead to sub-optimal policy and less sample efficiency.To take a step further for tackling the problem, we introduce an Offline-to-oNline Co-Evolutional (ONCE) framework, which enables bias-aware concurrent joint update for RL-based fine-tuning whilst takes advantages from GPT-2 based end-to-end modeling on US and DS.Extensive experiments demonstrate that ONCE builds high-quality loops of policy learning and dialogues data collection, and achieves state-of-the-art online and offline evaluation results on MultiWOZ2.1 dataset.Opensourced code will be implemented with Mindspore (MS, 2022) and released on our homepage 1 .
Attention weights accurately predict language representations in the brain

In Transformer-based language models (LMs) the attention mechanism converts token embeddings into contextual embeddings that incorporate information from neighboring words.The resulting contextual hidden state embeddings have enabled highly accurate models of brain responses, suggesting that the attention mechanism constructs contextual embeddings that carry information reflected in language-related brain representations.However, it is unclear whether the attention weights that are used to integrate information across words are themselves related to language representations in the brain.To address this question we analyzed functional magnetic resonance imaging (fMRI) recordings of participants reading English language narratives.We provided the narrative text as input to two LMs (BERT and GPT-2) and extracted their corresponding attention weights.We then used encoding models to determine how well attention weights can predict recorded brain responses.We find that attention weights accurately predict brain responses in much of the frontal and temporal cortices.Our results suggest that the attention mechanism itself carries information that is reflected in brain representations.Moreover, these results indicate cortical areas in which context integration may occur.
Can Visual Context Improve Automatic Speech Recognition for an Embodied Agent?

The usage of automatic speech recognition (ASR) systems are becoming omnipresent ranging from personal assistant to chatbots, home, and industrial automation systems, etc. Modern robots are also equipped with ASR capabilities for interacting with humans as speech is the most natural interaction modality.However, ASR in robots faces additional challenges as compared to a personal assistant.Being an embodied agent, a robot must recognize the physical entities around it and therefore reliably recognize the speech containing the description of such entities.However, current ASR systems are often unable to do so due to limitations in ASR training, such as generic datasets and open-vocabulary modeling.Also, adverse conditions during inference, such as noise, accented, and far-field speech makes the transcription inaccurate.In this work, we present a method to incorporate a robot's visual information into an ASR system and improve the recognition of a spoken utterance containing a visible entity.Specifically, we propose a new decoder biasing technique to incorporate the visual context while ensuring the ASR output does not degrade for incorrect context.We achieve a 59% relative reduction in WER from an unmodified ASR system.
Parameter-free Automatically Prompting: A Latent Pseudo Label Mapping Model for Prompt-based Learning

Prompt-based learning has achieved excellent performance in few-shot learning by mapping the outputs of the pre-trained language model to the labels with the help of a label mapping component.Existing manual label mapping (MLM) methods achieve good results but heavily rely on expensive human knowledge.Automatic label mapping (ALM) methods that learn the mapping functions with extra parameters have shown their potentiality.However, no effective ALM model comparable to MLM methods is developed yet due to the limited data.In this paper, we propose a Latent Pseudo Label Mapping (LPLM) method that optimizes the label mapping without human knowledge and extra parameters.LPLM is built upon a probabilistic latent model and is iteratively self-improved with the EM-style algorithm.The empirical results demonstrate that our LPLM method is superior to the mainstream ALM methods and significantly outperforms the SOTA method in few-shot classification tasks.Moreover, LPLM also shows impressively better performance than the vanilla MLM method which requires extra task-specific prior knowledge.
What Has Been Enhanced in my Knowledge-Enhanced Language Model?

A number of knowledge integration (KI) methods have recently been proposed to incorporate external knowledge into pretrained language models (LMs).Even though knowledgeenhanced LMs outperform base LMs on knowledge-intensive tasks, the inner-workings of these KI methods are not well-understood.For instance, it is unclear which knowledge is effectively integrated into knowledge-enhanced LMs and which is not; and if such integration leads to catastrophic forgetting of already learned knowledge.We show that existing model interpretation methods such as linear probes and prompts have some key limitations in answering these questions.We revisit KI from an information-theoretic view and propose a new theoretically sound probe called Graph Convolution Simulator (GCS) for KI interpretation.GCS uses graph attention on the corresponding knowledge graph for interpretation.In our experiments we verify that GCS can provide reasonable interpretation results for two well-known knowledge-enhanced LMs: ERNIE and K-Adapter.We also find that only a marginal amount of knowledge is successfully integrated in these models, and simply increasing the size of the KI corpus may not lead to better knowledge-enhanced LMs.
Revisiting text decomposition methods for NLI-based factuality scoring of summaries

Scoring the factuality of a generated summary involves measuring the degree to which a target text contains factual information using the input document as support.Given the similarities in the problem formulation, previous work has shown that Natural Language Inference models can be effectively repurposed to perform this task.As these models are trained to score entailment at a sentence level, several recent studies have shown that decomposing either the input document or the summary into sentences helps with factuality scoring.But is fine-grained decomposition always a winning strategy?In this paper we systematically compare different granularities of decomposition -from document to sub-sentence level, and we show that the answer is no.Our results show that incorporating additional context can yield improvement, but that this does not necessarily apply to all datasets.We also show that small changes to previously proposed entailment-based scoring methods can result in better performance, highlighting the need for caution in model and methodology selection for downstream tasks.
Sparse Mixers: Combining MoE and Mixing to build a more efficient BERT

We combine the capacity of sparsely gated Mixture-of-Experts (MoE) with the speed and stability of linear, mixing transformations to design the Sparse Mixer encoder model.Sparse Mixer slightly outperforms BERT on GLUE and SuperGLUE, but more importantly trains 65% faster and runs inference 61% faster.We also present a faster variant, prosaically named Fast Sparse Mixer, that marginally underperforms BERT on Su-perGLUE, but trains and runs nearly twice as fast.We justify the design of these two models by carefully ablating through various mixing mechanisms, MoE configurations, and hyperparameters.Sparse Mixer overcomes many of the latency and stability concerns of MoE models and offers the prospect of serving sparse student models, without resorting to distilling them to dense variants.