Reviewerly/exHarmony huggingface dataset:
Sentiment Analysis of Negative Comments on Social Media Using Long Short-Term Memory (LSTM) Method With TensorFlow Framework

There are still many unidentified negative comments on social media that have a negative impact on others' mental and physical well-being. Therefore, sentiment analysis is needed to filter and identify such types of comments, especially on social media. This research aims to analyze and classify unidentified negative comments spread across social media. Sentiment analysis and comment classification are performed using 7773 comments in the Indonesian language. The comments are then visualized using an embedding projector, which gives satisfactory results in classifying words in the comments, where words with negative or positive sentiments are clustered closely together. The model employed in this study is the Long Short Term Memory (LSTM) model, which achieved an accuracy rate of 77.70% and a validation accuracy of 85.20%. The trained model is then used for testing purposes, employing directly collected comments from social media, which give satisfactory results
Performance enhancement of surface codes via recursive minimum-weight perfect-match decoding

The minimum weight perfect matching (MWPM) decoder is the standard decoding strategy for quantum surface codes. However, it suffers a harsh decrease in performance when subjected to biased or nonidentical quantum noise. In this work, we modify the conventional MWPM decoder so that it considers the biases, the nonuniformities, and the relationship between $X$, $Y$, and $Z$ errors of the constituent qubits of a given surface code. Our modified approach, which we refer to as the recursive MWPM decoder, obtains an $18%$ improvement in the probability threshold ${p}_{\text{th}}$ under depolarizing noise. We also obtain significant performance improvements when considering biased noise and independent nonidentically distributed (i.ni.d.) error models derived from measurements performed on state-of-the-art quantum processors. In fact, when subjected to i.ni.d. noise, the recursive MWPM decoder yields a performance improvement of $105.5%$ over the conventional MWPM strategy, and in some cases, it even surpasses the performance obtained over the well-known depolarizing channel.
Developing Energy Access Ontology using Protege Tool

In this internet-based world where a large amount of information exists on the World Wide Web (WWW), the information retrieval systems are mainly centred on topic-based classification. Numerous efforts and time have been consumed for relevant information extraction through major search engines if the data is not organised correctly. Ontologies proved to be an effective technique for representing and retrieving information, which is the key idea in semantic web applications. Ontologies not only help in efficient knowledge representation and information retrieval, but it also helps in mapping the hidden knowledge about a subject. This paper discusses the process and method of building ontology on the “Energy access” domain. The methodology is based on the tools used in developing the ontology. Several tools are used to create an ontology. Protégé is a most popular tool for ontology editing and for developing ontology.1 In this paper, various aspects like super class and subclass hierarchy, creating a subclass, instances for class illustration, query retrieval process, visualisation, and graph views have been demonstrated by using protégé software.
Digital Right Management and Accessibility of Libraries Electronic Resources for Blind and Visually Impaired Users

In this digital era, where digital information is available in various formats, Digital Right Management has emerged as a great technology used by content owners to safeguard their intellectual property digitally. Digital rights management restricts the use of copyrighted digital works from unauthorised access. Although Digital Right Management appears to protect writers’ copyrights, it limits the copyright freedom of blind and visually impaired users who depend on accessibility features. The study aims to discuss and highlight the impact of Digital Right Management on the accessibility of libraries’ electronic resources for BVI users. Furthermore, it discusses how complex Digital Rights Management violates the fair-use rights of BVI users by making it impossible for them to access electronic resources. This review article outlines the key research findings on Digital Right Management hindrances encountered by blind and visually impaired users in accessing electronic resources. The results root in the literature surveyed discloses that the majority of libraries throughout the world are falling behind in delivering efficient access to electronic resources for BVI users. Libraries have a variety of obstacles to the accessibility of electronic content, including Digital Right Management challenges, license constraints, website design limitations, accessibility issues, and compatibility issues with assistive software. To overcome some of these barriers, librarians should review the implications of Digital Right Management for accessibility. Moreover, librarians must be knowledgeable about the numerous forms of visual impairments, including these users in accessibility studies, and consider the assistive technology they use. The article will review the literature on the following topics: find out how accessible electronic resources in libraries are for blind or visually impaired users; identify existing accessibility challenges arising from Digital Right Management technologies that impede access to electronic resources by blind or visually impaired users, and identify the challenges that they are facing.
Multi-armed bandits for resource efficient, online optimization of language model pre-training: the use case of dynamic masking

We design and evaluate a Bayesian optimization framework for resource efficient pretraining of Transformer-based language models (TLMs).TLM pre-training requires high computational resources and introduces many unresolved design choices, such as selecting its pre-training hyperparameters.We propose a multi-armed bandit framework for the sequential selection of pre-training hyperparameters, aimed at optimizing language model performance, in a resource efficient manner.We design a Thompson sampling algorithm, with a surrogate Gaussian process reward model of the Masked Language Model (MLM) pretraining objective, for its sequential minimization.Instead of MLM pre-training with fixed masking probabilities, the proposed Gaussian process-based Thompson sampling (GP-TS) accelerates pre-training by sequentially selecting masking hyperparameters that improve performance.We empirically demonstrate how GP-TS pre-trains language models efficiently, i.e., it achieves lower MLM loss in fewer epochs, across a variety of settings.In addition, GP-TS pre-trained TLMs attain competitive downstream performance, while avoiding expensive hyperparameter grid search.GP-TS provides an interactive framework for efficient and optimized TLM pre-training that, by circumventing costly hyperparameter selection, enables substantial computational savings.
Learn over Past, Evolve for Future: Forecasting Temporal Trends for Fake News Detection

Fake news detection has been a critical task for maintaining the health of the online news ecosystem.However, very few existing works consider the temporal shift issue caused by the rapidly-evolving nature of news data in practice, resulting in significant performance degradation when training on past data and testing on future data.In this paper, we observe that the appearances of news events on the same topic may display discernible patterns over time, and posit that such patterns can assist in selecting training instances that could make the model adapt better to future data.Specifically, we design an effective framework FTT (Forecasting Temporal Trends), which could forecast the temporal distribution patterns of news data and then guide the detector to fast adapt to future distribution.Experiments on the real-world temporally split dataset demonstrate the superiority of our proposed framework.The code is available at https://github.com/ICTMCG/FTT-ACL23.
GanLM: Encoder-Decoder Pre-training with an Auxiliary Discriminator

Pre-trained models have achieved remarkable success in natural language processing (NLP).However, existing pre-training methods underutilize the benefits of language understanding for generation.Inspired by the idea of Generative Adversarial Networks (GANs), we propose a GAN-style model for encoder-decoder pretraining by introducing an auxiliary discriminator, unifying the ability of language understanding and generation in a single model.Our model, named as GANLM, is trained with two pre-training objectives: replaced token detection and replaced token denoising.Specifically, given masked source sentences, the generator outputs the target distribution and the discriminator predicts whether the target sampled tokens from distribution are incorrect.The target sentence is replaced with misclassified tokens to construct noisy previous context, which is used to generate the gold sentence.In general, both tasks improve the ability of language understanding and generation by selectively using the denoising data.Extensive experiments in language generation benchmarks show that GANLM with the powerful language understanding capability outperforms various strong pre-trained language models (PLMs) and achieves state-of-the-art performance.
One Cannot Stand for Everyone! Leveraging Multiple User Simulators to train Task-oriented Dialogue Systems

User simulators are agents designed to imitate human users; recent advances have found that Task-oriented Dialogue (ToD) systems optimized toward a user simulator could better satisfy the need of human users.However, this might result in a sub-optimal ToD system if it is tailored to only one ad hoc user simulator, since human users can behave differently.In this paper, we propose a framework called MUST 1 to optimize ToD systems via leveraging Multiple User SimulaTors.
On Dataset Transferability in Active Learning for Transformers

Active learning (AL) aims to reduce labeling costs by querying the examples most beneficial for model learning.While the effectiveness of AL for fine-tuning transformer-based pre-trained language models (PLMs) has been demonstrated, it is less clear to what extent the AL gains obtained with one model transfer to others.We consider the problem of transferability of actively acquired datasets in text classification and investigate whether AL gains persist when a dataset built using AL coupled with a specific PLM is used to train a different PLM.We link the AL dataset transferability to the similarity of instances queried by the different PLMs and show that AL methods with similar acquisition sequences produce highly transferable datasets regardless of the models used.Additionally, we show that the similarity of acquisition sequences is influenced more by the choice of the AL method than the choice of the model.
Detoxifying Text with MaRCo: Controllable Revision with Experts and Anti-Experts

Text detoxification has the potential to mitigate the harms of toxicity by rephrasing text to remove offensive meaning, but subtle toxicity remains challenging to tackle.We introduce MARCO, a detoxification algorithm that combines controllable generation and text rewriting methods using a Product of Experts with autoencoder language models (LMs).MARCO uses likelihoods under a non-toxic LM (expert) and a toxic LM (anti-expert) to find candidate words to mask and replace.We evaluate our method on several subtle toxicity and microaggressions datasets, and show that it not only outperforms baselines on automatic metrics, but MARCO's rewrites are preferred 2.1× more in human evaluation.Its applicability to instances of subtle toxicity is especially promising, demonstrating a path forward for addressing increasingly elusive online hate.the quality of online conversations (e.g., through machine-in-the-loop interfaces; Hohenstein et al., 2021;Clark et al., 2018).We present MARCO, Mask and Replace with Context: a new, unsupervised algorithm for text detoxification that combines mask-and-replace text
Scaling Laws Under the Microscope: Predicting Transformer Performance from Small Scale Experiments

Neural scaling laws define a predictable relationship between a model's parameter count and its performance after training in the form of a power law.However, most research to date has not explicitly investigated whether scaling laws can be used to accelerate model development.In this work, we perform such an empirical investigation across a wide range of language understanding tasks, starting from models with as few as 10K parameters, and evaluate downstream performance across 9 language understanding tasks.We find that scaling laws emerge at finetuning time in some NLP tasks, and that they can also be exploited for debugging convergence when training large models.Moreover, for tasks where scaling laws exist, they can be used to predict the performance of larger models, which enables effective model selection.However, revealing scaling laws requires careful hyperparameter tuning and multiple runs for the purpose of uncertainty estimation, which incurs additional overhead, partially offsetting the computational benefits.
A Knowledge-Graph-Based Intrinsic Test for Benchmarking Medical Concept Embeddings and Pretrained Language Models

Using language models created from large data sources has improved the performance of several deep learning-based architectures, obtaining state-of-the-art results in several NLP extrinsic tasks.However, little research is related to creating intrinsic tests that allow us to compare the quality of different language models when obtaining contextualized embeddings.This gap increases even more when working on specific domains in languages other than English.This paper proposes a novel graph-based intrinsic test that allows us to measure the quality of different language models in clinical and biomedical domains in Spanish.Our results show that our intrinsic test performs better for clinical and biomedical language models than a general one.Also, it correlates with better outcomes for a NER task using a probing model over contextualized embeddings.We hope our work will help the clinical NLP research community to evaluate and compare new language models in other languages and find the most suitable models for solving downstream tasks.
Learning Cooperative Interactions for Multi-Overlap Aspect Sentiment Triplet Extraction

Aspect sentiment triplet extraction (ASTE) is an essential task, which aims to extract triplets(aspect, opinion, sentiment). However, overlapped triplets, especially multi-overlap triplets,make ASTE a challenge. Most existing methods suffer from multi-overlap triplets becausethey focus on the single interactions between an aspect and an opinion. To solve the aboveissues, we propose a novel multi-overlap triplet extraction method, which decodes the complexrelations between multiple aspects and opinions by learning their cooperative interactions. Overall, the method is based on an encoder-decoder architecture. During decoding, we design ajoint decoding mechanism, which employs a multi-channel strategy to generate aspects andopinions through the cooperative interactions between them jointly. Furthermore, we constructa correlation-enhanced network to reinforce the interactions between related aspectsand opinions for sentiment prediction. Besides, a relation-wise calibration scheme is adoptedto further improve performance. Experiments show that our method outperforms baselines,especially multi-overlap triplets.
Meaning Representations for Natural Languages: Design, Models and Applications

This tutorial reviews the design of common meaning representations, SoTA models for predicting meaning representations, and the applications of meaning representations in a wide range of downstream NLP tasks and real-world applications. Reporting by a diverse team of NLP researchers from academia and industry with extensive experience in designing, building and using meaning representations, our tutorial has three components: (1) an introduction to common meaning representations, including basic concepts and design challenges; (2) a review of SoTA methods on building models for meaning representations; and (3) an overview of applications of meaning representations in downstream NLP tasks and real-world applications. We will also present qualitative comparisons of common meaning representations and a quantitative study on how their differences impact model performance. Finally, we will share best practices in choosing the right meaning representation for downstream tasks.
Arabic Dialect Identification and Sentiment Classification using Transformer-based Models

In this paper, we present two deep learning approaches that are based on AraBERT, submitted to the Nuanced Arabic Dialect Identification (NADI) shared task of the Seventh Workshop for Arabic Natural Language Processing (WANLP 2022).NADI consists of two main sub-tasks, mainly country-level dialect and sentiment identification for dialectical Arabic.We present one system per sub-task.The first system is a multi-task learning model that consists of a shared AraBERT encoder with three taskspecific classification layers.This model is trained to jointly learn the country-level dialect of the tweet as well as the region-level and arealevel dialects.The second system is a distilled model of an ensemble of models trained using K-fold cross-validation.Each model in the ensemble consists of an AraBERT model and a classifier, fine-tuned on (K-1) folds of the training set.Our team Pythoneers achieved rank 6 on the first test set of the first sub-task, rank 9 on the second test set of the first sub-task, and rank 4 on the test set of the second sub-task.
Prompt Compression and Contrastive Conditioning for Controllability and Toxicity Reduction in Language Models

We explore the idea of compressing the prompts used to condition language models, and show that compressed prompts can retain a substantive amount of information about the original prompt.For severely compressed prompts, while fine-grained information is lost, abstract information and general sentiments can be retained with surprisingly few parameters, which can be useful in the context of decodetime algorithms for controllability and toxicity reduction.We explore contrastive conditioning to steer language model generation towards desirable text and away from undesirable text, and find that some complex prompts can be effectively compressed into a single token to guide generation.We also show that compressed prompts are largely compositional, and can be constructed such that they can be used to control independent aspects of generated text.
Effective Pretraining Objectives for Transformer-based Autoencoders

In this paper, we study trade-offs between efficiency, cost and accuracy when pretraining Transformer encoders with different pre-training objectives.For this purpose, we analyze features of common objectives and combine them to create new effective pre-training approaches.Specifically, we designed light token generators based on a straightforward statistical approach, which can replace ELEC-TRA computationally heavy generators, thus highly reducing cost.Our experiments also show that (i) there are more efficient alternatives to BERT's MLM, and (ii) it is possible to efficiently pre-train Transformer-based models using lighter generators without a significant drop in performance.
Improving Large-Scale Conversational Assistants using Model Interpretation based Training Sample Selection

Natural language understanding (NLU) models are a core component of large-scale conversational assistants.Collecting training data for these models through manual annotations is slow and expensive that impedes the pace of model improvement.We present a three stage approach to address this challenge: First, we identify a large set of relatively infrequent utterances from live traffic where the users implicitly communicated satisfaction with a response (such as by not interrupting), along with the existing model outputs as candidate annotations.Second, we identify a small subset of these utterances usings Integrated Gradients based importance scores computed with the current models.Finally, we augment our training sets with these utterances and retrain our models.We demonstrate the effectiveness of our approach in a large-scale conversational assistant, processing billions of utterances every week.By augmenting our training set with just 0.05% more utterances through our approach, we observe statistically significant improvements for infrequent tail utterances: a 0.45% reduction in semantic error rate (Se-mER) in offline experiments, and a 1.23% reduction in defect rates in online A/B tests.
Discourse-Aware Soft Prompting for Text Generation

Current efficient fine-tuning methods (e.g., adapters (Houlsby et al., 2019), prefix-tuning (Li and Liang, 2021), etc.) have optimized conditional text generation via training a small set of extra parameters of the neural language model, while freezing the rest for efficiency.While showing strong performance on some generation tasks, they don't generalize across all generation tasks.We show that soft-prompt based conditional text generation can be improved with simple and efficient methods that simulate modeling the discourse structure of human written text.We investigate two design choices: First, we apply hierarchical blocking on the prefix parameters to simulate a higherlevel discourse structure of human written text.Second, we apply attention sparsity on the prefix parameters at different layers of the network and learn sparse transformations on the softmax-function.We show that structured design of prefix parameters yields more coherent, faithful and relevant generations than the baseline prefix-tuning on all generation tasks.
Towards Generalizable and Robust Text-to-SQL Parsing

Text-to-SQL parsing tackles the problem of mapping natural language questions to executable SQL queries.In practice, text-to-SQL parsers often encounter various challenging scenarios, requiring them to be generalizable and robust.While most existing work addresses a particular generalization or robustness challenge, we aim to study it in a more comprehensive manner.In specific, we believe that text-to-SQL parsers should be (1) generalizable at three levels of generalization, namely i.i.d., zero-shot, and compositional, and (2) robust against input perturbations.To enhance these capabilities of the parser, we propose a novel TKK framework consisting of Task decomposition, Knowledge acquisition, and Knowledge composition to learn text-to-SQL parsing in stages.By dividing the learning process into multiple stages, our framework improves the parser's ability to acquire general SQL knowledge instead of capturing spurious patterns, making it more generalizable and robust.Experimental results under various generalization and robustness settings show that our framework is effective in all scenarios and achieves stateof-the-art performance on the Spider, SParC, and CoSQL datasets.
Is anisotropy really the cause of BERT embeddings not being semantic?

In this paper we conduct a set of experiments aimed to improve our understanding of the lack of semantic isometry in BERT, i.e. the lack of correspondence between the embedding and meaning spaces of its contextualized word representations.Our empirical results show that, contrary to popular belief, the anisotropy is not the root cause of the poor performance of these contextual models' embeddings in semantic tasks.What does affect both the anisotropy and semantic isometry is a set of known biases: frequency, subword, punctuation, and case.For each one of them, we measure its magnitude and the effect of its removal, showing that these biases contribute but do not completely explain the phenomenon of anisotropy and lack of semantic isometry of these contextual language models.