Reviewerly/exHarmony huggingface dataset:
Blended Latent Diffusion

The tremendous progress in neural image generation, coupled with the emergence of seemingly omnipotent vision-language models has finally enabled text-based interfaces for creating and editing images. Handling generic images requires a diverse underlying generative model, hence the latest works utilize diffusion models, which were shown to surpass GANs in terms of diversity. One major drawback of diffusion models, however, is their relatively slow inference time. In this paper, we present an accelerated solution to the task of local text-driven editing of generic images, where the desired edits are confined to a user-provided mask. Our solution leverages a text-to-image Latent Diffusion Model (LDM), which speeds up diffusion by operating in a lower-dimensional latent space and eliminating the need for resource-intensive CLIP gradient calculations at each diffusion step. We first enable LDM to perform local image edits by blending the latents at each step, similarly to Blended Diffusion. Next we propose an optimization-based solution for the inherent inability of LDM to accurately reconstruct images. Finally, we address the scenario of performing local edits using thin masks. We evaluate our method against the available baselines both qualitatively and quantitatively and demonstrate that in addition to being faster, it produces more precise results.
Sistem Informasi Online Pengelolaan Dana Sosial Pada Rumah Yatim Sumatera Utara

Rumah Yatim, a National Zakat Management Institution under BAZNAS, manages various forms of donations across Indonesia through 20 branch offices. However, data management using Microsoft Excel has drawbacks, which led to the creation of an Online Social Fund Management Information System in North Sumatra using MySQL, PHP, HTML, and the Waterfall method. The system ensures transparent and systematic data management, making it easier for donors to donate online and providing real-time information on donation management. The new system has allowed Rumah Yatim to receive donations more efficiently and track them more systematically and transparently, resulting in better decision-making processes and more effective distribution of funds to those in need. The system has also helped streamline administrative tasks and reduce the time and resources required for manual data entry and report creation, leading to overall improved effectiveness and efficiency in operations.
Quench dynamics of the Schwinger model via variational quantum algorithms

We investigate the real-time dynamics of the $(1+1)$-dimensional U(1) gauge theory known as the Schwinger model via variational quantum algorithms. Specifically, we simulate quench dynamics in the presence of an external electric field. First, we use a variational quantum eigensolver to obtain the ground state of the system in the absence of an external field. With this as the initial state, we perform real-time evolution under an external field via a fixed-depth, parameterized circuit whose parameters are updated using McLachlan's variational principle. We use the same Ansatz for initial state preparation and time evolution, by which we are able to reduce the overall circuit depth. We test our method with a classical simulator and confirm that the results agree well with exact diagonalization.
From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models

Language models (LMs) are pretrained on diverse data sources, including news, discussion forums, books, and online encyclopedias.A significant portion of this data includes opinions and perspectives which, on one hand, celebrate democracy and diversity of ideas, and on the other hand are inherently socially biased.Our work develops new methods to (1) measure political biases in LMs trained on such corpora, along social and economic axes, and (2) measure the fairness of downstream NLP models trained on top of politically biased LMs.We focus on hate speech and misinformation detection, aiming to empirically quantify the effects of political (social, economic) biases in pretraining data on the fairness of high-stakes social-oriented tasks.Our findings reveal that pretrained LMs do have political leanings that reinforce the polarization present in pretraining corpora, propagating social biases into hate speech predictions and misinformation detectors.We discuss the implications of our findings for NLP research and propose future directions to mitigate unfairness.
LLM-RM at SemEval-2023 Task 2: Multilingual Complex NER Using XLM-RoBERTa

Named Entity Recognition(NER) is a task of recognizing entities at a token level in a sentence.This paper focuses on solving NER tasks in a multilingual setting for complex named entities.Our team, LLM-RM participated in the recently organized SemEval 2023 task, Task 2: MultiCoNER II,Multilingual Complex Named Entity Recognition.We approach the problem by leveraging cross-lingual representation provided by fine-tuning XLM-Roberta base model on datasets of all of the 12 languages provided -Bangla, Chinese, English, Farsi, French, German, Hindi, Italian, Portuguese, Spanish, Swedish and Ukrainian.
Parallel Context Windows for Large Language Models

When applied to processing long text, Large Language Models (LLMs) are limited by their context window.Existing efforts to address this limitation involve training specialized architectures, and cannot be easily applied to offthe-shelf LLMs.We present Parallel Context Windows (PCW), a method that alleviates the context window restriction for any off-the-shelf LLM without further training.The key to the approach is to carve a long context into chunks ("windows"), restrict the attention mechanism to apply only within each window, and re-use the positional embeddings across the windows.Our main results test the PCW approach on in-context learning with models that range in size between 750 million and 178 billion parameters, and show substantial improvements for tasks with diverse input and output spaces.We show additional benefits in other settings where long context windows may be beneficial: multi-hop questions and retrieval-augmented question answering with multiple retrieved documents.Our results highlight Parallel Context Windows as a promising method for applying off-the-shelf LLMs in a range of settings that require long text sequences.We make our code publicly available at https://github.com/ai21labs/parallel-context-windows.input: Argentina brings the cup home.
Can Large Language Models Be an Alternative to Human Evaluations?

Human evaluation is indispensable and inevitable for assessing the quality of texts generated by machine learning models or written by humans.However, human evaluation is very difficult to reproduce and its quality is notoriously unstable, hindering fair comparisons among different natural language processing (NLP) models and algorithms.Recently, large language models (LLMs) have demonstrated exceptional performance on unseen tasks when only the task instructions are provided.In this paper, we explore if such an ability of the LLMs can be used as an alternative to human evaluation.We present the LLMs with the exact same instructions, samples to be evaluated, and questions used to conduct human evaluation, and then ask the LLMs to generate responses to those questions; we dub this LLM evaluation.We use human evaluation and LLM evaluation to evaluate the texts in two NLP tasks: openended story generation and adversarial attacks.We show that the result of LLM evaluation is consistent with the results obtained by expert human evaluation: the texts rated higher by human experts are also rated higher by the LLMs.We also find that the results of LLM evaluation are stable over different formatting of the task instructions and the sampling algorithm used to generate the answer.We are the first to show the potential of using LLMs to assess the quality of texts and discuss the limitations and ethical considerations of LLM evaluation.
“Low-Resource” Text Classification: A Parameter-Free Classification Method with Compressors

Deep neural networks (DNNs) are often used for text classification due to their high accuracy.However, DNNs can be computationally intensive, requiring millions of parameters and large amounts of labeled data, which can make them expensive to use, to optimize, and to transfer to out-of-distribution (OOD) cases in practice.In this paper, we propose a non-parametric alternative to DNNs that's easy, lightweight, and universal in text classification: a combination of a simple compressor like gzip with a k-nearest-neighbor classifier.Without any training parameters, our method achieves results that are competitive with non-pretrained deep learning methods on six in-distribution datasets.It even outperforms BERT on all five OOD datasets, including four low-resource languages.Our method also excels in the few-shot setting, where labeled data are too scarce to train DNNs effectively.Code is available at https://github.com/bazingagin/npc_gzip.Compressor Compressor Compressor concat NCD distance
On Second Thought, Let’s Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning

Warning: This paper contains several toxic and offensive statements.Generating a Chain of Thought (CoT) has been shown to consistently improve large language model (LLM) performance on a wide range of NLP tasks.However, prior work has mainly focused on logical reasoning tasks (e.g.arithmetic, commonsense QA); it remains unclear whether improvements hold for more diverse types of reasoning, especially in socially situated contexts.Concretely, we perform a controlled evaluation of zero-shot CoT across two socially sensitive domains: harmful questions and stereotype benchmarks.We find that zeroshot CoT reasoning in sensitive domains significantly increases a model's likelihood to produce harmful or undesirable output, with trends holding across different prompt formats and model variants.Furthermore, we show that harmful CoTs increase with model size, but decrease with improved instruction following.Our work suggests that zero-shot CoT should be used with caution on socially important tasks, especially when marginalized groups or sensitive topics are involved.
Element-aware Summarization with Large Language Models: Expert-aligned Evaluation and Chain-of-Thought Method

Automatic summarization generates concise summaries that contain key ideas of source documents.As the most mainstream datasets for the news sub-domain, CNN/DailyMail and BBC XSum have been widely used for performance benchmarking.However, the reference summaries of those datasets turn out to be noisy, mainly in terms of factual hallucination and information redundancy.To address this challenge, we first annotate new expertwriting Element-aware test sets following the "Lasswell Communication Model" proposed by Lasswell (1948), allowing reference summaries to focus on more fine-grained news elements objectively and comprehensively.Utilizing the new test sets, we observe the surprising zero-shot summary ability of LLMs, which addresses the issue of the inconsistent results between human preference and automatic evaluation metrics of LLMs' zero-shot summaries in prior work.Further, we propose a Summary Chain-of-Thought (SumCoT) technique to elicit LLMs to generate summaries step by step, which helps them integrate more finegrained details of source documents into the final summaries that correlate with the human writing mindset.Experimental results show our method outperforms state-of-the-art fine-tuned PLMs and zero-shot LLMs by +4.33/+4.77 in ROUGE-L on the two datasets, respectively.Dataset and code are publicly available at https://github.com/Alsace08/SumCoT.
Evaluating the Factual Consistency of Large Language Models Through News Summarization

While large language models (LLMs) have proven to be effective on a large variety of tasks, they are also known to hallucinate information.To measure whether an LLM prefers factually consistent continuations of its input, we propose a new benchmark called FIB (Factual Inconsistency Benchmark) that focuses on the task of summarization.Specifically, our benchmark involves comparing the scores an LLM assigns to a factually consistent versus a factually inconsistent summary for an input news article.For factually consistent summaries, we use human-written reference summaries that we manually verify as factually consistent.To generate summaries that are factually inconsistent, we generate summaries from a suite of summarization models that we have manually annotated as factually inconsistent.A model's factual consistency is then measured according to its accuracy, i.e. the proportion of documents where it assigns a higher score to the factually consistent summary.To validate the usefulness of FIB, we evaluate 23 large language models ranging from 1B to 176B parameters from six different model families including BLOOM and OPT.We find that existing LLMs generally assign a higher score to factually consistent summaries than to factually inconsistent summaries.However, if the factually inconsistent summaries occur verbatim in the document, then LLMs assign a higher score to these factually inconsistent summaries than factually consistent summaries.We validate design choices in our benchmark including the scoring method and source of distractor summaries.
Evaluation of ChatGPT on Biomedical Tasks: A Zero-Shot Comparison with Fine-Tuned Generative Transformers

ChatGPT is a large language model developed by OpenAI 1 .Despite its impressive performance across various tasks, no prior work has investigated its capability in the biomedical domain yet.To this end, this paper aims to evaluate the performance of ChatGPT on various benchmark biomedical tasks, such as relation extraction, document classification, question answering, and summarization.To the best of our knowledge, this is the first work that conducts an extensive evaluation of ChatGPT in the biomedical domain.Interestingly, we find based on our evaluation that in biomedical datasets that have smaller training sets, zeroshot ChatGPT even outperforms the state-ofthe-art fine-tuned generative transformer models, such as BioGPT and BioBART.This suggests that ChatGPT's pre-training on large text corpora makes it quite specialized even in the biomedical domain.Our findings demonstrate that ChatGPT has the potential to be a valuable tool for various tasks in the biomedical domain that lack large annotated data.
Prompt-Based Meta-Learning For Few-shot Text Classification

Few-shot Text Classification predicts the semantic label of a given text with a handful of supporting instances.Current meta-learning methods have achieved satisfying results in various few-shot situations.Still, they often require a large amount of data to construct many few-shot tasks for meta-training, which is not practical in real-world few-shot scenarios.Prompt-tuning has recently proved to be another effective few-shot learner by bridging the gap between pre-train and downstream tasks.In this work, we closely combine the two promising few-shot learning methodologies in structure and propose a Prompt-Based Meta-Learning (PBML) model to overcome the above meta-learning problem by adding the prompting mechanism.PBML assigns label word learning to base-learners and template learning to meta-learner, respectively.Experimental results show state-of-the-art performance on four text classification datasets under few-shot settings, with higher accuracy and good robustness.We demonstrate through lowresource experiments that our method alleviates the shortcoming that meta-learning requires too much data for meta-training.In the end, we use the visualization to interpret and verify that the meta-learning framework can help the prompting method converge better.We release our code to reproduce our experiments 1 .
Zero-Shot Text Classification with Self-Training

Recent advances in large pretrained language models have increased attention to zero-shot text classification.In particular, models finetuned on natural language inference datasets have been widely adopted as zero-shot classifiers due to their promising results and offthe-shelf availability.However, the fact that such models are unfamiliar with the target task can lead to instability and performance issues.We propose a plug-and-play method to bridge this gap using a simple self-training approach, requiring only the class names along with an unlabeled dataset, and without the need for domain expertise or trial and error.We show that fine-tuning the zero-shot classifier on its most confident predictions leads to significant performance gains across a wide range of text classification tasks, presumably since self-training adapts the zero-shot model to the task at hand.
LILA: A Unified Benchmark for Mathematical Reasoning

Swaroop Mishra, Matthew Finlayson, Pan Lu, Leonard Tang, Sean Welleck, Chitta Baral, Tanmay Rajpurohit, Oyvind Tafjord, Ashish Sabharwal, Peter Clark, Ashwin Kalyan. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022.
An Empirical Analysis of Memorization in Fine-tuned Autoregressive Language Models

Several recent works have shown that large language models present privacy risks through memorization of training data.Little attention, however, has been given to the fine-tuning phase and it is not well understood how memorization risk varies across different fine-tuning methods (such as fine-tuning the full model, the model head, and adapter).This presents increasing concern as the "pre-train and fine-tune" paradigm proliferates.We empirically study memorization of fine-tuning methods using membership inference and extraction attacks, and show that their susceptibility to attacks is very different.We observe that fine-tuning the head of the model has the highest susceptibility to attacks, whereas fine-tuning smaller adapters appears to be less vulnerable to known extraction attacks.
Affective Knowledge Enhanced Multiple-Graph Fusion Networks for Aspect-based Sentiment Analysis

Aspect-based sentiment analysis aims to identify sentiment polarity of social media users toward different aspects.Most recent methods adopt the aspect-centric latent tree to connect aspects and their corresponding opinion words, thinking that would facilitate establishing the relationship between aspects and opinion words.However, these methods ignore the roles of syntax dependency relation labels and affective semantic information in determining the sentiment polarity, resulting in the wrong prediction.In this paper, we propose a novel multi-graph fusion network (MGFN) based on latent graph to leverage the richer syntax dependency relation label information and affective semantic information of words.Specifically, we construct a novel syntax-aware latent graph (SaLG) to fully leverage the syntax dependency relation label information to facilitate the learning of sentiment representations.Subsequently, a multi-graph fusion module is proposed to fuse semantic information of surrounding contexts of aspects adaptively.Furthermore, we design an affective refinement strategy to guide the MGFN to capture significant affective clues.Extensive experiments on three datasets demonstrate that our MGFN model outperforms all state-of-the-art methods and verify the effectiveness of our model.
NADI 2022: The Third Nuanced Arabic Dialect Identification Shared Task

We describe the findings of the third Nuanced Arabic Dialect Identification Shared Task (NADI 2022).NADI aims at advancing state-of-the-art Arabic NLP, including Arabic dialects.It does so by affording diverse datasets and modeling opportunities in a standardized context where meaningful comparisons between models and approaches are possible.NADI 2022 targeted both dialect identification (Subtask 1) and dialectal sentiment analysis (Subtask 2) at the country level.A total of 41 unique teams registered for the shared task, of whom 21 teams have participated (with 105 valid submissions).Among these, 19 teams participated in Subtask 1, and 10 participated in Subtask 2. The winning team achieved F 1 =27.06 on Subtask 1 and F 1 =75.16 on Subtask 2, reflecting that both subtasks remain challenging and motivating future work in this area.We describe the methods employed by the participating teams and offer an outlook for NADI.
Sentence-level Media Bias Analysis Informed by Discourse Structures

As polarization continues to rise among both the public and the news media, increasing attention has been devoted to detecting media bias.Most recent work in the NLP community, however, identify bias at the level of individual articles.However, each article itself comprises multiple sentences, which vary in their ideological bias.In this paper, we aim to identify sentences within an article that can illuminate and explain the overall bias of the entire article.We show that understanding the discourse role of a sentence in telling a news story, as well as its relation with nearby sentences, can reveal the ideological leanings of an author even when the sentence itself appears merely neutral.In particular, we consider using a functional news discourse structure and PDTB discourse relations to inform bias sentence identification, and distill the auxiliary knowledge from the two types of discourse structure into our bias sentence identification system.Experimental results on benchmark datasets show that incorporating both the global functional discourse structure and local rhetorical discourse relations can effectively increase the recall of bias sentence identification by 8.27% -8.62%, as well as increase the precision by 2.82% -3.48% 1 .
Active Example Selection for In-Context Learning

With a handful of demonstration examples, large-scale language models show strong capability to perform various tasks by in-context learning from these examples, without any finetuning.We demonstrate that in-context learning performance can be highly unstable across samples of examples, indicating the idiosyncrasies of how language models acquire information.We formulate example selection for in-context learning as a sequential decision problem, and propose a reinforcement learning algorithm for identifying generalizable policies to select demonstration examples.For GPT-2, our learned policies demonstrate strong abilities of generalizing to unseen tasks in training, with a 5.8% improvement on average.Examples selected from our learned policies can even achieve a small improvement on GPT-3 Ada.However, the improvement diminishes on larger GPT-3 models, suggesting emerging capabilities of large language models.
ClinicalT5: A Generative Language Model for Clinical Text

In the past few years, large pre-trained language models (PLMs) have been widely adopted in different areas and have made fundamental improvements over a variety of downstream tasks in natural language processing (NLP).Meanwhile, domain-specific variants of PLMs are being proposed to address the needs of domains that demonstrate a specific pattern of writing and vocabulary, e.g., BioBERT for the biomedical domain and ClinicalBERT for the clinical domain.Recently, generative language models like BART and T5 are gaining popularity with their competitive performance on text generation as well as on tasks cast as generative problems.However, in the clinical domain, such domain-specific generative variants are still underexplored.To address this need, our work introduces a T5-based text-to-text transformer model pre-trained on clinical text, i.e., Clini-calT5.We evaluate the proposed model both intrinsically and extrinsically over a diverse set of tasks across multiple datasets, and show that ClinicalT5 dramatically outperforms T5 in the domain-specific tasks and compares favorably with its close baselines. 1