Reviewerly/exHarmony huggingface dataset:
The development of a virtual reality adventure game to prepare for a tsunami disaster

This research aimed to study and analyse the interaction between humans and a tsunami disaster simulation. Besides, it aimed to develop a virtual reality (VR) adventure game in regard to preparing for a tsunami disaster caused by an earthquake. It also assessed the effectiveness of virtual reality (VR) through three perceptions, namely visual perception, auditory perception, and kinesthetic perception with the connection between the human user and the Oculus quest 1 (VR glasses) to create virtual reality that was easy to understand and reflected interesting storytelling accompanied by beautiful patterns. The research objectives were 1) to study, collect, and analyse data related to the concept of creating virtual reality (VR), 2) to design and create virtual reality (VR) adventure game, and 3) to test and assess the perceptions of virtual reality (VR) to prepare for a tsunami disaster caused by an earthquake. The results of studies and analysis based on the learning style theory and participants’ behaviour showed that the behaviour was often instilled from experiences that they had gained. This created clarity in the role of the participants and helped them learn and practice using virtual reality technology in training and deal with tsunami disasters caused by earthquakes. The results of the perception assessment of the virtual reality (VR) prototype, overall, indicated that the adventure game in a virtual reality (VR) setting to prepare for a tsunami disaster caused by an earthquake was feasible for use.
The meaning of redundancy and consensus in quantum objectivity

While the terms "redundancy" and "consensus" are often used as synonyms in the context of quantum objectivity, we show here that these should be understood as two related but distinct notions, that quantify different features of the quantum-to-classical transition. We show that the two main frameworks used to measure quantum objectivity, namely spectrum broadcast structure and quantum Darwinism, are best suited to quantify redundancy and consensus, respectively. Furthermore, by analyzing explicit examples of states with nonlocally encoded information, we highlight the potentially stark difference between the degrees of redundancy and consensus. In particular, this causes a break in the hierarchical relations between spectrum broadcast structure and quantum Darwinism. Our framework provides a new perspective to interpret known and future results in the context of quantum objectivity, paving the way for a deeper understanding of the emergence of classicality from the quantum realm.
CaPE: Contrastive Parameter Ensembling for Reducing Hallucination in Abstractive Summarization

Hallucination is a known issue for neural abstractive summarization models.Recent work suggests that the degree of hallucination may depend on factual errors in the training data.In this work, we propose a new method called Contrastive Parameter Ensembling (CaPE) to use training data more effectively, utilizing variations in noise in training samples to reduce hallucination.Starting with a base model finetuned on an entire dataset, we additionally train expert and anti-expert models on clean and noisy subsets of the data, respectively.We then adjust the parameters of the base model by adding (subtracting) the parameters of the expert (anti-expert), advancing the recent work on additive parameter ensembling approaches.Trained on a much smaller data subset, expert and anti-expert models only fractionally (<14%) increases the total training time.Further, CaPE uses parameter ensembling and does not increase the inference time.Experimental results show that CaPE improves performance across different automatic factual metrics and human evaluation, with a maximum improvement of 16.69% and 15.38% on summary-level dependency-arc entailment accuracy for the XSUM and CNN/DM datasets.The CaPE model performs comparably to the base model on metrics of informativeness such as ROUGE.
End-to-end Knowledge Retrieval with Multi-modal Queries

We investigate knowledge retrieval with multimodal queries, i.e. queries containing information split across image and text inputs, a challenging task that differs from previous work on cross-modal retrieval.We curate a new dataset called ReMuQ 1 for benchmarking progress on this task.ReMuQ requires a system to retrieve knowledge from a large corpus by integrating contents from both text and image queries.We introduce a retriever model "ReViz" that can directly process input text and images to retrieve relevant knowledge in an end-to-end fashion without being dependent on intermediate modules such as object detectors or caption generators.We introduce a new pretraining task that is effective for learning knowledge retrieval with multimodal queries and also improves performance on downstream tasks.We demonstrate superior performance in retrieval on two datasets (ReMuQ and OK-VQA) under zeroshot settings as well as further improvements when finetuned on these datasets.K1: The Empire State Building is a 102-story Art Deco skyscraper in Midtown Manhattan, New York City K2: The 828 metre (2,717 ft) tall Burj Khalifa in Dubai has been the tallest building since 2010.The Burj Khalifa has been classified as megatall.
Leveraging Training Dynamics and Self-Training for Text Classification

The effectiveness of pre-trained language models in downstream tasks is highly dependent on the amount of labeled data available for training.Semi-supervised learning (SSL) is a promising technique that has seen wide attention recently due to its effectiveness in improving deep learning models when training data is scarce.Common approaches employ a teacher-student self-training framework, where a teacher network generates pseudo-labels for unlabeled data, which are then used to iteratively train a student network.In this paper, we propose a new self-training approach for text classification that leverages training dynamics of unlabeled data.We evaluate our approach on a wide range of text classification tasks, including emotion detection, sentiment analysis, question classification and gramaticality, which span a variety of domains, e.g, Reddit, Twitter, and online forums.Notably, our method is successful on all benchmarks, obtaining an average increase in F1 score of 3.5% over strong baselines in low resource settings.We make our code avilable at
Robust Domain Adaptation for Pre-trained Multilingual Neural Machine Translation Models

Recent literature has demonstrated the potential of multilingual Neural Machine Translation (mNMT) models. However, the most efficient models are not well suited to specialized industries. In these cases, internal data is scarce and expensive to find in all language pairs. Therefore, fine-tuning a mNMT model on a specialized domain is hard. In this context, we decided to focus on a new task: Domain Adaptation of a pre-trained mNMT model on a single pair of language while trying to maintain model quality on generic domain data for all language pairs. The risk of loss on generic domain and on other pairs is high. This task is key for mNMT model adoption in the industry and is at the border of many others. We propose a fine-tuning procedure for the generic mNMT that combines embeddings freezing and adversarial loss. Our experiments demonstrated that the procedure improves performances on specialized data with a minimal loss in initial performances on generic domain for all languages pairs, compared to a naive standard approach (+10.0 BLEU score on specialized data, -0.01 to -0.5 BLEU on WMT and Tatoeba datasets on the other pairs with M2M100).
Enhancing Dialogue Generation via Dynamic Graph Knowledge Aggregation

Incorporating external graph knowledge into neural chatbot models has been proven effective for enhancing dialogue generation.However, in conventional graph neural networks (GNNs), message passing on a graph is independent from text, resulting in the graph representation hidden space differing from that of the text.This training regime of existing models therefore leads to a semantic gap between graph knowledge and text.In this study, we propose a novel framework for knowledge graph enhanced dialogue generation.We dynamically construct a multi-hop knowledge graph with pseudo nodes to involve the language model in feature aggregation within the graph at all steps.To avoid the semantic biases caused by learning on vanilla subgraphs, the proposed framework applies hierarchical graph attention to aggregate graph features on pseudo nodes and then attains a global feature.Therefore, the framework can better utilise the heterogeneous features from both the post and external graph knowledge.Extensive experiments demonstrate that our framework outperforms state-of-the-art (SOTA) baselines on dialogue generation.Further analysis also shows that our representation learning framework can fill the semantic gap by coagulating representations of both text and graph knowledge.Moreover, the language model also learns how to better select knowledge triples for a more informative response via exploiting subgraph patterns within our feature aggregation process.
No clues good clues: out of context Lexical Relation Classification

The accurate prediction of lexical relations between words is a challenging task in Natural Language Processing (NLP).The most recent advances in this direction come with the use of pre-trained language models (PTLMs).A PTLM typically needs "well-formed" verbalized text to interact with it, either to fine-tune it or to exploit it.However, there are indications that commonly used PTLMs already encode enough linguistic knowledge to allow the use of minimal (or none) textual context for some linguistically motivated tasks, thus notably reducing human effort, the need for data pre-processing, and favoring techniques that are language neutral since do not rely on syntactic structures.In this work, we explore this idea for the tasks of lexical relation classification (LRC) and graded Lexical Entailment (LE).After finetuning PTLMs for LRC with different verbalizations, our evaluation results show that very simple prompts are competitive for LRC and significantly outperform graded LE SoTA.In order to gain a better insight into this phenomenon, we perform a number of quantitative statistical analyses on the results, as well as a qualitative visual exploration based on embedding projections.
Language model acceptability judgements are not always robust to context

Targeted syntactic evaluations of language models ask whether models show stable preferences for syntactically acceptable content over minimal-pair unacceptable inputs.Our best syntactic evaluation datasets, however, provide substantially less linguistic context than models receive during pretraining.This mismatch raises an important question: how robust are models' syntactic judgements across different contexts?In this paper, we vary the input contexts based on: length, the types of syntactic phenomena it contains, and whether or not there are grammatical violations.We find that model judgements are generally robust when placed in randomly sampled linguistic contexts, but are unstable when contexts match the test stimuli in syntactic structure.Among all tested models (GPT-2 and five variants of OPT), we find that model performance is affected when we provided contexts with matching syntactic structure: performance significantly improves when contexts are acceptable, and it significantly declines when they are unacceptable.This effect is amplified by the length of the context, except for unrelated inputs.We show that these changes in model performance are not explainable by acceptability-preserving syntactic perturbations.This sensitivity to highly specific syntactic features of the context can only be explained by the models' implicit in-context learning abilities.
Evaluating the Effectiveness of Natural Language Inference for Hate Speech Detection in Languages with Limited Labeled Data

Most research on hate speech detection has focused on English where a sizeable amount of labeled training data is available.However, to expand hate speech detection into more languages, approaches that require minimal training data are needed.In this paper, we test whether natural language inference (NLI) models which perform well in zero-and few-shot settings can benefit hate speech detection performance in scenarios where only a limited amount of labeled data is available in the target language.Our evaluation on five languages demonstrates large performance improvements of NLI fine-tuning over direct fine-tuning in the target language.However, the effectiveness of previous work that proposed intermediate finetuning on English data is hard to match.Only in settings where the English training data does not match the test domain, can our customised NLI-formulation outperform intermediate finetuning on English.Based on our extensive experiments, we propose a set of recommendations for hate speech detection in languages where minimal labeled training data is available. 1
Benchmarking Offensive and Abusive Language in Dutch Tweets

We present an extensive evaluation of different fine-tuned models to detect instances of offensive and abusive language in Dutch across three benchmarks: a standard held-out test, a taskagnostic functional benchmark, and a dynamic test set.We also investigate the use of data cartography to identify high quality training data.Our results show a relatively good quality of the manually annotated data used to train the models while highlighting some critical weakness.We have also found a good portability of trained models along the same language phenomena.As for the data cartography, we have found a positive impact only on the functional benchmark and when selecting data per annotated dimension rather than using the entire training material.
Cross-Lingual Retrieval Augmented Prompt for Low-Resource Languages

Multilingual Pretrained Language Models (MPLMs) perform strongly in cross-lingual transfer.We propose Prompts Augmented by Retrieval Crosslingually (PARC) to improve zero-shot performance on low-resource languages (LRLs) by augmenting the context with prompts consisting of semantically similar sentences retrieved from a high-resource language (HRL).PARC improves zero-shot performance on three downstream tasks (sentiment classification, topic categorization, natural language inference) with multilingual parallel test sets across 10 LRLs covering 6 language families in unlabeled (+5.1%) and labeled settings (+16.3%).PARC also outperforms finetuning by 3.7%.We find a significant positive correlation between cross-lingual transfer performance on one side, and the similarity between high-and low-resource languages as well as the amount of low-resource pretraining data on the other side.A robustness analysis suggests that PARC has the potential to achieve even stronger performance with more powerful MPLMs.
Evaluating Reading Comprehension Exercises Generated by LLMs: A Showcase of ChatGPT in Education Applications

The recent advancement of pre-trained Large Language Models (LLMs), such as OpenAI's ChatGPT, has led to transformative changes across fields.For example, developing intelligent systems in the educational sector that leverage the linguistic capabilities of LLMs demonstrates a visible potential.Though researchers have recently explored how Chat-GPT could possibly assist in student learning, few studies have applied these techniques to real-world classroom settings involving teachers and students.In this study, we implement a reading comprehension exercise generation system that provides high-quality and personalized reading materials for middle school English learners in China.Extensive evaluations of the generated reading passages and corresponding exercise questions, conducted both automatically and manually, demonstrate that the system-generated materials are suitable for students and even surpass the quality of existing human-written ones.By incorporating first-hand feedback and suggestions from experienced educators, this study serves as a meaningful pioneering application of ChatGPT, shedding light on the future design and implementation of LLM-based systems in the educational context.
Transformed Protoform Reconstruction

Protoform reconstruction is the task of inferring how morphemes or words sounded in ancestral languages of a set of daughter languages.Meloni et al. (2021) achieved the stateof-the-art on Latin protoform reconstruction with an RNN-based encoder-decoder with attention model.We update their model with the state-of-the-art seq2seq model-the Transformer.Our model outperforms their model on a suite of different metrics on two different datasets: Meloni et al.'s Romance data of 8,000+ cognates (spanning 5 languages) and a Chinese dataset (Hóu, 2004) of 800+ cognates (spanning 39 varieties).We also probe our model for potential phylogenetic signal contained in the model.Our code is publicly available 1 .* Equal contribution 1 https://github.com/cmu-llab/acl-2023 2 In fact, the proto-language from which Romance languages like Spanish and Italian are descended is not identical to Classical Latin but is, rather, a closely related and sparsely attested language sometimes called Proto-Romance or Vulgar Latin.
Say What You Mean! Large Language Models Speak Too Positively about Negative Commonsense Knowledge

Large language models (LLMs) have been widely studied for their ability to store and utilize positive knowledge.However, negative knowledge, such as "lions don't live in the ocean", is also ubiquitous in the world but rarely mentioned explicitly in the text.What do LLMs know about negative knowledge?This work examines the ability of LLMs to negative commonsense knowledge.We design a constrained keywords-to-sentence generation task (CG) and a Boolean question-answering task (QA) to probe LLMs.Our experiments reveal that LLMs frequently fail to generate valid sentences grounded in negative commonsense knowledge, yet they can correctly answer polar yes-or-no questions.We term this phenomenon the belief conflict of LLMs.Our further analysis shows that statistical shortcuts and negation reporting bias from language modeling pre-training cause this conflict.
CodePrompt: Task-Agnostic Prefix Tuning for Program and Language Generation

In order to solve the inefficient parameter update and storage issues of fine-tuning in Natural Language Generation (NLG) tasks, prompttuning methods have emerged as lightweight alternatives.Furthermore, efforts to reduce the gap between pre-training and fine-tuning have shown successful results in low-resource settings.As large Pre-trained Language Models (PLMs) for Program and Language Generation (PLG) tasks are constantly being developed, prompt tuning methods are necessary for the tasks.However, due to the gap between pretraining and fine-tuning different from PLMs for natural language, a prompt tuning method that reflects the traits of PLM for program language is needed.In this paper, we propose a Task-Agnostic prompt tuning method for the PLG tasks, CodePrompt, that combines Input-Dependent Prompt Template (to bridge the gap between pre-training and fine-tuning of PLMs for program and language) and Corpus-Specific Prefix Tuning (to update the parameters of PLMs for program and language efficiently).Also, we propose a method to provide richer prefix word information for limited prefix lengths.We prove that our method is effective in three PLG tasks, not only in the full-data setting but also in the low-resource setting and cross-domain setting.
Are Machine Rationales (Not) Useful to Humans? Measuring and Improving Human Utility of Free-text Rationales

Among the remarkable emergent capabilities of large language models (LMs) is free-text rationalization; beyond a certain scale, large LMs are capable of generating seemingly useful rationalizations, which in turn, can dramatically enhance their performances on leaderboards.This phenomenon raises a question: can machine generated rationales also be useful for humans, especially when lay humans try to answer questions based on those machine rationales?We observe that human utility of existing rationales is far from satisfactory, and expensive to estimate with human studies.Existing metrics like task performance of the LM generating the rationales, or similarity between generated and gold rationales are not good indicators of their human utility.While we observe that certain properties of rationales like conciseness and novelty are correlated with their human utility, estimating them without human involvement is challenging.We show that, by estimating a rationale's helpfulness in answering similar unseen instances, we can measure its human utility to a better extent.We also translate this finding into an automated score, GEN-U, that we propose, which can help improve LMs' ability to generate rationales with better human utility, while maintaining most of its task performance.Lastly, we release all code and collected data with this project.
Environmental Claim Detection

To transition to a green economy, environmental claims made by companies must be reliable, comparable, and verifiable.To analyze such claims at scale, automated methods are needed to detect them in the first place.However, there exist no datasets or models for this.Thus, this paper introduces the task of environmental claim detection.To accompany the task, we release an expert-annotated dataset and models trained on this dataset.We preview one potential application of such models: We detect environmental claims made in quarterly earning calls and find that the number of environmental claims has steadily increased since the Paris Agreement in 2015.
Distractor Generation based on Text2Text Language Models with Pseudo Kullback-Leibler Divergence Regulation

In this paper, we address the task of cloze-style multiple choice question (MCQs) distractor generation. Our study is featured by the following designs. First, we propose to formulate the cloze distractor generation as a Text2Text task. Second, we propose pseudo Kullback-Leibler Divergence for regulating the generation to consider the item discrimination index in education evaluation. Third, we explore the candidate augmentation strategy and multi-tasking training with cloze-related tasks to further boost the generation performance. Through experiments with benchmarking datasets, our best perfomring model advances the state-of-the-art result from 10.81 to 22.00 (p@1 score).
Back to Patterns: Efficient Japanese Morphological Analysis with Feature-Sequence Trie

Accurate neural models are much less efficient than non-neural models and are useless for processing billions of social media posts or handling user queries in real time with a limited budget.This study revisits the fastest patternbased NLP methods to make them as accurate as possible, thus yielding a strikingly simple yet surprisingly accurate morphological analyzer for Japanese.The proposed method induces reliable patterns from a morphological dictionary and annotated data.Experimental results on two standard datasets confirm that the method exhibits comparable accuracy to learning-based baselines, while boasting a remarkable throughput of over 1,000,000 sentences per second on a single modern CPU.