Reviewerly/exHarmony huggingface dataset:
SeeGULL: A Stereotype Benchmark with Broad Geo-Cultural Coverage Leveraging Generative Models

Stereotype benchmark datasets are crucial to detect and mitigate social stereotypes about groups of people in NLP models.However, existing datasets are limited in size and coverage, and are largely restricted to stereotypes prevalent in the Western society.This is especially problematic as language technologies gain hold across the globe.To address this gap, we present SeeGULL, a broad-coverage stereotype dataset, built by utilizing generative capabilities of large language models such as PaLM, and GPT-3, and leveraging a globally diverse rater pool to validate the prevalence of those stereotypes in society.SeeG-ULL is in English, and contains stereotypes about identity groups spanning 178 countries across 8 different geo-political regions across 6 continents, as well as state-level identities within the US and India.We also include fine-grained offensiveness scores for different stereotypes and demonstrate their global disparities.Furthermore, we include comparative annotations about the same groups by annotators living in the region vs. those that are based in North America, and demonstrate that within-region stereotypes about groups differ from those prevalent in North America.CONTENT WARNING: This paper contains stereotype examples that may be offensive.
In-Context Analogical Reasoning with Pre-Trained Language Models

Analogical reasoning is a fundamental capacity of human cognition that allows us to reason abstractly about novel situations by relating them to past experiences.While it is thought to be essential for robust reasoning in AI systems, conventional approaches require significant training and/or hard-coding of domain knowledge to be applied to benchmark tasks.Inspired by cognitive science research that has found connections between human language and analogy-making, we explore the use of intuitive language-based abstractions to support analogy in AI systems.Specifically, we apply large pre-trained language models (PLMs) to visual Raven's Progressive Matrices (RPM), a common relational reasoning test.By simply encoding the perceptual features of the problem into language form, we find that PLMs exhibit a striking capacity for zero-shot relational reasoning, exceeding human performance and nearing supervised vision-based methods.We explore different encodings that vary the level of abstraction over task features, finding that higherlevel abstractions further strengthen PLMs' analogical reasoning.Our detailed analysis reveals insights on the role of model complexity, incontext learning, and prior knowledge in solving RPM tasks.
Impact of COVID-19 on education in Mexico

It must be recognized that we were not prepared to face this atypical situation caused by the coronavirus disease (COVID-19), which has caused a crisis that will have significant negative effects on all social sectors, particularly health and education. , as well as employment and the evolution of poverty. (CEPAL1, 2020) This disease put people of all ages at risk and traveled from the most privileged places to the most modest; it lodged the same in those who are outside of letters, as in those who are distinguished by their knowledge, and indistinctly infected those who cultivate the highest values, as well as those who show off their detachment from them. In the field of education, the confinement and closure of schools and universities as measures to contain the pandemic, led governments to design educational strategies such as distance education, to guarantee the continuity of learning. Faced with this health emergency, emerging pedagogical practices arose where there was a unique transfer from the classroom and the recreation and rest spaces, to the living room and dining room of the house and, in the case of basic education, the participation of mothers and fathers to address academic problems. There are many limitations that both teachers and students have had to deal with, from the low or null connectivity and the lack of training in the use and management of Information and Communication Technologies (ICT), based on the assumption of that there would be a computer with internet access or a television, as well as the necessary human capacities to advise mothers and fathers or tutors of students on scientific, humanistic and artistic issues. According to World Bank estimates, the pandemic caused a lag in Mexico equivalent to two years of schooling. Before the pandemic, Mexicans reached an average learning level corresponding to 3rd grade of secondary school. Today your knowledge will only reach the equivalent of 1st grade of high school. (IMCO, 2021)
Scalable and Safe Remediation of Defective Actions in Self-Learning Conversational Systems

Off-Policy reinforcement learning has been a driving force for the state-of-the-art conversational AIs leading to more natural humanagent interactions and improving the user satisfaction for goal-oriented agents.However, in large-scale commercial settings, it is often challenging to balance between policy improvements and experience continuity on the broad spectrum of applications handled by such system.In the literature, off-policy evaluation and guard-railing on aggregate statistics has been commonly used to address this problem.In this paper, we propose a method for curating and leveraging high-precision samples sourced from historical regression incident reports to validate, safe-guard, and improve policies prior to the online deployment.We conducted extensive experiments using data from a real-world conversational system and actual regression incidents.The proposed method is currently deployed in our production system to protect customers against broken experiences and enable long-term policy improvements.
RC3: Regularized Contrastive Cross-lingual Cross-modal Pre-training

Multilingual vision-language (V&L) pre-training has achieved remarkable progress in learning universal representations across different modalities and languages. In spite of recent success, there still remain challenges limiting further improvements of V&L pre-trained models in multilingual settings. Particularly, current V&L pre-training methods rely heavily on strictly-aligned multilingual image-text pairs generated from English-centric datasets through machine translation. However, the cost of collecting and translating such strictly-aligned datasets is usually unbearable. In this paper, we propose Regularized Contrastive Cross-lingual Cross-modal (RC3) pre-training, which further exploits more abundant weakly-aligned multilingual image-text pairs. Specifically, we design a regularized cross-lingual visio-textual contrastive learning objective that constrains the representation proximity of weakly-aligned visio-textual inputs according to textual relevance. Besides, existing V&L pre-training approaches mainly deal with visual inputs by either region-of-interest (ROI) features or patch embeddings. We flexibly integrate the two forms of visual features into our model for pre-training and downstream multi-modal tasks. Extensive experiments on 5 downstream multi-modal tasks across 6 languages demonstrate the effectiveness of our proposed method over competitive contrast models with strong zero-shot capability.
Android-based Marketplace Application for Surakarta Local Products

During the Covid-19 pandemic, many companies suffered losses, leading to a reduction in employees and adversely affecting the community's economy. A significant portion of the local economy depends on the sale of locally produced goods. Moreover, the rapid advancement of information technology has intensified business competition, further impacting the development of local products. Unfortunately, the potential of information technology, particularly Marketplace platforms, to boost the economy and facilitate transactions between customers and sellers has been overlooked. To address these issues, this research aims to develop a Marketplace application to assist the community in selling and promoting their locally made products. The research involves several stages, including analysis, design, simulation prototyping, implementation, monitoring, and management. The result of this study is an Android-based Marketplace application designed to support the sale of local products.
Chatbot interaction development and assessment that assists teachers in inserting content into ITS MAZK

This paper aims to present the use of a chatbot as an alternative for teachers using an Intelli-gent Tutoring System and also verify MAZK's chatbot acceptance among the instructors, as this functionality was developed in MAZK's system exclusively for this purpose. A ques-tionnaire was carried out with 32 teachers, most of them beginners on MAZK. These in-structors should set up a class with the chatbot's help and it was noticeable that this re-source's use in their training was well accepted due to the practicality of an online tool. Still, some of the instructors had questions if they could learn all the aspects of the system only through the chatbot's help. Teachers who had previously used MAZK considered the chat-bot's development a necessary improvement as a supporting tool.
Neural Machine Translation for the Indigenous Languages of the Americas: An Introduction

Neural models have drastically advanced state of the art for machine translation (MT) between high-resource languages.Traditionally, these models rely on large amounts of training data, but many language pairs lack these resources.However, an important part of the languages in the world do not have this amount of data.Most languages from the Americas are among them, having a limited amount of parallel and monolingual data, if any.Here, we present an introduction to the interested reader to the basic challenges, concepts, and techniques that involve the creation of MT systems for these languages.Finally, we discuss the recent advances and findings and open questions, product of an increased interest of the NLP community in these languages.
How do humans perceive adversarial text? A reality check on the validity and naturalness of word-based adversarial attacks

Natural Language Processing (NLP) models based on Machine Learning (ML) are susceptible to adversarial attacks -malicious algorithms that imperceptibly modify input text to force models into making incorrect predictions.However, evaluations of these attacks ignore the property of imperceptibility or study it under limited settings.This entails that adversarial perturbations would not pass any human quality gate and do not represent real threats to human-checked NLP systems.To bypass this limitation and enable proper assessment (and later, improvement) of NLP model robustness, we have surveyed 378 human participants about the perceptibility of text adversarial examples produced by state-of-the-art methods.Our results underline that existing text attacks are impractical in real-world scenarios where humans are involved.This contrasts with previous smaller-scale human studies, which reported overly optimistic conclusions regarding attack success.Through our work, we hope to position human perceptibility as a first-class success criterion for text attacks, and provide guidance for research to build effective attack algorithms and, in turn, design appropriate defence mechanisms.
Teachers' digital competences: Concepts, models and challenges for the educational integration of technologies

Digital technologies have caused numerous transformations in educational contexts, revolutionizing pedagogical practices, inviting new organizations of school spaces, redesigning curricula and reinventing models of communication between teachers and students.
Pengaruh Penerapan Media Pembelajaran Monopoli Petualangan Nusantara terhadap Motivasi Belajar Siswa dalam Mata Pelajaran IPS Kelas V di Sekolah Dasar Negeri Timbang

This study aims to: (1) To find out the application of archipelago adventure monopoly learning media to students' learning motivation in social studies subjects in class V at SD Negeri Timbang. (2) To find out the effectiveness of the application of archipelago adventure monopoly learning media on student learning motivation in social studies class V at SD Negeri Timbang. (3) To find out the effect of the application of archipelago adventure monopoly learning media on students' learning motivation in social studies class V at SD Negeri Timbang. This study uses a Quasi Experimental Design research type. The research design used is True experimental design. This design is a type of research design that belongs to the Posttest-Only Control Design approach. In this study, the population was all fifth grade students at SD Negeri Timbang, Cigandamekar sub-district, while for the sample, there were 21 class Va students and 22 class Vb students. The average posttest result in the control class (Va) was 69. And the average score in the experiment class 86 which experienced an increase of 12.05. Based on the results of calculations using SPSS 29.00, it is known that the value of Tcount (3.111) &gt; Ttable (2.073) and a significance value of 0.003 &lt;0.05, it can be concluded that the variable influence of Monopoly Media Adventures of the Archipelago (X) affects students' learning motivation in social studies class V at SD Negeri Tambang Cigandamekar, Kuningan.
HiPool: Modeling Long Documents Using Graph Neural Networks

Encoding long sequences in Natural Language Processing (NLP) is a challenging problem.Though recent pretraining language models achieve satisfying performances in many NLP tasks, they are still restricted by a pre-defined maximum length, making them challenging to be extended to longer sequences.So some recent works utilize hierarchies to model long sequences.However, most of them apply sequential models for upper hierarchies, suffering from long dependency issues.In this paper, we alleviate these issues through a graph-based method.We first chunk the sequence with a fixed length to model the sentence-level information.We then leverage graphs to model intraand cross-sentence correlations with a new attention mechanism.Additionally, due to limited standard benchmarks for long document classification (LDC), we propose a new challenging benchmark, totaling six datasets with up to 53k samples and 4034 average tokens' length.Evaluation shows our model surpasses competitive baselines by 2.6% in F1 score, and 4.8% on the longest sequence dataset.Our method is shown to outperform hierarchical sequential models with better performance and scalability, especially for longer sequences.
UAlberta at SemEval-2023 Task 1: Context Augmentation and Translation for Multilingual Visual Word Sense Disambiguation

We describe the systems of the University of Alberta team for the SemEval-2023 Visual Word Sense Disambiguation (V-WSD) Task.We present a novel algorithm that leverages glosses retrieved from BabelNet, in combination with text and image encoders.Furthermore, we compare language-specific encoders against the application of English encoders to translated texts.As the contexts given in the task datasets are extremely short, we also experiment with augmenting these contexts with descriptions generated by a language model.This yields substantial improvements in accuracy.We describe and evaluate additional V-WSD methods which use image generation and text-conditioned image segmentation.Overall, the results of our official submission rank us 18 out of 56 teams.Some of our unofficial results are even better than the official ones.
FPT: Improving Prompt Tuning Efficiency via Progressive Training

Recently, prompt tuning (PT) has gained increasing attention as a parameter-efficient way of tuning pre-trained language models (PLMs).Despite extensively reducing the number of tunable parameters and achieving satisfying performance, PT is training-inefficient due to its slow convergence.To improve PT's training efficiency, we first make some novel observations about the prompt transferability of "partial PLMs", which are defined by compressing a PLM in depth or width.We observe that the soft prompts learned by different partial PLMs of various sizes are similar in the parameter space, implying that these soft prompts could potentially be transferred among partial PLMs.Inspired by these observations, we propose Fast Prompt Tuning (FPT), which starts by conducting PT using a small-scale partial PLM, and then progressively expands its depth and width until the full-model size.After each expansion, we recycle the previously learned soft prompts as initialization for the enlarged partial PLM and then proceed PT.We demonstrate the feasibility of FPT on 5 tasks and show that FPT could save over 30% training computations while achieving comparable performance.The codes are publicly available at https://github.com/thunlp/FastPromptTuning.
Unsupervised Semantic Variation Prediction using the Distribution of Sibling Embeddings

Languages are dynamic entities, where the meanings associated with words constantly change with time.Detecting the semantic variation of words is an important task for various NLP applications that must make time-sensitive predictions.Existing work on semantic variation prediction have predominantly focused on comparing some form of an averaged contextualised representation of a target word computed from a given corpus.However, some of the previously associated meanings of a target word can become obsolete over time (e.g. meaning of gay as happy), while novel usages of existing words are observed (e.g. meaning of cell as a mobile phone).We argue that mean representations alone cannot accurately capture such semantic variations and propose a method that uses the entire cohort of the contextualised embeddings of the target word, which we refer to as the sibling distribution.Experimental results on SemEval-2020 Task 1 benchmark dataset for semantic variation prediction show that our method outperforms prior work that consider only the mean embeddings, and is comparable to the current state-of-the-art. Moreover, a qualitative analysis shows that our method detects important semantic changes in words that are not captured by the existing methods.
A Prompt Based Approach for Euphemism Detection

Euphemism is an indirect way to express sensitive topics. People can comfortably communicate with each other about sensitive topics or taboos by using euphemisms. The Euphemism Detection Shared Task in the Third Workshop on Figurative Language Processing co-located with EMNLP 2022 provided a euphemism detection dataset that was divided into the train set and the test set. We made euphemism detection experiments by prompt tuning pre-trained language models on the dataset. We used RoBERTa as the pre-trained language model and created suitable templates and verbalizers for the euphemism detection task. Our approach achieved the third-best score in the euphemism detection shared task. This paper describes our model participating in the task.
Evaluation of Response Generation Models: Shouldn’t It Be Shareable and Replicable?

Human Evaluation (HE) of automatically generated responses is necessary for the advancement of human-machine dialogue research.Current automatic evaluation measures are poor surrogates, at best.There are no agreed-upon HE protocols and it is difficult to develop them.As a result, researchers either perform nonreplicable, non-transparent and inconsistent procedures or, worse, limit themselves to automated metrics.We propose to standardize the human evaluation of response generation models by publicly sharing a detailed protocol.The proposal includes the task design, annotators recruitment, task execution, and annotation reporting.Such protocol and process can be used as-is, as-a-whole, in-part, or modified and extended by the research community.We validate the protocol by evaluating two conversationally fine-tuned state-of-the-art models (GPT-2 and T5) for the complex task of personalized response generation.We invite the community to use this protocol -or its future community amended versions -as a transparent, replicable, and comparable approach to HE of generated responses 1 .
Food for Thought: How can we exploit contextual embeddings in the translation of idiomatic expressions?

Idiomatic expressions (or idioms) are phrases where the meaning of the phrase cannot be determined from the meaning of the individual words in the expression.Translating idioms between languages is therefore a challenging task.Transformer models based on contextual embeddings have advanced the state-ofthe-art across many domains in the field of natural language processing.While research using transformers has advanced both idiom detection as well as idiom disambiguation, idiom translation has not seen a similar advancement.In this work, we investigate two approaches to fine-tuning a pretrained Text-to-Text Transfer Transformer (T5) model to perform idiom translation from English to German.The first approach directly translates English idiom-containing sentences to German, while the second is underpinned by idiom paraphrasing, firstly paraphrasing English idiomatic expressions to their simplified English versions before translating them to German.Results of our evaluation show that each of the approaches is able to generate adequate translations.
Grounded Keys-to-Text Generation: Towards Factual Open-Ended Generation

Large pre-trained language models have recently enabled open-ended generation frameworks (e.g., prompt-to-text NLG) to tackle a variety of tasks going beyond the traditional data-to-text generation.While this framework is more general, it is under-specified and often leads to a lack of controllability restricting their real-world usage.We propose a new grounded keys-to-text generation task: the task is to generate a factual description about an entity given a set of guiding keys, and grounding passages.To address this task, we introduce a new dataset, called ENTDEGEN.Inspired by recent QAbased evaluation measures, we propose an automatic metric, MAFE, for factual correctness of generated descriptions.Our ENTDESCRIP-TOR model is equipped with strong rankers to fetch helpful passages and generate entity descriptions.Experimental result shows a good correlation (60.14) between our proposed metric and human judgments of factuality.Our rankers significantly improved the factual correctness of generated descriptions (15.95% and 34.51% relative gains in recall and precision).Finally, our ablation study highlights the benefit of combining keys and groundings.
A Token-pair Framework for Information Extraction from Dialog Transcripts in SereTOD Challenge

This paper describes our solution for Sere-TOD Challenge Track 1: Information extraction from dialog transcripts.We propose a token-pair framework to simultaneously identify entity and value mentions and link them into corresponding triples.As entity mentions are usually coreferent, we adopt a baseline model for coreference resolution.We exploit both annotated transcripts and unsupervised dialogs for training.With model ensemble and post-processing strategies, our system significantly outperforms the baseline solution and ranks first in triple f1 and third in entity f1.
An Empirical Revisiting of Linguistic Knowledge Fusion in Language Understanding Tasks

Though linguistic knowledge emerges during large-scale language model pretraining, recent work attempt to explicitly incorporate humandefined linguistic priors into task-specific finetuning.Infusing language models with syntactic or semantic knowledge from parsers has shown improvements on many language understanding tasks.To further investigate the effectiveness of structural linguistic priors, we conduct empirical study of replacing parsed graphs or trees with trivial ones (rarely carrying linguistic knowledge e.g., balanced tree) for tasks in the GLUE benchmark.Encoding with trivial graphs achieves competitive or even better performance in fully-supervised and few-shot settings.It reveals that the gains might not be significantly attributed to explicit linguistic priors but rather to more feature interactions brought by fusion layers.Hence we call for attention to using trivial graphs as necessary baselines to design advanced knowledge fusion methods in the future.