Reviewerly/exHarmony huggingface dataset:
Text Representation Distillation via Information Bottleneck Principle

Pre-trained language models (PLMs) have recently shown great success in text representation field. However, the high computational cost and high-dimensional representation of PLMs pose significant challenges for practical applications. To make models more accessible, an effective method is to distill large models into smaller representation models. In order to relieve the issue of performance degradation after distillation, we propose a novel Knowledge Distillation method called IBKD. This approach is motivated by the Information Bottleneck principle and aims to maximize the mutual information between the final representation of the teacher and student model, while simultaneously reducing the mutual information between the student model’s representation and the input data. This enables the student model to preserve important learned information while avoiding unnecessary information, thus reducing the risk of over-fitting. Empirical studies on two main downstream applications of text representation (Semantic Textual Similarity and Dense Retrieval tasks) demonstrate the effectiveness of our proposed approach.
Towards Example-Based NMT with Multi-Levenshtein Transformers

Retrieval-Augmented Machine Translation (RAMT) is attracting growing attention. This is because RAMT not only improves translation metrics, but is also assumed to implement some form of domain adaptation. In this contribution, we study another salient trait of RAMT, its ability to make translation decisions more transparent by allowing users to go back to examples that contributed to these decisions. For this, we propose a novel architecture aiming to increase this transparency. This model adapts a retrieval-augmented version of the Levenshtein Transformer and makes it amenable to simultaneously edit multiple fuzzy matches found in memory. We discuss how to perform training and inference in this model, based on multi-way alignment algorithms and imitation learning. Our experiments show that editing several examples positively impacts translation scores, notably increasing the number of target spans that are copied from existing instances.
STEER: Semantic Turn Extension-Expansion Recognition for Voice Assistants

Leon Zhang, Jiarui Lu, Joel Ruben Antony Moniz, Aditya Kulkarni, Dhivya Piraviperumal, Tien Dung Tran, Nick Tzou, Hong Yu. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Industry Track. 2023.
Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedback

Viet Lai, Chien Nguyen, Nghia Ngo, Thuat Nguyen, Franck Dernoncourt, Ryan Rossi, Thien Nguyen. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. 2023.
Findings of the WMT 2023 Shared Task on Discourse-Level Literary Translation: A Fresh Orb in the Cosmos of LLMs

Longyue Wang, Zhaopeng Tu, Yan Gu, Siyou Liu, Dian Yu, Qingsong Ma, Chenyang Lyu, Liting Zhou, Chao-Hong Liu, Yufeng Ma, Weiyu Chen, Yvette Graham, Bonnie Webber, Philipp Koehn, Andy Way, Yulin Yuan, Shuming Shi. Proceedings of the Eighth Conference on Machine Translation. 2023.
CRUSH4SQL: Collective Retrieval Using Schema Hallucination For Text2SQL

Existing Text-to-SQL generators require the entire schema to be encoded with the user text. This is expensive or impractical for large databases with tens of thousands of columns. Standard dense retrieval techniques are inadequate for schema subsetting of a large structured database, where the correct semantics of retrieval demands that we rank sets of schema elements rather than individual documents. In response, we propose a two-stage process for effective coverage during retrieval. First, we use an LLM to hallucinate a minimal DB schema that it deems adequate to answer the query. We use the hallucinated schema to retrieve a subset of the actual schema, by composing the results from multiple dense retrievals. Remarkably, hallucination — generally considered a nuisance — turns out to be actually useful as a bridging mechanism. Since no existing benchmarks exist for schema subsetting on large databases, we introduce two benchmarks: (1) A semi-synthetic dataset of 4502 schema elements, by taking a union of schema on the well-known SPIDER dataset, and (2) A real-life benchmark called SocialDB sourced from an actual large data warehouse comprising of 17844 schema elements. We show that our method leads to significantly higher recall than SOTA retrieval-based augmentation methods.
CLEME: Debiasing Multi-reference Evaluation for Grammatical Error Correction

Evaluating the performance of Grammatical Error Correction (GEC) systems is a challenging task due to its subjectivity. Designing an evaluation metric that is as objective as possible is crucial to the development of GEC task. However, mainstream evaluation metrics, i.e., reference-based metrics, introduce bias into the multi-reference evaluation by extracting edits without considering the presence of multiple references. To overcome this issue, we propose Chunk-LE Multi-reference Evaluation (CLEME), designed to evaluate GEC systems in the multi-reference evaluation setting. CLEME builds chunk sequences with consistent boundaries for the source, the hypothesis and references, thus eliminating the bias caused by inconsistent edit boundaries. Furthermore, we observe the consistent boundary could also act as the boundary of grammatical errors, based on which the F0.5 score is then computed following the correction independence assumption. We conduct experiments on six English reference sets based on the CoNLL-2014 shared task. Extensive experiments and detailed analyses demonstrate the correctness of our discovery and the effectiveness of CLEME. Further analysis reveals that CLEME is robust to evaluate GEC systems across reference sets with varying numbers of references and annotation styles. All the source codes of CLEME are released at https://github.com/THUKElab/CLEME.
Small Language Models Fine-tuned to Coordinate Larger Language Models improve Complex Reasoning

Large Language Models (LLMs) prompted to generate chain-of-thought (CoT) exhibit impressive reasoning capabilities. Recent attempts at prompt decomposition toward solving complex, multi-step reasoning problems depend on the ability of the LLM to simultaneously decompose and solve the problem. A significant disadvantage is that foundational LLMs are typically not available for fine-tuning, making adaptation computationally prohibitive. We believe (and demonstrate) that problem decomposition and solution generation are distinct capabilites, better addressed in separate modules, than by one monolithic LLM. We introduce DaSLaM, which uses a decomposition generator to decompose complex problems into subproblems that require fewer reasoning steps. These subproblems are answered by a solver. We use a relatively small (13B parameters) LM as the decomposition generator, which we train using policy gradient optimization to interact with a solver LM (regarded as black-box) and guide it through subproblems, thereby rendering our method solver-agnostic. Evaluation on multiple different reasoning datasets reveal that with our method, a 175 billion parameter LM (text-davinci-003) can produce competitive or even better performance, compared to its orders-of-magnitude larger successor, GPT-4. Additionally, we show that DaSLaM is not limited by the solver’s capabilities as a function of scale; e.g., solver LMs with diverse sizes give significant performance improvement with our solver-agnostic decomposition technique. Exhaustive ablation studies evince the superiority of our modular finetuning technique over exorbitantly large decomposer LLMs, based on prompting alone.
Self-supervised Post-processing Method to Enrich Pretrained Word Vectors

Retrofitting techniques, which inject external resources into word representations, have compensated for the weakness of distributed representations in semantic and relational knowledge between words. However, the previous methods require additional external resources and strongly depend on the lexicon. To address the issues, we propose a simple extension of extrofitting, self-supervised extrofitting: extrofitting by its own word vector distribution. Our methods improve the vanilla embeddings on all of word similarity tasks without any external resources. Moreover, the method is also effective in various languages, which implies that our method will be useful in lexicon-scarce languages. As downstream tasks, we show its benefits in dialogue state tracking and text classification tasks, reporting better and generalized results compared to other word vector specialization methods.
The Locality and Symmetry of Positional Encodings

Positional Encodings (PEs) are used to inject word-order information into transformer-based language models. While they can significantly enhance the quality of sentence representations, their specific contribution to language models is not fully understood, especially given recent findings that various positional encodings are insensitive to word order. In this work, we conduct a systematic study of positional encodings in Bidirectional Masked Language Models (BERT-style) , which complements existing work in three aspects: (1) We uncover the core function of PEs by identifying two common properties, Locality and Symmetry; (2) We show that the two properties are closely correlated with the performances of downstream tasks; (3) We quantify the weakness of current PEs by introducing two new probing tasks, on which current PEs perform poorly. We believe that these results are the basis for developing better PEs for transformer-based language models.
PRHLT’s Submission to WLAC 2023

This paper describes our submission to the Word-Level AutoCompletion shared task of WMT23. We participated in the English–German and German–English categories. We extended our last year segment-based interactive machine translation approach to address its weakness when no context is available. Additionally, we fine-tune the pre-trained mT5 large language model to be used for autocompletion.
Symbol tuning improves in-context learning in language models

Jerry Wei, Le Hou, Andrew Lampinen, Xiangning Chen, Da Huang, Yi Tay, Xinyun Chen, Yifeng Lu, Denny Zhou, Tengyu Ma, Quoc Le. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023.
Non-Programmers Can Label Programs Indirectly via Active Examples: A Case Study with Text-to-SQL

Can non-programmers annotate natural language utterances with complex programs that represent their meaning? We introduce APEL, a framework in which non-programmers select among candidate programs generated by a seed semantic parser (e.g., Codex). Since they cannot understand the candidate programs, we ask them to select indirectly by examining the programs’ input-ouput examples. For each utterance, APEL actively searches for a simple input on which the candidate programs tend to produce different outputs. It then asks the non-programmers only to choose the appropriate output, thus allowing us to infer which program is correct and could be used to fine-tune the parser. As a first case study, we recruited human non-programmers to use APEL to re-annotate SPIDER, a text-to-SQL dataset. Our approach achieved the same annotation accuracy as the original expert annotators (75%) and exposed many subtle errors in the original annotations.
Multilingual Generation and Answering of Questions from Texts and Knowledge Graphs

The ability to bridge Question Generation (QG) and Question Answering (QA) across structured and unstructured modalities has the potential for aiding different NLP applications. One key application is in QA-based methods that have recently been shown to be useful for automatically evaluating Natural Language (NL) texts generated from Knowledge Graphs (KG). While methods have been proposed for QG-QA across these modalities, these efforts have been in English only; in this work, we bring multilinguality (Brazilian Portuguese and Russian) to multimodal (KG and NL) QG-QA. Using synthetic data generation and machine translation to produce QG-QA data that is aligned between graph and text, we are able to train multimodal, multi-task models that can perform multimodal QG and QA in Portuguese and Russian. We show that our approach outperforms a baseline which is derived from previous work on English and adapted to handle these two languages.
ScanDL: A Diffusion Model for Generating Synthetic Scanpaths on Texts

Eye movements in reading play a crucial role in psycholinguistic research studying the cognitive mechanisms underlying human language processing. More recently, the tight coupling between eye movements and cognition has also been leveraged for language-related machine learning tasks such as the interpretability, enhancement, and pre-training of language models, as well as the inference of reader- and text-specific properties. However, scarcity of eye movement data and its unavailability at application time poses a major challenge for this line of research. Initially, this problem was tackled by resorting to cognitive models for synthesizing eye movement data. However, for the sole purpose of generating human-like scanpaths, purely data-driven machine-learning-based methods have proven to be more suitable. Following recent advances in adapting diffusion processes to discrete data, we propose ScanDL, a novel discrete sequence-to-sequence diffusion model that generates synthetic scanpaths on texts. By leveraging pre-trained word representations and jointly embedding both the stimulus text and the fixation sequence, our model captures multi-modal interactions between the two inputs. We evaluate ScanDL within- and across-dataset and demonstrate that it significantly outperforms state-of-the-art scanpath generation methods. Finally, we provide an extensive psycholinguistic analysis that underlines the model’s ability to exhibit human-like reading behavior. Our implementation is made available at https://github.com/DiLi-Lab/ScanDL.
Exchange-of-Thought: Enhancing Large Language Model Capabilities through Cross-Model Communication

Large Language Models (LLMs) have recently made significant strides in complex reasoning tasks through the Chain-of-Thought technique. Despite this progress, their reasoning is often constrained by their intrinsic understanding, lacking external insights. To address this, we propose Exchange-of-Thought (EoT), a novel framework that enables cross-model communication during problem-solving. Drawing inspiration from network topology, EoT integrates four unique communication paradigms: Memory, Report, Relay, and Debate. This paper delves into the communication dynamics and volume associated with each paradigm. To counterbalance the risks of incorrect reasoning chains, we implement a robust confidence evaluation mechanism within these communications. Our experiments across diverse complex reasoning tasks demonstrate that EoT significantly surpasses established baselines, underscoring the value of external insights in enhancing LLM performance. Furthermore, we show that EoT achieves these superior results in a cost-effective manner, marking a promising advancement for efficient and collaborative AI problem-solving.
Mixture of Soft Prompts for Controllable Data Generation

Large language models (LLMs) effectively generate fluent text when the target output follows natural language patterns. However, structured prediction tasks confine the output format to a limited ontology, causing even very large models to struggle since they were never trained with such restrictions in mind. The difficulty of using LLMs for direct prediction is exacerbated in few-shot learning scenarios, which commonly arise due to domain shift and resource limitations. We flip the problem on its head by leveraging the LLM as a tool for data augmentation rather than direct prediction. Our proposed Mixture of Soft Prompts (MSP) serves as a parameter-efficient procedure for generating multi-attribute data in a controlled manner. Denoising mechanisms are further applied to improve the quality of synthesized data. Automatic metrics show our method is capable of producing diverse and natural text, while preserving label semantics. Moreover, MSP achieves state-of-the-art results on three benchmarks when compared against strong baselines. Our method offers an alternate data-centric approach for applying LLMs to complex prediction tasks.
Drilling Down into the Discourse Structure with LLMs for Long Document Question Answering

We address the task of evidence retrieval for long document question answering, which involves locating relevant paragraphs within a document to answer a question. We aim to assess the applicability of large language models (LLMs) in the task of zero-shot long document evidence retrieval, owing to their unprecedented performance across various NLP tasks. However, currently the LLMs can consume limited context lengths as input, thus providing document chunks as inputs might overlook the global context while missing out on capturing the inter-segment dependencies. Moreover, directly feeding the large input sets can incur significant computational costs, particularly when processing the entire document (and potentially incurring monetary expenses with enterprise APIs like OpenAI’s GPT variants). To address these challenges, we propose a suite of techniques that exploit the discourse structure commonly found in documents. By utilizing this structure, we create a condensed representation of the document, enabling a more comprehensive understanding and analysis of relationships between different parts. We retain 99.6% of the best zero-shot approach’s performance, while processing only 26% of the total tokens used by the best approach in the information seeking evidence retrieval setup. We also show how our approach can be combined with *self-ask* reasoning agent to achieve best zero-shot performance in complex multi-hop question answering, just ≈ 4% short of zero-shot performance using gold evidence.
Comparison to the Iron Level on the well water and PDAM water sample using ICP-OES Spectrophotometer

Iron (Fe) is one of the elements that can be found in almost every place on earth, in all geological levels, and in all water bodies. In general, the iron in the water can be dissolved. The content of Fe ions in drilled well water can range between 5-7 mg/L. ICP-OES spectrophotometry is a type of ICP that utilizes plasma as a source of atomization and excitation. The purpose of this research is to determine the difference in Fe content between well water samples and PDAM water samples in the Pemulutan sub-district using the ICP-OES spectrophotometer. The type of research used was cross-sectional. The population in this study was well-watered and PDAM-watered in Pemulutan District. The sampling technique in this study was simply random. The place of this research was in the Palembang Health Laboratory Center. The results showed that the average difference in the examination of Fe levels in well water samples and PDAM water using the sda was 0.13198 mg/L for well water and 0.00445 mg/L for PDAM water. The conclusions were that PDAM water is safer to use for sanitation activities or consumption purposes than well water. The iron (Fe) content in well water is higher than the iron (Fe) content in PDAM water.
Pelatihan Peningkatan Berbicara Bahasa Inggris Dengan Metode Audio Lingual

The Audio Lingual (MAL) method refers to a very large learning competence in the ability to speak English. PPA Filadelpia Medan Foundation is an educational foundation that is oriented towards the progress of students to make children who are moral and achievers. This foundation also accommodates the children of the community who have a weak or underprivileged economy. Teachers who teach students have been set to become teachers who have a holistic spirit and dreams. Teachers who teach English and non-English subjects are very interested in learning English in order to develop their students. The trainer for Community Service Activities has a big heart to develop this foundation through English speaking training for teachers and foundation employees. The results conclude that the implementation of PKM activities starts from May 2021 to July 2021 for 12 times with meeting times with a total of 16 participants with system clusters. The Audio Lingual (MAL) method with a combination of Screen Cast O Matic becomes an instrument to carry out face-to-face training and online learning. The results of the observations prove that under the answers and learning outcomes through the audio lingual approach there is a tendency to learn, speak English and explore sources and information that are shown. This is evidenced by the statements of the respondents to prepare them in the world market which must be full of creativity that is flexible, directed, and achieved with high quality..
Pembuatan Totebag Berbahan Dasar Kain Sisa Hasil Produksi Dengan Memanfaatkan Tukang Jahit Keliling Guna Meningkatkan Perekonomian Masyarakat Kota Solo

&#x0D; &#x0D; &#x0D; &#x0D; &#x0D; &#x0D; &#x0D; &#x0D; 292&#x0D; &#x0D; &#x0D; &#x0D; &#x0D; Poverty is one of the problems that often arise in various cities and even countries. Especially in this era of the Covid-19 pandemic. It is undeniable that this pandemic has resulted in a decline in the economy in Indonesia, especially in the city of Solo. The city of Solo is one of the cultural cities that has the potential for Batik as one of the icons of this city. Not only the potential of craft resources, the potential of human resources, which is often called craftsmen, is also widely found in this city. One of them is the itinerant tailor who goes around every day from one complex to another to help sew the clothes of the local residents. However, with the impact of COVID-19, many itinerant tailors have started to lose their jobs and have a negative impact on their economic growth. This has a negative impact on the sustainable development goals in point 8 regarding decent work and economic growth. In this paper, we use the Design Thinking method which aims to help improve the economy of the people in the city of Solo and also increase the potential of human resources through itinerant tailors who are our partners in processing the waste of the remaining batik cloth produced into a tote bag with a beautiful and elegant design. modern&#x0D; &#x0D; &#x0D; &#x0D;
Experimental Demonstration of Neural Adaptive Quantum Tomography for Two-Qubit States

We experimentally implement a neural adaptive quantum tomography scheme for two-qubit states. We aim to demonstrate speed-up and retained accuracy in the reconstruction compared to Bayesian adaptive quantum tomography.
A Wearable System for Continuous Monitoring and Assessment of Speech, Gait, and Cognitive Decline for Early Diagnosis of ADRD

Early detection of cognitive decline is essential to study mild cognitive impairment and Alzheimer’s Disease in order to develop targeted interventions and prevent or stop the progression of dementia. This requires continuous and longitudinal assessment and tracking of the related physiological and behavioral changes during daily life. In this paper, we present a low cost and low power wearable system custom designed to track the trends in speech, gait, and cognitive stress while also considering the important human factor needs such as privacy and compliance. In the form factors of a wristband and waist-patch, this multimodal, multi-sensor system measures inertial signals, sound, heart rate, electrodermal activity and pulse transit time. A total power consumption of 2.6 mW without any duty cycling allows for more than 3 weeks of run time between charges when 1500 mAh batteries are used.Clinical Relevance— Much earlier detection of Alzheimer’s disease and related dementias may be possible by continuous monitoring of physiological and behavioral state using application specific wearable sensors during the activities of daily life.