Reviewerly/exHarmony huggingface dataset:
JiuZhang 2.0: A Unified Chinese Pre-trained Language Model for Multi-task Mathematical Problem Solving

Although pre-trained language models~(PLMs) have recently advanced the research progress in mathematical reasoning, they are not specially designed as a capable multi-task solver, suffering from high cost for multi-task deployment (e.g. a model copy for a task) and inferior performance on complex mathematical problems in practical applications. To address these issues, we propose JiuZhang 2.0, a unified Chinese PLM specially for multi-task mathematical problem solving. Our idea is to maintain a moderate-sized model and employ the cross-task knowledge sharing to improve the model capacity in a multi-task setting. Specially, we construct a Mixture-of-Experts (MoE) architecture for modeling mathematical text, to capture the common mathematical knowledge across tasks. For optimizing the MoE architecture, we design multi-task continual pre-training and multi-task fine-tuning strategies for multi-task adaptation. These training strategies can effectively decompose the knowledge from the task data and establish the cross-task sharing via expert networks. To further improve the general capacity of solving different complex tasks, we leverage large language models (LLMs) as complementary models to iteratively refine the generated solution by our PLM, via in-context learning. Extensive experiments have demonstrated the effectiveness of our model.
FS-REAL: Towards Real-World Cross-Device Federated Learning

Federated Learning (FL) aims to train high-quality models in collaboration with distributed clients while not uploading their local data, which attracts increasing attention in both academia and industry. However, there is still a considerable gap between the flourishing FL research and real-world scenarios, mainly caused by the characteristics of heterogeneous devices and its scales. Most existing works conduct evaluations with homogeneous devices, which are mismatched with the diversity and variability of heterogeneous devices in real-world scenarios. Moreover, it is challenging to conduct research and development at scale with heterogeneous devices due to limited resources and complex software stacks. These two key factors are important yet underexplored in FL research as they directly impact the FL training dynamics and final performance, making the effectiveness and usability of FL algorithms unclear. To bridge the gap, in this paper, we propose an efficient and scalable prototyping system for real-world cross-device FL, FS-REAL. It supports heterogeneous device runtime, contains parallelism and robustness enhanced FL server, and provides implementations and extensibility for advanced FL utility features such as personalization, communication compression and asynchronous aggregation. To demonstrate the usability and efficiency of FS-REAL, we conduct extensive experiments with various device distributions, quantify and analyze the effect of the heterogeneous device and various scales, and further provide insights and open discussions about real-world FL scenarios. Our system is released to help to pave the way for further real-world FL research and broad applications involving diverse devices and scales.
TWIN: Personalized Clinical Trial Digital Twin Generation

Clinical trial digital twins are virtual patients that reflect personal characteristics in a high degree of granularity and can be used to simulate various patient outcomes under different conditions. With the growth of clinical trial databases captured by Electronic Data Capture (EDC) systems, there is a growing interest in using machine learning models to generate digital twins. This can benefit the drug development process by reducing the sample size required for participant recruitment, improving patient outcome predictive modeling, and mitigating privacy risks when sharing synthetic clinical trial data. However, prior research has mainly focused on generating Electronic Healthcare Records (EHRs), which often assume large training data and do not account for personalized synthetic patient record generation. In this paper, we propose a sample-efficient method TWIN for generating personalized clinical trial digital twins. TWIN can produce digital twins of patient-level clinical trial records with high fidelity to the targeting participant's record and preserves the temporal relations across visits and events. We compare our method with various baselines for generating real-world patient-level clinical trial data. The results show that TWIN generates synthetic trial data with high fidelity to facilitate patient outcome predictions in low-data scenarios and strong privacy protection against real patients from the trials.
HiMacMic: Hierarchical Multi-Agent Deep Reinforcement Learning with Dynamic Asynchronous Macro Strategy

Multi-agent deep reinforcement learning (MADRL) has been widely used in many scenarios such as robotics and game AI. However, existing methods mainly focus on the optimization of agents' micro policies without considering the macro strategy. As a result, they cannot perform well in complex or sparse reward scenarios like the StarCraft Multi-Agent Challenge (SMAC) and Google Research Football (GRF). To this end, we propose a hierarchical MADRL framework called "HiMacMic" with dynamic asynchronous macro strategy. Spatially, HiMacMic determines a critical position by using a positional heat map. Temporally, the macro strategy dynamically decides its deadline and updates it asynchronously among agents. We validate HiMacMic in four widely used benchmarks, namely: Overcooked, GRF, SMAC and SMAC-v2 with nine chosen scenarios. Results show that HiMacMic not only converges faster and achieves higher results than ten existing approaches, but also shows its adaptability to different environment settings.
Pretrained Language Representations for Text Understanding: A Weakly-Supervised Perspective

Language representations pretrained on general-domain corpora and adapted to downstream task data have achieved enormous success in building natural language understanding (NLU) systems. While the standard supervised fine-tuning of pretrained language models (PLMs) has proven an effective approach for superior NLU performance, it often necessitates a large quantity of costly human-annotated training data. For example, the enormous success of ChatGPT and GPT-4 can be largely credited to their supervised fine-tuning with massive manually-labeled prompt-response training pairs. Unfortunately, obtaining large-scale human annotations is in general infeasible for most practitioners. To broaden the applicability of PLMs to various tasks and settings, weakly-supervised learning offers a promising direction to minimize the annotation requirements for PLM adaptions.
Privacy in Advertising: Analytics and Modeling

Privacy in general, and differential privacy (DP) in particular, have become important topics in data mining and machine learning. Digital advertising is a critical component of the internet and is powered by large-scale data analytics and machine learning models; privacy concerns around these are on the rise. Despite the central importance of private ad analytics and training privacy-preserving ad prediction models, there has been relatively little exposure of this subject to the broader KDD community. In the past three years, the interest in privacy and the interest in online advertising have been steadily growing in KDD. The aim of this tutorial is to provide KDD researchers with an introduction to the problems that arise in private analytics and modeling in advertising, survey recent results, and describe the main research challenges in the space.
What Is a Semantic Annotation?

This chapter is about semantic annotation, discussed from formal and computational perspectives. Annotation is viewed as a scholarly technique or methodology. This chapter explains what annotation was in general and is now, how annotation has gradually developed and become applicable to the automatic building of larger data in language, and what applications semantic annotation aims at and what principles govern the modeling of semantic annotation schemes. There are two governing principles discussed: the partiality and situatedness of information to be annotated. I also mention the use of machine learning techniques for the automatic annotation of language data, which is represented either in a textual form or in a graphic form.
Normalizing TimeML with Some Modifications

In this chapter, I explain how TimeML, a specification language for the annotation of event-associated temporal expressions in language, was normalized as an ISO international standard, known as ISO-TimeML, with some modifications. ISO’s working group developed TimeML into an ISO standard on event-associated temporal annotation by making four modifications of TimeML: (i) abstract specification of the annotation scheme, (ii) adoption of standoff annotation, (iii) merging of two tags, <EVENT/> and <MAKEINSTNACE/>, to a single tag <EVENT/>, and (iv) treating duration (e.g., two hours) as measurement. Following Bunt’s (2010) proposal for the construction of semantic annotation schemes and his subsequent work, I then formalize ISO-TimeML by presenting a metamodel for its design, an abstract syntax for formulating its specification language in set-theoretic terms, and an XML-based concrete syntax. I also analyze base structures as consisting of two substructures, anchoring and content structures, into the annotation structures of the normalized TimeML.
Modeling a Semantic Annotation Scheme

In this chapter, I discuss how the abstract syntax for a semantic annotation scheme is modeled on a formal grammar of language. I design a semantic annotation scheme as consisting of three components: a (nonempty) set of annotation structures, a syntax, and a semantics. The metatheoretic syntax formally defines, or generates, annotation structures each of which consists of base structures and link structures. The semantics interprets annotation structures, while validating the formulation of the syntax. I also discuss how the specification of attribute-value assignments determine the well-formedness of annotation structures and its substructures, anchoring and content (feature) structures. This chapter focuses on the formulation of an abstract syntax.
Dynamic Paths, Projection, and Orientation

In this chapter, I introduce the four types of category path: static, dynamic, oriented, and projected, while characterizing them for the interpretation of path-related information in language. Static paths are finite paths with two endpoints, but neither of the endpoints is identified intrinsically as the start or the end of a path. Dynamic paths are trajectories caused by motions. Oriented paths are simply directed to some goals and may not reach the goals. Projected paths are virtual or intended, which are not actually traversed but devised in the mind of a human or rational agent. To discuss their characteristic features in formal terms, I introduce Pustejovsky and Yocum’s (2013) axioms on motions and derive a corollary based on them. This corollary relates a mover to an event-path. I then show how the movement link (moveLink) is reformulated to link a mover to a motion-triggered event-path with the relation traverses. I also analyze the notions of orientation and projection with respect to the frames of reference, either absolute, relative, or intrinsic, while showing how these frames apply to the annotation and interpretation of oriented or projected paths.
Representation and Serialization

This chapter introduces a concrete syntax. It is ideally isomorphic to an abstract syntax that specifies a semantic annotation scheme, while providing a format for representing annotation structures. This format can represent them either in a serialized way from left to right, or in graphic images or tabular forms with linking arrows. Representation formats may vary, depending on kinds of the use of annotation. Human readers, for instance, prefer tabular formats especially for illustrations or demonstrations. For the purposes of merging, comparing, or exchanging various types of annotations or different annotations of the same type, graphs are considered useful. In this chapter, I introduce a graphic annotation format, called GrAF, for linguistic annotation. For the construction of larger corpora, however, there are practical computing reasons to prefer a serialization of annotations. For the serialized representation of annotation structures, this chapter mainly discusses two formats: (i) XML and (ii) pFormat, a predicate-logic-like representation format, which represents annotation structures in a strictly serial (linear) manner by avoiding embedded structures.
Real-Time Sign Language Recognition using a Multimodal Deep Learning Approach

Sign language recognition is an important area of research that aims to provide greater access to communication and information for individuals who are deaf or hard of hearing. In this paper, we present a new approach for real-time sign language recognition using a multimodal deep learning approach. The proposed approach integrates video and inertial sensor data for improved recognition accuracy and robustness.The proposed approach uses a convolutional neural network (CNN) to extract features from the video data and a Recurrent neural network (RNN) is used to capture the temporal dynamics of the sign language gestures.Sign language recognition systems use technologies such as computer vision, machine learning, and artificial intelligence to analyze and understand the gestures and movements of the signer. These systems have the potential to improve the quality of life for individuals who are deaf or hard of hearing by providing them with greater access to communication and information in a variety of settings.
Two-step Masked Language Model for Domain-adapting Multi-modal Task-oriented Dialogue Systems

The next generation of artificial intelligence (AI) is required to be capable of proper communication to enable eloquent interaction with human beings. Thus, AI models require powerful language understanding and generation capabilities. Therefore, designing an adequate dialogue-specific task is an important topic in dialogue research. In this paper, we improve our model that won in two out of four subtasks in SIMMC2.0 challenge using a dialogue-specific pre-training task, which learns the distinctive features of dialogues. In addition, we introduce an efficient method to pre-train the encoder-decoder transformer using an auxiliary task for the encoder. Experimental results indicate that superior performance is achieved compared to our previous model, revealing a new approach to improve task-oriented dialogue systems.
Open-Domain Hierarchical Event Schema Induction by Incremental Prompting and Verification

Event schemas are a form of world knowledge about the typical progression of events.Recent methods for event schema induction use information extraction systems to construct a large number of event graph instances from documents, and then learn to generalize the schema from such instances.In contrast, we propose to treat event schemas as a form of commonsense knowledge that can be derived from large language models (LLMs).This new paradigm greatly simplifies the schema induction process and allows us to handle both hierarchical relations and temporal relations between events in a straightforward way.Since event schemas have complex graph structures, we design an incremental prompting and verification method INCSCHEMA to break down the construction of a complex event graph into three stages: event skeleton construction, event expansion, and event-event relation verification.Compared to directly using LLMs to generate a linearized graph, INCSCHEMA can generate large and complex schemas with 7.2% F1 improvement in temporal relations and 31.0%F1 improvement in hierarchical relations.In addition, compared to the previous state-of-the-art closed-domain schema induction model, human assessors were able to cover ∼10% more events when translating the schemas into coherent stories and rated our schemas 1.3 points higher (on a 5-point scale) in terms of readability. 1 Relation Allen's base relations e1 starts before e2? e1 ends before e2? e1 duration longer than e2? e1 ≺ e2 e1 precedes e2, e1 meets e2 Yes Yes -e1 ≻ e2 e1 is preceded by e2, e1 is met by e2 No No -e1 ⊂ e2 e1 starts e2, e1 during e2, e1 finishes e2 No Yes No e1 ⊃ e2 e1 is started by e2, e1 contains e2, e1 is finished by e2 Yes No Yes e1 ∥ e2 e1 overlaps with e2, e1 is equal to e2 Yes No No e1 ∥ e2 e1 is overlapped by e2 No Yes Yes
Back Translation for Speech-to-text Translation Without Transcripts

The success of end-to-end speech-to-text translation (ST) is often achieved by utilizing source transcripts, e.g., by pre-training with automatic speech recognition (ASR) and machine translation (MT) tasks, or by introducing additional ASR and MT data.Unfortunately, transcripts are only sometimes available since numerous unwritten languages exist worldwide.In this paper, we aim to utilize large amounts of targetside monolingual data to enhance ST without transcripts.Motivated by the remarkable success of back translation in MT, we develop a back translation algorithm for ST (BT4ST) to synthesize pseudo ST data from monolingual target data.To ease the challenges posed by short-to-long generation and one-to-many mapping, we introduce self-supervised discrete units and achieve back translation by cascading a target-to-unit model and a unit-to-speech model.With our synthetic ST data, we achieve an average boost of 2.3 BLEU on MuST-C En→De, En→Fr, and En→Es datasets.More experiments show that our method is especially effective in low-resource scenarios.
Understanding and Improving the Robustness of Terminology Constraints in Neural Machine Translation

In this work, we study the robustness of two typical terminology translation methods: Placeholder (PH) and Code-Switch (CS), concerning (1) the number of constraints and (2) the target constraint length.We identify that existing terminology constraint test sets, such as IATE, Wiktionary, and TICO, are blind to this issue due to oversimplified constraint settings.To solve it, we create a new challenging test set of English-German, increasing the average constraint count per sentence from 1.1∼1.7 to 6.1 and the length per target constraint from 1.1∼1.2words to 3.4 words.Then we find that PH and CS methods degrade as the number of constraints increases, but they have complementary strengths.Specifically, PH is better at retaining high constraint accuracy but lower translation quality as measured by BLEU and COMET scores.In contrast, CS has the opposite results.Based on these observations, we propose a simple but effective method combining the advantages of PH and CS.This approach involves training a model like PH to predict the term labels, and then during inference replacing those labels with target terminology text like CS, so that the subsequent generation is aware of the target term content.Extensive experimental results show that this approach can achieve high constraint accuracy and translation quality simultaneously, regardless of the number or length of constraints.
Harmful Language Datasets: An Assessment of Robustness

The automated detection of harmful language has been of great importance for the online world, especially with the growing importance of social media and, consequently, polarisation.There are many open challenges to high quality detection of harmful text, from dataset creation to generalisable application, thus calling for more systematic studies.In this paper, we explore re-annotation as a means of examining the robustness of already existing labelled datasets, showing that, despite using alternative definitions, the inter-annotator agreement remains very inconsistent, highlighting the intrinsically subjective and variable nature of the task.In addition, we build automatic toxicity detectors using the existing datasets, with their original labels, and we evaluate them on our multi-definition and multi-source datasets.Surprisingly, while other studies show that hate speech detection models perform better on data that are derived from the same distribution as the training set, our analysis demonstrates this is not necessarily true. A Annotation instructions and interface
Enhancing Personalized Dialogue Generation with Contrastive Latent Variables: Combining Sparse and Dense Persona

The personalized dialogue explores the consistent relationship between dialogue generation and personality.Existing personalized dialogue agents model persona profiles from three resources: sparse or dense persona descriptions and dialogue histories.However, sparse structured persona attributes are explicit but uninformative, dense persona texts contain rich persona descriptions with much noise, and dialogue history query is both noisy and uninformative for persona modeling.In this work, we combine the advantages of the three resources to obtain a richer and more accurate persona.We design a Contrastive Latent Variable-based model (CLV) that clusters the dense persona descriptions into sparse categories, which are combined with the history query to generate personalized responses.Experimental results on Chinese and English datasets demonstrate our model's superiority in personalization.
Narrative Cloze as a Training Objective: Towards Modeling Stories Using Narrative Chain Embeddings

We present a novel approach to modeling narratives using narrative chain embeddings.A new dataset of narrative chains extracted from German news texts is presented.With neural methods, we produce models for both German and English that achieve state-of-the-art performance on the Multiple Choice Narrative Cloze task.Subsequently, we perform an extrinsic evaluation of the embeddings our models produce and show that they perform rather poorly in identifying narratively similar texts.We explore some of the reasons for this underperformance and discuss the upsides of our approach.We provide an outlook on alternative ways to model narratives, as well as techniques for evaluating such models.
Lifting the Curse of Capacity Gap in Distilling Language Models

Pretrained language models (LMs) have shown compelling performance on various downstream tasks, but unfortunately they require a tremendous amount of inference compute.Knowledge distillation finds a path to compress LMs to small ones with a teacher-student paradigm.However, when the capacity gap between the teacher and the student is large, a curse of capacity gap appears, invoking a deficiency in distilling LMs.While a few studies have been carried out to fill the gap, the curse is not yet well tackled.In this paper, we aim at lifting the curse of capacity gap via enlarging the capacity of the student without notably increasing the inference compute.Largely motivated by sparse activation regime of mixture of experts (MOE), we propose a mixture of minimal experts (MINIMOE), which imposes extra parameters to the student but introduces almost no additional inference compute.Experimental results on GLUE and CoNLL demonstrate the curse of capacity gap is lifted by the magic of MINIMOE to a large extent.MINIMOE also achieves the state-of-the-art performance at small FLOPs compared with a range of competitive baselines.With a compression rate as much as ∼50×, MINIMOE preserves ∼95% GLUE score of the teacher.