Reviewerly/exHarmony huggingface dataset:
Exploring the Impact of Model Scaling on Parameter-Efficient Tuning

Yusheng Su, Chi-Min Chan, Jiali Cheng, Yujia Qin, Yankai Lin, Shengding Hu, Zonghan Yang, Ning Ding, Xingzhi Sun, Guotong Xie, Zhiyuan Liu, Maosong Sun. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023.
Harnessing Dataset Cartography for Improved Compositional Generalization in Transformers

Neural networks have revolutionized language modeling and excelled in various downstream tasks. However, the extent to which these models achieve compositional generalization comparable to human cognitive abilities remains a topic of debate. While existing approaches in the field have mainly focused on novel architectures and alternative learning paradigms, we introduce a pioneering method harnessing the power of dataset cartography (Swayamdipta et al., 2020). By strategically identifying a subset of compositional generalization data using this approach, we achieve a remarkable improvement in model accuracy, yielding enhancements of up to 10% on CFQ and COGS datasets. Notably, our technique incorporates dataset cartography as a curriculum learning criterion, eliminating the need for hyperparameter tuning while consistently achieving superior performance. Our findings highlight the untapped potential of dataset cartography in unleashing the full capabilities of compositional generalization within Transformer models.
LEAF: Linguistically Enhanced Event Temporal Relation Framework

Linguistic structures can implicitly imply diverse types of event relations that have been previously underexplored. For example, the sentence “John was cooking freshly made noodles for the family gathering” contains no explicit temporal indicators between the events, such as before. Despite this, it is easy for humans to conclude, based on syntax, that the noodles were made before John started cooking, and that the family gathering starts after John starts cooking. We introduce Linguistically enhanced Event TemporAl relation Framework (LEAF), a simple and effective approach to acquiring rich temporal knowledge of events from large-scale corpora. This method improves pre-trained language models by automatically extracting temporal relation knowledge from unannotated corpora using diverse temporal knowledge patterns. We begin by manually curating a comprehensive list of atomic patterns that imply temporal relations between events. These patterns involve event pairs in which one event is contained within the argument of the other. Using transitivity, we discover compositional patterns and assign labels to event pairs involving these patterns. Finally, we make language models learn the rich knowledge by pre-training with the acquired temporal relation supervision. Experiments show that our method outperforms or rivals previous models on two event relation datasets: MATRES and TB-Dense. Our approach is also simpler from past works and excels at identifying complex compositional event relations.
Learning Retrieval Augmentation for Personalized Dialogue Generation

Personalized dialogue generation, focusing on generating highly tailored responses by leveraging persona profiles and dialogue context, has gained significant attention in conversational AI applications. However, persona profiles, a prevalent setting in current personalized dialogue datasets, typically composed of merely four to five sentences, may not offer comprehensive descriptions of the persona about the agent, posing a challenge to generate truly personalized dialogues. To handle this problem, we propose Learning Retrieval Augmentation for Personalized DialOgue Generation (LAPDOG), which studies the potential of leveraging external knowledge for persona dialogue generation. Specifically, the proposed LAPDOG model consists of a story retriever and a dialogue generator. The story retriever uses a given persona profile as queries to retrieve relevant information from the story document, which serves as a supplementary context to augment the persona profile. The dialogue generator utilizes both the dialogue history and the augmented persona profile to generate personalized responses. For optimization, we adopt a joint training framework that collaboratively learns the story retriever and dialogue generator, where the story retriever is optimized towards desired ultimate metrics (e.g., BLEU) to retrieve content for the dialogue generator to generate personalized responses. Experiments conducted on the CONVAI2 dataset with ROCStory as a supplementary data source show that the proposed LAPDOG method substantially outperforms the baselines, indicating the effectiveness of the proposed method. The LAPDOG model code is publicly available for further exploration.
X-SNS: Cross-Lingual Transfer Prediction through Sub-Network Similarity

Cross-lingual transfer (XLT) is an emergent ability of multilingual language models that preserves their performance on a task to a significant extent when evaluated in languages that were not included in the fine-tuning process. While English, due to its widespread usage, is typically regarded as the primary language for model adaption in various tasks, recent studies have revealed that the efficacy of XLT can be amplified by selecting the most appropriate source languages based on specific conditions. In this work, we propose the utilization of sub-network similarity between two languages as a proxy for predicting the compatibility of the languages in the context of XLT. Our approach is model-oriented, better reflecting the inner workings of foundation models. In addition, it requires only a moderate amount of raw text from candidate languages, distinguishing it from the majority of previous methods that rely on external resources. In experiments, we demonstrate that our method is more effective than baselines across diverse tasks. Specifically, it shows proficiency in ranking candidates for zero-shot XLT, achieving an improvement of 4.6% on average in terms of NDCG@3. We also provide extensive analyses that confirm the utility of sub-networks for XLT prediction.
Enhancing Scalability of Pre-trained Language Models via Efficient Parameter Sharing

In this paper, we propose a highly parameter-efficient approach to scaling pre-trained language models (PLMs) to a deeper model depth. Unlike prior work that shares all parameters or uses extra blocks, we design a more capable parameter-sharing architecture based on matrix product operator (MPO), an efficient tensor decomposition method to factorize the parameter matrix into a set of local tensors. Based on such a decomposition, we share the important local tensor across all layers for reducing the model size and meanwhile keep layer-specific tensors (also using Adapters) for enhancing the adaptation flexibility. To improve the model training, we further propose a stable initialization algorithm tailored for the MPO-based architecture. Extensive experiments have demonstrated the effectiveness of our proposed model in enhancing scalability and achieving higher performance (i.e., with fewer parameters than BERT-base, we successfully scale the model depth by a factor of 4x and even achieve 0.1 points higher than BERT-large for GLUE score). The code to reproduce the results of this paper can be found at https://github.com/RUCAIBox/MPOBERT-code.
Clinical Risk Prediction Models with Meta-Learning Prototypes of Patient Heterogeneity

Hospitalized patients sometimes have complex health conditions, such as multiple diseases, underlying diseases, and complications. The heterogeneous patient conditions may have various representations. A generalized model ignores the differences among heterogeneous patients, and personalized models, even with transfer learning, are still limited to the small amount of training data and the repeated training process. Meta-learning provides a solution for training similar patients based on few-shot learning; however, cannot address common cross-domain patients. Inspired by prototypical networks [1], we proposed a meta-prototype for Electronic Health Records (EHR), a meta-learning-based model with flexible prototypes representing the heterogeneity in patients. We apply this technique to cardiovascular diseases in MIMIC-III and compare it against a set of benchmark models, and demonstrate its ability to address heterogeneous patient health conditions and improve the model performances from 1.2% to 11.9% on different metrics and prediction tasks.Clinical relevance— Developing an adaptive EHR risk prediction model for outcomes-driven phenotyping of heterogeneous patient health conditions.
Rerender A Video: Zero-Shot Text-Guided Video-to-Video Translation

Large text-to-image diffusion models have exhibited impressive proficiency in generating high-quality images. However, when applying these models to video domain, ensuring temporal consistency across video frames remains a formidable challenge. This paper proposes a novel zero-shot text-guided video-to-video translation framework to adapt image models to videos. The framework includes two parts: key frame translation and full video translation. The first part uses an adapted diffusion model to generate key frames, with hierarchical cross-frame constraints applied to enforce coherence in shapes, textures and colors. The second part propagates the key frames to other frames with temporal-aware patch matching and frame blending. Our framework achieves global style and local texture temporal consistency at a low cost (without re-training or optimization). The adaptation is compatible with existing image diffusion techniques, allowing our framework to take advantage of them, such as customizing a specific subject with LoRA, and introducing extra spatial guidance with ControlNet. Extensive experimental results demonstrate the effectiveness of our proposed framework over existing methods in rendering high-quality and temporally-coherent videos. Code is available at our project page: https://www.mmlab-ntu.com/project/rerender/
Example-Based Sampling with Diffusion Models

Much effort has been put into developing samplers with specific properties, such as producing blue noise, low-discrepancy, lattice or Poisson disk samples. These samplers can be slow if they rely on optimization processes, may rely on a wide range of numerical methods, are not always differentiable. The success of recent diffusion models for image generation suggests that these models could be appropriate for learning how to generate point sets from examples. However, their convolutional nature makes these methods impractical for dealing with scattered data such as point sets. We propose a generic way to produce 2-d point sets imitating existing samplers from observed point sets using a diffusion model. We address the problem of convolutional layers by leveraging neighborhood information from an optimal transport matching to a uniform grid, that allows us to benefit from fast convolutions on grids, and to support the example-based learning of non-uniform sampling patterns. We demonstrate how the differentiability of our approach can be used to optimize point sets to enforce properties.
Control of chaotic systems through reservoir computing

Chaos is an important dynamic feature, which generally occurs in deterministic and stochastic nonlinear systems and is an inherent characteristic that is ubiquitous. Many difficulties have been solved and new research perspectives have been provided in many fields. The control of chaos is another problem that has been studied. In recent years, a recurrent neural network has emerged, which is widely used to solve many problems in nonlinear dynamics and has fast and accurate computational speed. In this paper, we employ reservoir computing to control chaos in dynamic systems. The results show that the reservoir calculation algorithm with a control term can control the chaotic phenomenon in a dynamic system. Meanwhile, the method is applicable to dynamic systems with random noise. In addition, we investigate the problem of different values for neurons and leakage rates in the algorithm. The findings indicate that the performance of machine learning techniques can be improved by appropriately constructing neural networks.
Overcoming Data Scarcity in Human Activity Recognition<sup>*</sup>

Wearable sensors have become increasingly popular in recent years, with technological advances leading to cheaper, more widely available, and smaller devices. As a result, there has been a growing interest in applying machine learning techniques for Human Activity Recognition (HAR) in healthcare. These techniques can improve patient care and treatment by accurately detecting and analyzing various activities and behaviors. However, current approaches often require large amounts of labeled data, which can be difficult and time-consuming to obtain. In this study, we propose a new approach that uses synthetic sensor data generated by 3D engines and Generative Adversarial Networks to overcome this obstacle. We evaluate the synthetic data using several methods and compare them to real-world data, including classification results with baseline models. Our results show that synthetic data can improve the performance of deep neural networks, achieving a better F <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</inf> -score for less complex activities on a known dataset by 8.4% to 73% than state-of-the-art results. However, as we showed in a self-recorded nursing activity dataset of longer duration, this effect diminishes with more complex activities. This research highlights the potential of synthetic sensor data generated from multiple sources to overcome data scarcity in HAR.
Narrow-band loss — a novel loss function focused on target boundary

Loss functions widely employed in medical image segmentation, e.g., Dice or Generalized Dice, treat each pixel of segmentation target(s) equally. These region-based loss functions are concerned with the overall segmentation accuracy. However, in clinical applications, the focus of attention is often the boundary area of the target organ(s). Existing region-based loss functions lack attention to boundary areas. We designed narrow-band loss, which computes the integration of the predicted probability within the narrow-band around the target boundary. From the aspect of how it's derived, Narrow-band loss belongs to the region-based loss function. The difference from normal region-based loss is that Narrow-band loss calculates based on the degree of coincidence of the region surrounding the organ boundary. The advantage is that narrow-band loss can guide networks to focus on the target's boundary and neighborhood. We also generalize narrow-band loss to multi-target segmentation. We tested narrow-band loss on two datasets of different parts of the human body: the brain dataset with 416 cases, each case with 35 labels, and the abdominal dataset with 50 cases, each case with 12 labels. Narrow-band loss has improved greatly in hd95 metric and dice metric compared with baseline, which is dice loss only.
From Human Hand to Grasp Surface Detection, Tracking &amp; Analysis

Hands are paramount for dexterous interactions that humans exhibit in daily life. Understanding the intricacies of human hand-object interactions is therefore necessary. Unfortunately, the limitations of state-of-the-art technologies make capturing the full hand-object complexity unfeasible, giving rise to the need for new technological means to achieve this aim. In this work, we propose an end-to-end framework in which individualized hand models are derived and used to capture quantitative personalized hand-object interaction information, precisely, hand shape, kinematics, and contact surfaces. The results of this study serve as a proof of concept that such a framework can significantly deepen personalized hand-object interaction analyses, providing, in perspective, insights for medical diagnoses and rehabilitation, among others.Clinical relevance— Our work showcases the need to incorporate bespoke human hand models in individualized hand function assessment technologies, as hand-object interaction information is subject-dependent.
A Precise Hip Protection System with Multi-scale Fall Warning Algorithm Based on Offset Displacement

Falls occur frequently in daily life and the damage to the body is irreversible. Therefore, it is crucial to implement timely and effective warning and protection systems for falls to minimize the damage caused by falls. Currently, the fall warning algorithm has shortcomings such as low recognition rates for falls and fall-risk movements and insufficient lead-time, the time before the subject impacts the floor, making it difficult for falling protection devices to function effectively. In this study, a multi-scale falls warning algorithm based on offset displacement is built, and a hip protection system is designed. The performance of the algorithm and the system is validated using 150 falling and 500 fall-risk actions from 10 volunteers. The results showed that the recognition accuracy for falling actions is 98.7% and the recognition accuracy for fall-risk actions is 99.4%, with an average lead-time of 402ms. The protection rate for falling movements reached 98.7%. This proposed algorithm and hip protection system have the potential to be applied in elderly communities, hospitals, and homes to reduce the damage caused by falls.Clinical Relevance— This study provides important reference for clinicians in analyzing fall behaviors to patients at risk of falls in clinical settings, offering valuable technical support for ensuring the safety of patients in danger of falling. It also contributes to further promoting the development of falling-prevention medical devices.
Artifact Augmentation for Learning-based Quality Control of Whole Slide Images

The acquisition of whole slide images is prone to artifacts that can require human control and re-scanning, both in clinical workflows and in research-oriented settings. Quality control algorithms are a first step to overcome this challenge, as they limit the use of low quality images. Developing quality control systems in histopathology is not straightforward, also due to the limited availability of data related to this topic. We address the problem by proposing a tool to augment data with artifacts. The proposed method seamlessly generates and blends artifacts from an external library to a given histopathology dataset. The datasets augmented by the blended artifacts are then used to train an artifact detection network in a supervised way. We use the YOLOv5 model for the artifact detection with a slightly modified training pipeline. The proposed tool can be extended into a complete framework for the quality assessment of whole slide images.Clinical relevance- The proposed method may be useful for the initial quality screening of whole slide images. Each year, millions of whole slide images are acquired and digitized worldwide. Numerous of them contain artifacts affecting the following AI-oriented analysis. Therefore, a tool operating at the acquisition phase and improving the initial quality assessment is crucial to increase the performance of digital pathology algorithms, e.g., early cancer diagnosis.
Comparing the Graphical Features of Simple Artificial Neural Networks and Cortical Development

Brain development is characterized by changes in connections and information processing complexity. These changes inspire the training process of artificial neural network (ANN), which requires adjusting the neuron weights and biases to enhance efficiency in performing a specific task. In this work, we found affinities in the ratio of positive and negative weights in simple ANNs during training with that of excitatory and inhibitory synapses in the cortex. Additionally, we present a graphical representation of simple ANNs formed by pruning unimportant weights and aligning neurons and connections of different layers. Our findings suggest a strong relationship between the accuracy of simple neural network and graphical representation features, with graphical features at the inflection point resembling the graphical representation of the cortex.
Development of a low-cost and user-friendly system to create personalized human digital twin

Creating a digital twin has enormous potential in biomedical engineering. However, it is also technically challenging. No existing system can allow people who don't have the art-and-design background to create their own digital twin. To fill this gap, this study proposes a low-cost wearable system and a user-friendly framework for creating personalized digital twins with a fast speed and high fidelity. The personalized human digital twin can capture synchronized facial expressions, gaze direction, and whole-body movements for real-time rendering. The system simplifies the complex process of creating digital humans, and allows for the creation of data-driven characters without specialized skills.Clinical Relevance- This system can be used to help doctors keep track record of the patient's health status in a more visual and realistic way, supporting them in making more accurate clinical decisions, and facilitating a more detailed medical intervention.
Domain-Adversarial Transformer Network for Multiphase Liver Tumor Segmentation

Accurate liver tumor segmentation is a prerequisite for data-driven tumor analysis. Multiphase computed tomography (CT) with extensive liver tumor characteristics is typically used as the most crucial diagnostic basis. However, the large variations in contrast, texture, and tumor structure between CT phases limit the generalization capabilities of the associated segmentation algorithms. Inadequate feature integration across phases might also lead to a performance decrease. To address these issues, we present a domain-adversarial transformer (DA-Tran) network for segmenting liver tumors from multiphase CT images. A DA module is designed to generate domain-adapted feature maps from the non-contrast-enhanced (NC) phase, arterial (ART) phase, portal venous (PV) phase, and delay phase (DP) images. These domain-adapted feature maps are then combined with 3D transformer blocks to capture patch-structured similarity and global context attention. The experimental findings show that DA-Tran produces cutting-edge tumor segmentation outcomes, making it an ideal candidate for this co-segmentation challenge.