OpenRaiser/Intern-Atlas huggingface dataset:
EVEDIT: Event-based Knowledge Editing with Deductive Editing Boundaries
Venue: arXiv cs.CL 2024
Year: 2024

The dynamic nature of real-world information necessitates efficient knowledge editing (KE) in large language models (LLMs) for knowledge updating. However, current KE approaches, which typically operate on (subject, relation, object) triples, ignore the contextual information and the relation among different knowledge. Such editing methods could thus encounter an uncertain editing boundary, leaving a lot of relevant knowledge in ambiguity: Queries that could be answered pre-edit cannot be reliably answered afterward. In this work, we analyze this issue by introducing a theoretical framework for KE that highlights an overlooked set of knowledge that remains unchanged and aids in knowledge deduction during editing, which we name as the deduction anchor. We further address this issue by proposing a novel task of event-based knowledge editing that pairs facts with event descriptions. This task manifests not only a closer simulation of real-world editing scenarios but also a more logically sound setting, implicitly defining the deduction anchor to address the issue of indeterminate editing boundaries. We empirically demonstrate the superiority of event-based editing over the existing setting on resolving uncertainty in edited models, and curate a new benchmark dataset EvEdit derived from the CounterFact dataset. Moreover, while we observe that the event-based setting is significantly challenging for existing approaches, we propose a novel approach Self-Edit that showcases stronger performance, achieving 55.6% consistency improvement while maintaining the naturalness of generation.
Wireless Distributed Matrix-Vector Multiplication using Over-the-Air Computation and Analog Coding
Venue: arXiv cs.IT 2024
Year: 2024

In this paper, we propose an over-the-air (OTA)-based approach for distributed matrix-vector multiplications in the context of distributed machine learning (DML). Thanks to OTA computation, the column-wise partitioning of a large matrix enables efficient workload distribution among workers (i.e., local computing nodes) based on their computing capabilities. In addition, without requiring additional bandwidth, it allows the system to remain scalable even as the number of workers increases to mitigate the impact of slow workers, known as stragglers. However, despite the improvements, there are still instances where some workers experience deep fading and become stragglers, preventing them from transmitting their results. By analyzing the mean squared error (MSE), we demonstrate that incorporating more workers in the OTA-based approach leads to MSE reduction without the need for additional radio resources. Furthermore, we introduce an analog coding scheme to further enhance the performance and compare it with conventional coded multiplication (CM) schemes. Through simulations, it is shown that the OTA-based approach achieves comparable performance to CM schemes while potentially requiring fewer radio resources.
Reasoning before Comparison: LLM-Enhanced Semantic Similarity Metrics for Domain Specialized Text Analysis
Venue: arXiv cs.CL 2024
Year: 2024

In this study, we leverage LLM to enhance the semantic analysis and develop similarity metrics for texts, addressing the limitations of traditional unsupervised NLP metrics like ROUGE and BLEU. We develop a framework where LLMs such as GPT-4 are employed for zero-shot text identification and label generation for radiology reports, where the labels are then used as measurements for text similarity. By testing the proposed framework on the MIMIC data, we find that GPT-4 generated labels can significantly improve the semantic similarity assessment, with scores more closely aligned with clinical ground truth than traditional NLP metrics. Our work demonstrates the possibility of conducting semantic analysis of the text data using semi-quantitative reasoning results by the LLMs for highly specialized domains. While the framework is implemented for radiology report similarity analysis, its concept can be extended to other specialized domains as well.
Evaluating the Stability of Deep Learning Latent Feature Spaces
Venue: arXiv cs.LG 2024
Year: 2024

High-dimensional datasets present substantial challenges in statistical modeling across various disciplines, necessitating effective dimensionality reduction methods. Deep learning approaches, notable for their capacity to distill essential features from complex data, facilitate modeling, visualization, and compression through reduced dimensionality latent feature spaces, have wide applications from bioinformatics to earth sciences. This study introduces a novel workflow to evaluate the stability of these latent spaces, ensuring consistency and reliability in subsequent analyses. Stability, defined as the invariance of latent spaces to minor data, training realizations, and parameter perturbations, is crucial yet often overlooked. Our proposed methodology delineates three stability types, sample, structural, and inferential, within latent spaces, and introduces a suite of metrics for comprehensive evaluation. We implement this workflow across 500 autoencoder realizations and three datasets, encompassing both synthetic and real-world scenarios to explain latent space dynamics. Employing k-means clustering and the modified Jonker-Volgenant algorithm for class alignment, alongside anisotropy metrics and convex hull analysis, we introduce adjusted stress and Jaccard dissimilarity as novel stability indicators. Our findings highlight inherent instabilities in latent feature spaces and demonstrate the workflow's efficacy in quantifying and interpreting these instabilities. This work advances the understanding of latent feature spaces, promoting improved model interpretability and quality control for more informed decision-making for diverse analytical workflows that leverage deep learning.
Multi-dimensional Evaluation of Empathetic Dialog Responses
Venue: arXiv cs.CL 2024
Year: 2024

Empathy is critical for effective and satisfactory conversational communication. Prior efforts to measure conversational empathy mostly focus on expressed communicative intents -- that is, the way empathy is expressed. Yet, these works ignore the fact that conversation is also a collaboration involving both speakers and listeners. In contrast, we propose a multi-dimensional empathy evaluation framework to measure both \emph{expressed intents from the speaker's perspective} and \emph{perceived empathy from the listener's perspective}. We apply our analytical framework to examine internal customer-service dialogues. We find the two dimensions (expressed intent types and perceived empathy) are inter-connected, while perceived empathy has high correlations with dialogue satisfaction levels. To reduce the annotation cost, we explore different options to automatically measure conversational empathy: prompting LLMs and training language model-based classifiers. Our experiments show that prompting methods with even popular models like GPT-4 and Flan family models perform relatively poorly on both public and our internal datasets. In contrast, instruction-finetuned classifiers based on Flan-T5 family models outperform prior works and competitive baselines. We conduct a detailed ablation study to give more insights into instruction finetuning method's strong performance.
Analysis of Fatigue-Induced Compensatory Movements in Bicep Curls: Gaining Insights for the Deployment of Wearable Sensors
Venue: arXiv cs.RO 2024
Year: 2024

A common challenge in Bicep Curls rehabilitation is muscle compensation, where patients adopt alternative movement patterns when the primary muscle group cannot act due to injury or fatigue, significantly decreasing the effectiveness of rehabilitation efforts. The problem is exacerbated by the growing trend toward transitioning from in-clinic to home-based rehabilitation, where constant monitoring and correction by physiotherapists are limited. Developing wearable sensors capable of detecting muscle compensation becomes crucial to address this challenge. This study aims to gain insights into the optimal deployment of wearable sensors through a comprehensive study of muscle compensation in Bicep Curls. We collect upper limb joint kinematics and surface electromyography signals (sEMG) from eight muscles in 12 healthy subjects during standard and fatigue stages. Two muscle synergies are derived from sEMG signals and are analyzed comprehensively along with joint kinematics. Our findings reveal a shift in the relative contribution of forearm muscles to shoulder muscles, accompanied by a significant increase in activation amplitude for both synergies. Additionally, more pronounced movement was observed at the shoulder joint during fatigue. These results suggest focusing on the shoulder muscle activities and joint motions when deploying wearable sensors to effectively detect compensatory movements.
Can Deception Detection Go Deeper? Dataset, Evaluation, and Benchmark for Deception Reasoning
Venue: arXiv cs.CL 2024
Year: 2024

Deception detection has attracted increasing attention due to its importance in real-world scenarios. Its main goal is to detect deceptive behaviors from multimodal clues such as gestures, facial expressions, prosody, etc. However, these bases are usually subjective and related to personal habits. Therefore, we extend deception detection to deception reasoning, further providing objective evidence to support subjective judgment. Specifically, we provide potential lies and basic facts and then analyze why this sentence may be a lie by combining factual inconsistencies and intent behind them. Compared with deception detection, this task is more applicable to real-world scenarios. For example, in interrogation, the police should judge whether a person is lying based on solid evidence. This paper presents our initial attempts at this task, including constructing a dataset and defining evaluation metrics. Meanwhile, this task can serve as a benchmark for evaluating the complex reasoning capability of large language models. Our code and data are provided in the supplementary material.
The Road to Trust: Building Enclaves within Confidential VMs
Venue: arXiv cs.CR 2024
Year: 2024

Integrity is critical for maintaining system security, as it ensures that only genuine software is loaded onto a machine. Although confidential virtual machines (CVMs) function within isolated environments separate from the host, it is important to recognize that users still encounter challenges in maintaining control over the integrity of the code running within the trusted execution environments (TEEs). The presence of a sophisticated operating system (OS) raises the possibility of dynamically creating and executing any code, making user applications within TEEs vulnerable to interference or tampering if the guest OS is compromised. To address this issue, this paper introduces NestedSGX, a framework which leverages virtual machine privilege level (VMPL), a recent hardware feature available on AMD SEV-SNP to enable the creation of hardware enclaves within the guest VM. Similar to Intel SGX, NestedSGX considers the guest OS untrusted for loading potentially malicious code. It ensures that only trusted and measured code executed within the enclave can be remotely attested. To seamlessly protect existing applications, NestedSGX aims for compatibility with Intel SGX by simulating SGX leaf functions. We have also ported the SGX SDK and the Occlum library OS to NestedSGX, enabling the use of existing SGX toolchains and applications in the system. Performance evaluations show that context switches in NestedSGX take about 32,000 -- 34,000 cycles, approximately $1.9\times$ -- $2.1\times$ higher than that of Intel SGX. NestedSGX incurs minimal overhead in most real-world applications, with an average overhead below 2% for computation and memory intensive workloads and below 15.68% for I/O intensive workloads.
Thyroid ultrasound diagnosis improvement via multi-view self-supervised learning and two-stage pre-training
Venue: arXiv cs.CV 2024
Year: 2024

Thyroid nodule classification and segmentation in ultrasound images are crucial for computer-aided diagnosis; however, they face limitations owing to insufficient labeled data. In this study, we proposed a multi-view contrastive self-supervised method to improve thyroid nodule classification and segmentation performance with limited manual labels. Our method aligns the transverse and longitudinal views of the same nodule, thereby enabling the model to focus more on the nodule area. We designed an adaptive loss function that eliminates the limitations of the paired data. Additionally, we adopted a two-stage pre-training to exploit the pre-training on ImageNet and thyroid ultrasound images. Extensive experiments were conducted on a large-scale dataset collected from multiple centers. The results showed that the proposed method significantly improves nodule classification and segmentation performance with limited manual labels and outperforms state-of-the-art self-supervised methods. The two-stage pre-training also significantly exceeded ImageNet pre-training.
Advancing Translation Preference Modeling with RLHF: A Step Towards Cost-Effective Solution
Venue: arXiv cs.CL 2024
Year: 2024

Faithfulness, expressiveness, and elegance is the constant pursuit in machine translation. However, traditional metrics like \textit{BLEU} do not strictly align with human preference of translation quality. In this paper, we explore leveraging reinforcement learning with human feedback (\textit{RLHF}) to improve translation quality. It is non-trivial to collect a large high-quality dataset of human comparisons between translations, especially for low-resource languages. To address this issue, we propose a cost-effective preference learning strategy, optimizing reward models by distinguishing between human and machine translations. In this manner, the reward model learns the deficiencies of machine translation compared to human and guides subsequent improvements in machine translation. Experimental results demonstrate that \textit{RLHF} can effectively enhance translation quality and this improvement benefits other translation directions not trained with \textit{RLHF}. Further analysis indicates that the model's language capabilities play a crucial role in preference learning. A reward model with strong language capabilities can more sensitively learn the subtle differences in translation quality and align better with real human translation preferences.
Randomness-Efficient Constructions of Capacity-Achieving List-Decodable Codes
Venue: arXiv cs.IT 2024
Year: 2024

We wish to generate list-decodable codes over small alphabets using as little randomness as possible. Specifically, we hope to generate codes achieving what we term the Elias bound, which means that they are $(ρ,L)$-list-decodable with rate $R \geq 1-h(ρ)-O(1/L)$. A long line of work shows that uniformly random linear codes (RLCs) achieve the Elias bound: hence, we know $O(n^2)$ random bits suffice. Prior works demonstrate that just $O(Ln)$ random bits suffice, via puncturing of low-bias codes. These recent constructions are combinatorial. We provide two new constructions, which are algebraic. Compared to prior works, our constructions are simpler and more direct. Furthermore, our codes are designed in such a way that their duals are also quite easy to analyze. Our first construction -- which can be seen as a generalization of the Wozencraft ensemble -- achieves the Elias bound and consumes $Ln$ random bits. Additionally, its dual code achieves the GV-bound with high probability, and both the primal and dual admit quasilinear-time encoding algorithms. The second construction consumes $2nL$ random bits and yields a code where both it and its dual achieve the Elias bound. As we discuss, properties of a dual code are often crucial for applications in cryptography. In all of the above cases -- including the prior works achieving randomness complexity $O(Ln)$ -- the codes are designed to "approximate" RLCs. Namely, for a given locality parameter $L$ we construct codes achieving the same $L$-local properties as RLCs. This allows one to appeal to known list-decodability results for RLCs and thereby conclude that the code approximating an RLC also achieves the Elias bound. As a final contribution, we indicate that such a proof strategy is inherently unable to generate list-decodable codes of rate $R$ over $\mathbb F_q$ with less than $L(1-R)n\log_2(q)$ bits of randomness.
LongAgent: Scaling Language Models to 128k Context through Multi-Agent Collaboration
Venue: arXiv cs.CL 2024
Year: 2024

Large language models (LLMs) have demonstrated impressive performance in understanding language and executing complex reasoning tasks. However, LLMs with long context windows have been notorious for their expensive training costs and high inference latency. Even the most advanced models such as GPT-4 and Claude2 often make mistakes when processing inputs of over $100k$ tokens, a phenomenon also known as \textit{lost in the middle}. In this paper, we propose \textsc{LongAgent}, a method based on multi-agent collaboration, which scales LLMs (e.g., LLaMA) to a context of 128K and demonstrates potential superiority in long-text processing compared to GPT-4. In \textsc{LongAgent}, a leader is responsible for understanding user intent and directing team members to acquire information from documents. Due to members' hallucinations, it is non-trivial for a leader to obtain accurate information from the responses of dozens to hundreds of members. To address this, we develop an \textit{inter-member communication} mechanism to resolve response conflicts caused by hallucinations through information sharing. Our experimental results indicate that \textsc{LongAgent} offers a promising alternative for long-text processing. The agent team instantiated with LLaMA-7B achieves significant improvements in tasks such as 128k-long text retrieval, multi-hop question answering, compared to GPT-4.
Imitation Learning-Based Online Time-Optimal Control with Multiple-Waypoint Constraints for Quadrotors
Venue: arXiv cs.RO 2024
Year: 2024

Over the past decade, there has been a remarkable surge in utilizing quadrotors for various purposes due to their simple structure and aggressive maneuverability, such as search and rescue, delivery and autonomous drone racing, etc. One of the key challenges preventing quadrotors from being widely used in these scenarios is online waypoint-constrained time-optimal trajectory generation and control technique. This letter proposes an imitation learning-based online solution to efficiently navigate the quadrotor through multiple waypoints with time-optimal performance. The neural networks (WN&CNets) are trained to learn the control law from the dataset generated by the time-consuming CPC algorithm and then deployed to generate the optimal control commands online to guide the quadrotors. To address the challenge of limited training data and the hover maneuver at the final waypoint, we propose a transition phase strategy that utilizes MINCO trajectories to help the quadrotor 'jump over' the stop-and-go maneuver when switching waypoints. Our method is demonstrated in both simulation and real-world experiments, achieving a maximum speed of 5.6m/s while navigating through 7 waypoints in a confined space of 5.5m*5.5m*2.0m. The results show that with a slight loss in optimality, the WN&CNets significantly reduce the processing time and enable online optimal control for multiple-waypoint constrained flight tasks.
BGE Landmark Embedding: A Chunking-Free Embedding Method For Retrieval Augmented Long-Context Large Language Models
Venue: arXiv cs.CL 2024
Year: 2024

Large language models (LLMs) call for extension of context to handle many critical applications. However, the existing approaches are prone to expensive costs and inferior quality of context extension. In this work, we proposeExtensible Embedding, which realizes high-quality extension of LLM's context with strong flexibility and cost-effectiveness. Extensible embedding stand as an enhancement of typical token embedding, which represents the information for an extensible scope of context instead of a single token. By leveraging such compact input units of higher information density, the LLM can access to a vast scope of context even with a small context window. Extensible embedding is systematically optimized in architecture and training method, which leads to multiple advantages. 1) High flexibility of context extension, which flexibly supports ad-hoc extension of diverse context lengths. 2) Strong sample efficiency of training, which enables the embedding model to be learned in a cost-effective way. 3) Superior compatibility with the existing LLMs, where the extensible embedding can be seamlessly introduced as a plug-in component. Comprehensive evaluations on long-context language modeling and understanding tasks verify extensible embedding as an effective, efficient, flexible, and compatible method to extend the LLM's context.
Extensible Embedding: A Flexible Multipler For LLM's Context Length
Venue: arXiv cs.CL 2024
Year: 2024

Large language models (LLMs) call for extension of context to handle many critical applications. However, the existing approaches are prone to expensive costs and inferior quality of context extension. In this work, we propose Extensible Embedding, which realizes high-quality extension of LLM's context with strong flexibility and cost-effectiveness. Extensible embedding stand as an enhancement of typical token embedding, which represents the information for an extensible scope of context instead of a single token. By leveraging such compact input units of higher information density, the LLM can access to a vast scope of context even with a small context window. Extensible embedding is systematically optimized in architecture and training method, which leads to multiple advantages. 1) High flexibility of context extension, which flexibly supports ad-hoc extension of diverse context lengths. 2) Strong sample efficiency of training, which enables the embedding model to be learned in a cost-effective way. 3) Superior compatibility with the existing LLMs, where the extensible embedding can be seamlessly introduced as a plug-in component. Comprehensive evaluations on long-context language modeling and understanding tasks verify extensible embedding as an effective, efficient, flexible, and compatible method to extend the LLM's context.
SDiT: Spiking Diffusion Model with Transformer
Venue: arXiv cs.CV 2024
Year: 2024

Spiking neural networks (SNNs) have low power consumption and bio-interpretable characteristics, and are considered to have tremendous potential for energy-efficient computing. However, the exploration of SNNs on image generation tasks remains very limited, and a unified and effective structure for SNN-based generative models has yet to be proposed. In this paper, we explore a novel diffusion model architecture within spiking neural networks. We utilize transformer to replace the commonly used U-net structure in mainstream diffusion models. It can generate higher quality images with relatively lower computational cost and shorter sampling time. It aims to provide an empirical baseline for research of generative models based on SNNs. Experiments on MNIST, Fashion-MNIST, and CIFAR-10 datasets demonstrate that our work is highly competitive compared to existing SNN generative models.