OpenRaiser/Intern-Atlas huggingface dataset:
The Complexity of Manipulation of k-Coalitional Games on Graphs
Venue: arXiv cs.GT 2024
Year: 2024

In many settings, there is an organizer who would like to divide a set of agents into $k$ coalitions, and cares about the friendships within each coalition. Specifically, the organizer might want to maximize utilitarian social welfare, maximize egalitarian social welfare, or simply guarantee that every agent will have at least one friend within his coalition. However, in many situations, the organizer is not familiar with the friendship connections, and he needs to obtain them from the agents. In this setting, a manipulative agent may falsely report friendship connections in order to increase his utility. In this paper, we analyze the complexity of finding manipulation in such $k$-coalitional games on graphs. We also introduce a new type of manipulation, socially-aware manipulation, in which the manipulator would like to increase his utility without decreasing the social welfare. We then study the complexity of finding socially-aware manipulation in our setting. Finally, we examine the frequency of socially-aware manipulation and the running time of our algorithms via simulation results.
Segment Using Just One Example
Venue: arXiv cs.CV 2024
Year: 2024

Semantic segmentation is an important topic in computer vision with many relevant application in Earth observation. While supervised methods exist, the constraints of limited annotated data has encouraged development of unsupervised approaches. However, existing unsupervised methods resemble clustering and cannot be directly mapped to explicit target classes. In this paper, we deal with single shot semantic segmentation, where one example for the target class is provided, which is used to segment the target class from query/test images. Our approach exploits recently popular Segment Anything (SAM), a promptable foundation model. We specifically design several techniques to automatically generate prompts from the only example/key image in such a way that the segmentation is successfully achieved on a stitch or concatenation of the example/key and query/test images. Proposed technique does not involve any training phase and just requires one example image to grasp the concept. Furthermore, no text-based prompt is required for the proposed method. We evaluated the proposed techniques on building and car classes.
A MAC Protocol with Time Reversal for Wireless Networks within Computing Packages
Venue: arXiv cs.ET 2024
Year: 2024

Wireless Network-on-Chip (WNoC) is a promising concept which provides a solution to overcome the scalability issues in prevailing networks-in-package for many-core processors. However, the electromagnetic propagation inside the chip package leads to energy reverberation, resulting in Inter-Symbol Interference (ISI) with high delay spreads. Time Reversal (TR) is a technique that benefits the unique time-invariant channel with rich multipath effects to focus the energy to the desired transceiver. TR mitigates both ISI and co-channel interference, hence providing parallel communications in both space and time. Thus, TR is a versatile candidate to improve the aggregate bandwidth of wireless on-chip networks provided that a Medium Access Control (MAC) is used to efficiently share the wireless medium. In this paper, we explore a simple yet resilient TR-based MAC protocol (TR-MAC) design for WNoC. We propose to manage multiple parallel transmissions with simultaneous spatial channels in the same time slot with TR precoding and focused energy detection at the transceiver. Our results show that TR-MAC can be employed in massive computing architectures with improved latency and throughput while matching with the stringent requirements of the physical layer.
CMU's IWSLT 2024 Simultaneous Speech Translation System
Venue: arXiv cs.CL 2024
Year: 2024

This paper describes CMU's submission to the IWSLT 2024 Simultaneous Speech Translation (SST) task for translating English speech to German text in a streaming manner. Our end-to-end speech-to-text (ST) system integrates the WavLM speech encoder, a modality adapter, and the Llama2-7B-Base model as the decoder. We employ a two-stage training approach: initially, we align the representations of speech and text, followed by full fine-tuning. Both stages are trained on MuST-c v2 data with cross-entropy loss. We adapt our offline ST model for SST using a simple fixed hold-n policy. Experiments show that our model obtains an offline BLEU score of 31.1 and a BLEU score of 29.5 under 2 seconds latency on the MuST-C-v2 tst-COMMON.
A Stability-first Approach to Running TCP over Starlink
Venue: arXiv cs.NI 2024
Year: 2024

The end-to-end connectivity patterns between two points on Earth are highly volatile if mediated via a Low-Earth orbit (LEO) satellite constellation. This is rooted in the enormous speeds at which satellites in LEO must travel relative to the Earth's surface. While changes in end-to-end routes are rare events in stationary and terrestrial applications, they are a dominating factor for connection-oriented services running over LEO constellations and mega-constellations. This paper discusses how TCP-over-constellations is affected by the need for rerouting and how orbital route selection algorithms impact the end-to-end performance of communication. In contrast to the state of the art that primarily optimizes for instantaneous shortest routes (i.e. lowest delay), we propose several algorithms that have route stability and longevity in their focus. We show that this shift in focus comes with vastly improved end-to-end communication performance, and we discuss peculiar effects of the typical TCP-like implementations, taking inspiration from the Starlink constellation in our empirical investigations. The spectrum of algorithms proposed provides a basis for co-designing suitable orbital route selection algorithms and tailored transport control algorithms.
One Step Diffusion-based Super-Resolution with Time-Aware Distillation
Venue: arXiv cs.CV 2024
Year: 2024

Diffusion-based image super-resolution (SR) methods have shown promise in reconstructing high-resolution images with fine details from low-resolution counterparts. However, these approaches typically require tens or even hundreds of iterative samplings, resulting in significant latency. Recently, techniques have been devised to enhance the sampling efficiency of diffusion-based SR models via knowledge distillation. Nonetheless, when aligning the knowledge of student and teacher models, these solutions either solely rely on pixel-level loss constraints or neglect the fact that diffusion models prioritize varying levels of information at different time steps. To accomplish effective and efficient image super-resolution, we propose a time-aware diffusion distillation method, named TAD-SR. Specifically, we introduce a novel score distillation strategy to align the data distribution between the outputs of the student and teacher models after minor noise perturbation. This distillation strategy enables the student network to concentrate more on the high-frequency details. Furthermore, to mitigate performance limitations stemming from distillation, we integrate a latent adversarial loss and devise a time-aware discriminator that leverages diffusion priors to effectively distinguish between real images and generated images. Extensive experiments conducted on synthetic and real-world datasets demonstrate that the proposed method achieves comparable or even superior performance compared to both previous state-of-the-art (SOTA) methods and the teacher model in just one sampling step. Codes are available at https://github.com/LearningHx/TAD-SR.
QirK: Question Answering via Intermediate Representation on Knowledge Graphs
Venue: arXiv cs.DB 2024
Year: 2024

We demonstrate QirK, a system for answering natural language questions on Knowledge Graphs (KG). QirK can answer structurally complex questions that are still beyond the reach of emerging Large Language Models (LLMs). It does so using a unique combination of database technology, LLMs, and semantic search over vector embeddings. The glue for these components is an intermediate representation (IR). The input question is mapped to IR using LLMs, which is then repaired into a valid relational database query with the aid of a semantic search on vector embeddings. This allows a practical synthesis of LLM capabilities and KG reliability. A short video demonstrating QirK is available at https://youtu.be/6c81BLmOZ0U.
Image Scaling Attack Simulation: A Measure of Stealth and Detectability
Venue: arXiv cs.HC 2024
Year: 2024

Cybersecurity practices require effort to be maintained, and one weakness is a lack of awareness regarding potential attacks not only in the usage of machine learning models, but also in their development process. Previous studies have determined that preprocessing attacks, such as image scaling attacks, have been difficult to detect by humans (through visual response) and computers (through entropic algorithms). However, these studies fail to address the real-world performance and detectability of these attacks. The purpose of this work is to analyze the relationship between awareness of image scaling attacks with respect to demographic background and experience. We conduct a survey where we gather the subjects' demographics, analyze the subjects' experience in cybersecurity, record their responses to a poorly-performing convolutional neural network model that has been unknowingly hindered by an image scaling attack of a used dataset, and document their reactions after it is revealed that the images used within the broken models have been attacked. We find in this study that the overall detection rate of the attack is low enough to be viable in a workplace or academic setting, and even after discovery, subjects cannot conclusively determine benign images from attacked images.
Fast Inference for Probabilistic Answer Set Programs via the Residual Program
Venue: arXiv cs.AI 2024
Year: 2024

When we want to compute the probability of a query from a Probabilistic Answer Set Program, some parts of a program may not influence the probability of a query, but they impact on the size of the grounding. Identifying and removing them is crucial to speed up the computation. Algorithms for SLG resolution offer the possibility of returning the residual program which can be used for computing answer sets for normal programs that do have a total well-founded model. The residual program does not contain the parts of the program that do not influence the probability. In this paper, we propose to exploit the residual program for performing inference. Empirical results on graph datasets show that the approach leads to significantly faster inference.
Learning-based Models for Vulnerability Detection: An Extensive Study
Venue: arXiv cs.SE 2024
Year: 2024

Though many deep learning-based models have made great progress in vulnerability detection, we have no good understanding of these models, which limits the further advancement of model capability, understanding of the mechanism of model detection, and efficiency and safety of practical application of models. In this paper, we extensively and comprehensively investigate two types of state-of-the-art learning-based approaches (sequence-based and graph-based) by conducting experiments on a recently built large-scale dataset. We investigate seven research questions from five dimensions, namely model capabilities, model interpretation, model stability, ease of use of model, and model economy. We experimentally demonstrate the priority of sequence-based models and the limited abilities of both LLM (ChatGPT) and graph-based models. We explore the types of vulnerability that learning-based models skilled in and reveal the instability of the models though the input is subtlely semantical-equivalently changed. We empirically explain what the models have learned. We summarize the pre-processing as well as requirements for easily using the models. Finally, we initially induce the vital information for economically and safely practical usage of these models.
Multi-task Heterogeneous Graph Learning on Electronic Health Records
Venue: arXiv cs.LG 2024
Year: 2024

Learning electronic health records (EHRs) has received emerging attention because of its capability to facilitate accurate medical diagnosis. Since the EHRs contain enriched information specifying complex interactions between entities, modeling EHRs with graphs is shown to be effective in practice. The EHRs, however, present a great degree of heterogeneity, sparsity, and complexity, which hamper the performance of most of the models applied to them. Moreover, existing approaches modeling EHRs often focus on learning the representations for a single task, overlooking the multi-task nature of EHR analysis problems and resulting in limited generalizability across different tasks. In view of these limitations, we propose a novel framework for EHR modeling, namely MulT-EHR (Multi-Task EHR), which leverages a heterogeneous graph to mine the complex relations and model the heterogeneity in the EHRs. To mitigate the large degree of noise, we introduce a denoising module based on the causal inference framework to adjust for severe confounding effects and reduce noise in the EHR data. Additionally, since our model adopts a single graph neural network for simultaneous multi-task prediction, we design a multi-task learning module to leverage the inter-task knowledge to regularize the training process. Extensive empirical studies on MIMIC-III and MIMIC-IV datasets validate that the proposed method consistently outperforms the state-of-the-art designs in four popular EHR analysis tasks -- drug recommendation, and predictions of the length of stay, mortality, and readmission. Thorough ablation studies demonstrate the robustness of our method upon variations to key components and hyperparameters.
Disentangle and denoise: Tackling context misalignment for video moment retrieval
Venue: arXiv cs.CV 2024
Year: 2024

Video Moment Retrieval, which aims to locate in-context video moments according to a natural language query, is an essential task for cross-modal grounding. Existing methods focus on enhancing the cross-modal interactions between all moments and the textual description for video understanding. However, constantly interacting with all locations is unreasonable because of uneven semantic distribution across the timeline and noisy visual backgrounds. This paper proposes a cross-modal Context Denoising Network (CDNet) for accurate moment retrieval by disentangling complex correlations and denoising irrelevant dynamics.Specifically, we propose a query-guided semantic disentanglement (QSD) to decouple video moments by estimating alignment levels according to the global and fine-grained correlation. A Context-aware Dynamic Denoisement (CDD) is proposed to enhance understanding of aligned spatial-temporal details by learning a group of query-relevant offsets. Extensive experiments on public benchmarks demonstrate that the proposed CDNet achieves state-of-the-art performances.
The MAGIC of Data Management: Understanding the Value and Activities of Data Management
Venue: arXiv cs.DB 2024
Year: 2024

In an era dominated by information technology, the critical discipline of data management remains undervalued compared to the innovations it enables, such as artificial intelligence and social media. The ambiguity surrounding what constitutes data management and its associated activities complicates efforts to explain its importance and ensure data are collected, stored and used in a way that maximizes value and avoids failures. This paper aims to address these shortcomings by presenting a simple framework for understanding data management, referred to as MAGIC. MAGIC encompasses five key activities: Modeling, Acquisition, Governance, Infrastructuring, and Consumption support tasks. By delineating these components, the MAGIC framework provides a clear, accessible approach to data management that can be used for teaching, research and practice.
Boosting Unconstrained Face Recognition with Targeted Style Adversary
Venue: arXiv cs.CV 2024
Year: 2024

While deep face recognition models have demonstrated remarkable performance, they often struggle on the inputs from domains beyond their training data. Recent attempts aim to expand the training set by relying on computationally expensive and inherently challenging image-space augmentation of image generation modules. In an orthogonal direction, we present a simple yet effective method to expand the training data by interpolating between instance-level feature statistics across labeled and unlabeled sets. Our method, dubbed Targeted Style Adversary (TSA), is motivated by two observations: (i) the input domain is reflected in feature statistics, and (ii) face recognition model performance is influenced by style information. Shifting towards an unlabeled style implicitly synthesizes challenging training instances. We devise a recognizability metric to constraint our framework to preserve the inherent identity-related information of labeled instances. The efficacy of our method is demonstrated through evaluations on unconstrained benchmarks, outperforming or being on par with its competitors while offering nearly a 70\% improvement in training speed and 40\% less memory consumption.
Adaptive Behavioral AI: Reinforcement Learning to Enhance Pharmacy Services
Venue: arXiv cs.LG 2024
Year: 2024

Pharmacies are critical in healthcare systems, particularly in low- and middle-income countries. Procuring pharmacists with the right behavioral interventions or nudges can enhance their skills, public health awareness, and pharmacy inventory management, ensuring access to essential medicines that ultimately benefit their patients. We introduce a reinforcement learning operational system to deliver personalized behavioral interventions through mobile health applications. We illustrate its potential by discussing a series of initial experiments run with SwipeRx, an all-in-one app for pharmacists, including B2B e-commerce, in Indonesia. The proposed method has broader applications extending beyond pharmacy operations to optimize healthcare delivery.
See It All: Contextualized Late Aggregation for 3D Dense Captioning
Venue: arXiv cs.CV 2024
Year: 2024

3D dense captioning is a task to localize objects in a 3D scene and generate descriptive sentences for each object. Recent approaches in 3D dense captioning have adopted transformer encoder-decoder frameworks from object detection to build an end-to-end pipeline without hand-crafted components. However, these approaches struggle with contradicting objectives where a single query attention has to simultaneously view both the tightly localized object regions and contextual environment. To overcome this challenge, we introduce SIA (See-It-All), a transformer pipeline that engages in 3D dense captioning with a novel paradigm called late aggregation. SIA simultaneously decodes two sets of queries-context query and instance query. The instance query focuses on localization and object attribute descriptions, while the context query versatilely captures the region-of-interest of relationships between multiple objects or with the global scene, then aggregated afterwards (i.e., late aggregation) via simple distance-based measures. To further enhance the quality of contextualized caption generation, we design a novel aggregator to generate a fully informed caption based on the surrounding context, the global environment, and object instances. Extensive experiments on two of the most widely-used 3D dense captioning datasets demonstrate that our proposed method achieves a significant improvement over prior methods.
Graph Triple Attention Network: A Decoupled Perspective
Venue: arXiv cs.LG 2024
Year: 2024

Graph Transformers (GTs) have recently achieved significant success in the graph domain by effectively capturing both long-range dependencies and graph inductive biases. However, these methods face two primary challenges: (1) multi-view chaos, which results from coupling multi-view information (positional, structural, attribute), thereby impeding flexible usage and the interpretability of the propagation process. (2) local-global chaos, which arises from coupling local message passing with global attention, leading to issues of overfitting and over-globalizing. To address these challenges, we propose a high-level decoupled perspective of GTs, breaking them down into three components and two interaction levels: positional attention, structural attention, and attribute attention, alongside local and global interaction. Based on this decoupled perspective, we design a decoupled graph triple attention network named DeGTA, which separately computes multi-view attentions and adaptively integrates multi-view local and global information. This approach offers three key advantages: enhanced interpretability, flexible design, and adaptive integration of local and global information. Through extensive experiments, DeGTA achieves state-of-the-art performance across various datasets and tasks, including node classification and graph classification. Comprehensive ablation studies demonstrate that decoupling is essential for improving performance and enhancing interpretability. Our code is available at: https://github.com/wangxiaotang0906/DeGTA
Alignment-Enhanced Decoding:Defending via Token-Level Adaptive Refining of Probability Distributions
Venue: arXiv cs.CL 2024
Year: 2024

Large language models are susceptible to jailbreak attacks, which can result in the generation of harmful content. While prior defenses mitigate these risks by perturbing or inspecting inputs, they ignore competing objectives, the underlying cause of alignment failures. In this paper, we propose Alignment-Enhanced Decoding (AED), a novel defense that employs adaptive decoding to address the root causes of jailbreak issues. We first define the Competitive Index to quantify alignment failures and utilize feedback from self-evaluation to compute post-alignment logits. Then, AED adaptively combines AED and post-alignment logits with the original logits to obtain harmless and helpful distributions. Consequently, our method enhances safety alignment while maintaining helpfulness. We conduct experiments across five models and four common jailbreaks, with the results validating the effectiveness of our approach. Code is available at https://github.com/GIGABaozi/AED.git.
Composing Automatic Differentiation with Custom Derivatives of Higher-Order Functions
Venue: arXiv cs.PL 2024
Year: 2024

Recent theoretical work on automatic differentiation (autodiff) has focused on characteristics such as correctness and efficiency while assuming that all derivatives are automatically generated by autodiff using program transformation, with the exception of a fixed set of derivatives for primitive operations. However, in practice this assumption is insufficient: the programmer often needs to provide custom derivatives for composite functions to achieve efficiency and numerical stability. In this work, we start from the untyped lambda calculus with a reverse-mode autodiff operator, extend it with an operator to attach manual derivatives, and demonstrate its utility via several examples.