OpenRaiser/Intern-Atlas huggingface dataset:
Gaussian Time Machine: A Real-Time Rendering Methodology for Time-Variant Appearances
Venue: arXiv cs.CV 2024
Year: 2024

Recent advancements in neural rendering techniques have significantly enhanced the fidelity of 3D reconstruction. Notably, the emergence of 3D Gaussian Splatting (3DGS) has marked a significant milestone by adopting a discrete scene representation, facilitating efficient training and real-time rendering. Several studies have successfully extended the real-time rendering capability of 3DGS to dynamic scenes. However, a challenge arises when training images are captured under vastly differing weather and lighting conditions. This scenario poses a challenge for 3DGS and its variants in achieving accurate reconstructions. Although NeRF-based methods (NeRF-W, CLNeRF) have shown promise in handling such challenging conditions, their computational demands hinder real-time rendering capabilities. In this paper, we present Gaussian Time Machine (GTM) which models the time-dependent attributes of Gaussian primitives with discrete time embedding vectors decoded by a lightweight Multi-Layer-Perceptron(MLP). By adjusting the opacity of Gaussian primitives, we can reconstruct visibility changes of objects. We further propose a decomposed color model for improved geometric consistency. GTM achieved state-of-the-art rendering fidelity on 3 datasets and is 100 times faster than NeRF-based counterparts in rendering. Moreover, GTM successfully disentangles the appearance changes and renders smooth appearance interpolation.
Subspace Mixed-FEM for Real-Time Heterogeneous Elastodynamics
Venue: arXiv cs.GR 2024
Year: 2024

We propose a reduced space mixed finite element method (MFEM) built on a Skinning Eigenmode subspace and material-aware cubature scheme. Our solver is well-suited for simulating scenes with large material and geometric heterogeneities in real-time. This mammoth geometry is composed of 98,175 vertices and 531,565 tetrahedral elements and with a heterogenous composition of widely varying materials of muscles ($E= 5\times10^5$ Pa), joints ($E=1\times10^5$ Pa), and bone ($E=1\times10^{10}$ Pa). The resulting simulation runs at 120 frames per second (FPS).
Towards Counting Markov Equivalence Classes with Logical Constraints
Venue: arXiv cs.LO 2024
Year: 2024

We initiate the study of counting Markov Equivalence Classes (MEC) under logical constraints. MECs are equivalence classes of Directed Acyclic Graphs (DAGs) that encode the same conditional independence structure among the random variables of a DAG model. Observational data can only allow to infer a DAG model up to Markov Equivalence. However, Markov equivalent DAGs can represent different causal structures, potentially super-exponentially many. Hence, understanding MECs combinatorially is critical to understanding the complexity of causal inference. In this paper, we focus on analysing MECs of size one, with logical constraints on the graph topology. We provide a polynomial-time algorithm (w.r.t. the number of nodes) for enumerating essential DAGs (the only members of an MEC of size one) with arbitrary logical constraints expressed in first-order logic with two variables and counting quantifiers (C^2). Our work brings together recent developments in tractable first-order model counting and combinatorics of MECs.
Grounding Toxicity in Real-World Events across Languages
Venue: arXiv cs.CL 2024
Year: 2024

Social media conversations frequently suffer from toxicity, creating significant issues for users, moderators, and entire communities. Events in the real world, like elections or conflicts, can initiate and escalate toxic behavior online. Our study investigates how real-world events influence the origin and spread of toxicity in online discussions across various languages and regions. We gathered Reddit data comprising 4.5 million comments from 31 thousand posts in six different languages (Dutch, English, German, Arabic, Turkish and Spanish). We target fifteen major social and political world events that occurred between 2020 and 2023. We observe significant variations in toxicity, negative sentiment, and emotion expressions across different events and language communities, showing that toxicity is a complex phenomenon in which many different factors interact and still need to be investigated. We will release the data for further research along with our code.
Towards Explainable Test Case Prioritisation with Learning-to-Rank Models
Venue: arXiv cs.SE 2024
Year: 2024

Test case prioritisation (TCP) is a critical task in regression testing to ensure quality as software evolves. Machine learning has become a common way to achieve it. In particular, learning-to-rank (LTR) algorithms provide an effective method of ordering and prioritising test cases. However, their use poses a challenge in terms of explainability, both globally at the model level and locally for particular results. Here, we present and discuss scenarios that require different explanations and how the particularities of TCP (multiple builds over time, test case and test suite variations, etc.) could influence them. We include a preliminary experiment to analyse the similarity of explanations, showing that they do not only vary depending on test case-specific predictions, but also on the relative ranks.
Diffusion-Based Cloud-Edge-Device Collaborative Learning for Next POI Recommendations
Venue: arXiv cs.IR 2024
Year: 2024

The rapid expansion of Location-Based Social Networks (LBSNs) has highlighted the importance of effective next Point-of-Interest (POI) recommendations, which leverage historical check-in data to predict users' next POIs to visit. Traditional centralized deep neural networks (DNNs) offer impressive POI recommendation performance but face challenges due to privacy concerns and limited timeliness. In response, on-device POI recommendations have been introduced, utilizing federated learning (FL) and decentralized approaches to ensure privacy and recommendation timeliness. However, these methods often suffer from computational strain on devices and struggle to adapt to new users and regions. This paper introduces a novel collaborative learning framework, Diffusion-Based Cloud-Edge-Device Collaborative Learning for Next POI Recommendations (DCPR), leveraging the diffusion model known for its success across various domains. DCPR operates with a cloud-edge-device architecture to offer region-specific and highly personalized POI recommendations while reducing on-device computational burdens. DCPR minimizes on-device computational demands through a unique blend of global and local learning processes. Our evaluation with two real-world datasets demonstrates DCPR's superior performance in recommendation accuracy, efficiency, and adaptability to new users and regions, marking a significant step forward in on-device POI recommendation technology.
Thermodynamic Natural Gradient Descent
Venue: arXiv cs.LG 2024
Year: 2024

Second-order training methods have better convergence properties than gradient descent but are rarely used in practice for large-scale training due to their computational overhead. This can be viewed as a hardware limitation (imposed by digital computers). Here we show that natural gradient descent (NGD), a second-order method, can have a similar computational complexity per iteration to a first-order method, when employing appropriate hardware. We present a new hybrid digital-analog algorithm for training neural networks that is equivalent to NGD in a certain parameter regime but avoids prohibitively costly linear system solves. Our algorithm exploits the thermodynamic properties of an analog system at equilibrium, and hence requires an analog thermodynamic computer. The training occurs in a hybrid digital-analog loop, where the gradient and Fisher information matrix (or any other positive semi-definite curvature matrix) are calculated at given time intervals while the analog dynamics take place. We numerically demonstrate the superiority of this approach over state-of-the-art digital first- and second-order training methods on classification tasks and language model fine-tuning tasks.
Federated Learning in Healthcare: Model Misconducts, Security, Challenges, Applications, and Future Research Directions -- A Systematic Review
Venue: arXiv cs.CR 2024
Year: 2024

Data privacy has become a major concern in healthcare due to the increasing digitization of medical records and data-driven medical research. Protecting sensitive patient information from breaches and unauthorized access is critical, as such incidents can have severe legal and ethical complications. Federated Learning (FL) addresses this concern by enabling multiple healthcare institutions to collaboratively learn from decentralized data without sharing it. FL's scope in healthcare covers areas such as disease prediction, treatment customization, and clinical trial research. However, implementing FL poses challenges, including model convergence in non-IID (independent and identically distributed) data environments, communication overhead, and managing multi-institutional collaborations. A systematic review of FL in healthcare is necessary to evaluate how effectively FL can provide privacy while maintaining the integrity and usability of medical data analysis. In this study, we analyze existing literature on FL applications in healthcare. We explore the current state of model security practices, identify prevalent challenges, and discuss practical applications and their implications. Additionally, the review highlights promising future research directions to refine FL implementations, enhance data security protocols, and expand FL's use to broader healthcare applications, which will benefit future researchers and practitioners.
AI-Protected Blockchain-based IoT environments: Harnessing the Future of Network Security and Privacy
Venue: arXiv cs.CR 2024
Year: 2024

Integrating blockchain technology with the Internet of Things offers transformative possibilities for enhancing network security and privacy in the contemporary digital landscape, where interconnected devices and expansive networks are ubiquitous. This paper explores the pivotal role of artificial intelligence in bolstering blockchain-enabled IoT systems, potentially marking a significant leap forward in safeguarding data integrity and confidentiality across networks. Blockchain technology provides a decentralized and immutable ledger, ideal for the secure management of device identities and transactions in IoT networks. When coupled with AI, these systems gain the ability to not only automate and optimize security protocols but also adaptively respond to new and evolving cyber threats. This dual capability enhances the resilience of networks against cyber-attacks, a critical consideration as IoT devices increasingly permeate critical infrastructures. The synergy between AI and blockchain in IoT is profound. AI algorithms can analyze vast amounts of data from IoT devices to detect patterns and anomalies that may signify security breaches. Concurrently, blockchain can ensure that data records are tamper-proof, enhancing the reliability of AI-driven security measures. Moreover, this research evaluates the implications of AI-enhanced blockchain systems on privacy protection within IoT networks. IoT devices often collect sensitive personal data, making privacy a paramount concern. AI can facilitate the development of new protocols that ensure data privacy and user anonymity without compromising the functionality of IoT systems. Through comprehensive analysis and case studies, this paper aims to provide an in-depth understanding of how AI-enhanced blockchain technology can revolutionize network security and privacy in IoT environments.
Maximum Manifold Capacity Representations in State Representation Learning
Venue: arXiv cs.LG 2024
Year: 2024

The expanding research on manifold-based self-supervised learning (SSL) builds on the manifold hypothesis, which suggests that the inherent complexity of high-dimensional data can be unraveled through lower-dimensional manifold embeddings. Capitalizing on this, DeepInfomax with an unbalanced atlas (DIM-UA) has emerged as a powerful tool and yielded impressive results for state representations in reinforcement learning. Meanwhile, Maximum Manifold Capacity Representation (MMCR) presents a new frontier for SSL by optimizing class separability via manifold compression. However, MMCR demands extensive input views, resulting in significant computational costs and protracted pre-training durations. Bridging this gap, we present an innovative integration of MMCR into existing SSL methods, incorporating a discerning regularization strategy that enhances the lower bound of mutual information. We also propose a novel state representation learning method extending DIM-UA, embedding a nuclear norm loss to enforce manifold consistency robustly. On experimentation with the Atari Annotated RAM Interface, our method improves DIM-UA significantly with the same number of target encoding dimensions. The mean F1 score averaged over categories is 78% compared to 75% of DIM-UA. There are also compelling gains when implementing SimCLR and Barlow Twins. This supports our SSL innovation as a paradigm shift, enabling more nuanced high-dimensional data representations.
MAGIC: Map-Guided Few-Shot Audio-Visual Acoustics Modeling
Venue: arXiv cs.CV 2024
Year: 2024

Few-shot audio-visual acoustics modeling seeks to synthesize the room impulse response in arbitrary locations with few-shot observations. To sufficiently exploit the provided few-shot data for accurate acoustic modeling, we present a *map-guided* framework by constructing acoustic-related visual semantic feature maps of the scenes. Visual features preserve semantic details related to sound and maps provide explicit structural regularities of sound propagation, which are valuable for modeling environment acoustics. We thus extract pixel-wise semantic features derived from observations and project them into a top-down map, namely the **observation semantic map**. This map contains the relative positional information among points and the semantic feature information associated with each point. Yet, limited information extracted by few-shot observations on the map is not sufficient for understanding and modeling the whole scene. We address the challenge by generating a **scene semantic map** via diffusing features and anticipating the observation semantic map. The scene semantic map then interacts with echo encoding by a transformer-based encoder-decoder to predict RIR for arbitrary speaker-listener query pairs. Extensive experiments on Matterport3D and Replica dataset verify the efficacy of our framework.
Just rotate it! Uncertainty estimation in closed-source models via multiple queries
Venue: arXiv cs.CV 2024
Year: 2024

We propose a simple and effective method to estimate the uncertainty of closed-source deep neural network image classification models. Given a base image, our method creates multiple transformed versions and uses them to query the top-1 prediction of the closed-source model. We demonstrate significant improvements in the calibration of uncertainty estimates compared to the naive baseline of assigning 100\% confidence to all predictions. While we initially explore Gaussian perturbations, our empirical findings indicate that natural transformations, such as rotations and elastic deformations, yield even better-calibrated predictions. Furthermore, through empirical results and a straightforward theoretical analysis, we elucidate the reasons behind the superior performance of natural transformations over Gaussian noise. Leveraging these insights, we propose a transfer learning approach that further improves our calibration results.
Automatically Identifying Local and Global Circuits with Linear Computation Graphs
Venue: arXiv cs.LG 2024
Year: 2024

Circuit analysis of any certain model behavior is a central task in mechanistic interpretability. We introduce our circuit discovery pipeline with Sparse Autoencoders (SAEs) and a variant called Transcoders. With these two modules inserted into the model, the model's computation graph with respect to OV and MLP circuits becomes strictly linear. Our methods do not require linear approximation to compute the causal effect of each node. This fine-grained graph identifies both end-to-end and local circuits accounting for either logits or intermediate features. We can scalably apply this pipeline with a technique called Hierarchical Attribution. We analyze three kinds of circuits in GPT-2 Small: bracket, induction, and Indirect Object Identification circuits. Our results reveal new findings underlying existing discoveries.
Image-of-Thought Prompting for Visual Reasoning Refinement in Multimodal Large Language Models
Venue: arXiv cs.AI 2024
Year: 2024

Recent advancements in Chain-of-Thought (CoT) and related rationale-based works have significantly improved the performance of Large Language Models (LLMs) in complex reasoning tasks. With the evolution of Multimodal Large Language Models (MLLMs), enhancing their capability to tackle complex multimodal reasoning problems is a crucial frontier. However, incorporating multimodal rationales in CoT has yet to be thoroughly investigated. We propose the Image-of-Thought (IoT) prompting method, which helps MLLMs to extract visual rationales step-by-step. Specifically, IoT prompting can automatically design critical visual information extraction operations based on the input images and questions. Each step of visual information refinement identifies specific visual rationales that support answers to complex visual reasoning questions. Beyond the textual CoT, IoT simultaneously utilizes visual and textual rationales to help MLLMs understand complex multimodal information. IoT prompting has improved zero-shot visual reasoning performance across various visual understanding tasks in different MLLMs. Moreover, the step-by-step visual feature explanations generated by IoT prompting elucidate the visual reasoning process, aiding in analyzing the cognitive processes of large multimodal models
On connections between k-coloring and Euclidean k-means
Venue: arXiv cs.CG 2024
Year: 2024

In the Euclidean $k$-means problems we are given as input a set of $n$ points in $\mathbb{R}^d$ and the goal is to find a set of $k$ points $C\subseteq \mathbb{R}^d$, so as to minimize the sum of the squared Euclidean distances from each point in $P$ to its closest center in $C$. In this paper, we formally explore connections between the $k$-coloring problem on graphs and the Euclidean $k$-means problem. Our results are as follows: $\bullet$ For all $k\ge 3$, we provide a simple reduction from the $k$-coloring problem on regular graphs to the Euclidean $k$-means problem. Moreover, our technique extends to enable a reduction from a structured max-cut problem (which may be considered as a partial 2-coloring problem) to the Euclidean $2$-means problem. Thus, we have a simple and alternate proof of the NP-hardness of Euclidean 2-means problem. $\bullet$ In the other direction, we mimic the $O(1.7297^n)$ time algorithm of Williams [TCS'05] for the max-cut of problem on $n$ vertices to obtain an algorithm for the Euclidean 2-means problem with the same runtime, improving on the naive exhaustive search running in $2^n\cdot \text{poly}(n,d)$ time. $\bullet$ We prove similar results and connections as above for the Euclidean $k$-min-sum problem.
A Methodology to Identify Physical or Computational Experiment Conditions for Uncertainty Mitigation
Venue: arXiv cs.CE 2024
Year: 2024

Complex engineering systems require integration of simulation of sub-systems and calculation of metrics to drive design decisions. This paper introduces a methodology for designing computational or physical experiments for system-level uncertainty mitigation purposes. The methodology follows a previously determined problem ontology, where physical, functional and modeling architectures are decided upon. By carrying out sensitivity analysis techniques utilizing system-level tools, critical epistemic uncertainties can be identified. Afterwards, a framework is introduced to design specific computational and physical experimentation for generating new knowledge about parameters, and for uncertainty mitigation. The methodology is demonstrated through a case study on an early-stage design Blended-Wing-Body (BWB) aircraft concept, showcasing how aerostructures analyses can be leveraged for mitigating system-level uncertainty, by computer experiments or guiding physical experimentation. The proposed methodology is versatile enough to tackle uncertainty management across various design challenges, highlighting the potential for more risk-informed design processes.
Chain of Targeted Verification Questions to Improve the Reliability of Code Generated by LLMs
Venue: arXiv cs.SE 2024
Year: 2024

LLM-based assistants, such as GitHub Copilot and ChatGPT, have the potential to generate code that fulfills a programming task described in a natural language description, referred to as a prompt. The widespread accessibility of these assistants enables users with diverse backgrounds to generate code and integrate it into software projects. However, studies show that code generated by LLMs is prone to bugs and may miss various corner cases in task specifications. Presenting such buggy code to users can impact their reliability and trust in LLM-based assistants. Moreover, significant efforts are required by the user to detect and repair any bug present in the code, especially if no test cases are available. In this study, we propose a self-refinement method aimed at improving the reliability of code generated by LLMs by minimizing the number of bugs before execution, without human intervention, and in the absence of test cases. Our approach is based on targeted Verification Questions (VQs) to identify potential bugs within the initial code. These VQs target various nodes within the Abstract Syntax Tree (AST) of the initial code, which have the potential to trigger specific types of bug patterns commonly found in LLM-generated code. Finally, our method attempts to repair these potential bugs by re-prompting the LLM with the targeted VQs and the initial code. Our evaluation, based on programming tasks in the CoderEval dataset, demonstrates that our proposed method outperforms state-of-the-art methods by decreasing the number of targeted errors in the code between 21% to 62% and improving the number of executable code instances to 13%.
Resurrection Attack: Defeating Xilinx MPU's Memory Protection
Venue: arXiv cs.CR 2024
Year: 2024

Memory protection units (MPUs) are hardware-assisted security features that are commonly used in embedded processors such as the ARM 940T, Infineon TC1775, and Xilinx Zynq. MPUs partition the memory statically, and set individual protection attributes for each partition. MPUs typically define two protection domains: user mode and supervisor mode. Normally, this is sufficient for protecting the kernel and applications. However, we have discovered a way to access a process memory due to a vulnerability in Xilinx MPU (XMPU) implementation that we call Resurrection Attack. We find that XMPU security policy protects user memory from unauthorized access when the user is active. However, when a user's session is terminated, the contents of the memory region of the terminated process are not cleared. An attacker can exploit this vulnerability by gaining access to the memory region after it has been reassigned. The attacker can read the data from the previous user's memory region, thereby compromising the confidentiality. To prevent the Resurrection Attack, the memory region of a terminated process must be cleared. However, this is not the case in the XMPU implementation, which allows our attack to succeed. The Resurrection Attack is a serious security flaw that could be exploited to steal sensitive data or gain unauthorized access to a system. It is important for users of Xilinx FPGAs to be aware of this vulnerability until this flaw is addressed.