OpenRaiser/Intern-Atlas huggingface dataset:
Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representations
Venue: arXiv cs.AI 2024
Year: 2024

Causal abstraction is a promising theoretical framework for explainable artificial intelligence that defines when an interpretable high-level causal model is a faithful simplification of a low-level deep learning system. However, existing causal abstraction methods have two major limitations: they require a brute-force search over alignments between the high-level model and the low-level one, and they presuppose that variables in the high-level model will align with disjoint sets of neurons in the low-level one. In this paper, we present distributed alignment search (DAS), which overcomes these limitations. In DAS, we find the alignment between high-level and low-level models using gradient descent rather than conducting a brute-force search, and we allow individual neurons to play multiple distinct roles by analyzing representations in non-standard bases-distributed representations. Our experiments show that DAS can discover internal structure that prior approaches miss. Overall, DAS removes previous obstacles to conducting causal abstraction analyses and allows us to find conceptual structure in trained neural nets.
Resource-aware Cyber Deception for Microservice-based Applications
Venue: arXiv cs.CR 2024
Year: 2024

Cyber deception can be a valuable addition to traditional cyber defense mechanisms, especially for modern cloud-native environments with a fading security perimeter. However, pre-built decoys used in classical computer networks are not effective in detecting and mitigating malicious actors due to their inability to blend with the variety of applications in such environments. On the other hand, decoys cloning the deployed microservices of an application can offer a high-fidelity deception mechanism to intercept ongoing attacks within production environments. However, to fully benefit from this approach, it is essential to use a limited amount of decoy resources and devise a suitable cloning strategy to minimize the impact on legitimate services performance. Following this observation, we formulate a non-linear integer optimization problem that maximizes the number of attack paths intercepted by the allocated decoys within a fixed resource budget. Attack paths represent the attacker's movements within the infrastructure as a sequence of violated microservices. We also design a heuristic decoy placement algorithm to approximate the optimal solution and overcome the computational complexity of the proposed formulation. We evaluate the performance of the optimal and heuristic solutions against other schemes that use local vulnerability metrics to select which microservices to clone as decoys. Our results show that the proposed allocation strategy achieves a higher number of intercepted attack paths compared to these schemes while requiring approximately the same number of decoys.
From RGB images to Dynamic Movement Primitives for planar tasks
Venue: arXiv cs.RO 2024
Year: 2024

DMP have been extensively applied in various robotic tasks thanks to their generalization and robustness properties. However, the successful execution of a given task may necessitate the use of different motion patterns that take into account not only the initial and target position but also features relating to the overall structure and layout of the scene. To make DMP applicable to a wider range of tasks and further automate their use, we design a framework combining deep residual networks with DMP, that can encapsulate different motion patterns of a planar task, provided through human demonstrations on the RGB image plane. We can then automatically infer from new raw RGB visual input the appropriate DMP parameters, i.e. the weights that determine the motion pattern and the initial/target positions. We compare our method against another SoA method for inferring DMP from images and carry out experimental validations in two different planar tasks.
Generative Modeling with Flow-Guided Density Ratio Learning
Venue: arXiv cs.LG 2024
Year: 2024

We present Flow-Guided Density Ratio Learning (FDRL), a simple and scalable approach to generative modeling which builds on the stale (time-independent) approximation of the gradient flow of entropy-regularized f-divergences introduced in recent work. Specifically, the intractable time-dependent density ratio is approximated by a stale estimator given by a GAN discriminator. This is sufficient in the case of sample refinement, where the source and target distributions of the flow are close to each other. However, this assumption is invalid for generation and a naive application of the stale estimator fails due to the large chasm between the two distributions. FDRL proposes to train a density ratio estimator such that it learns from progressively improving samples during the training process. We show that this simple method alleviates the density chasm problem, allowing FDRL to generate images of dimensions as high as $128\times128$, as well as outperform existing gradient flow baselines on quantitative benchmarks. We also show the flexibility of FDRL with two use cases. First, unconditional FDRL can be easily composed with external classifiers to perform class-conditional generation. Second, FDRL can be directly applied to unpaired image-to-image translation with no modifications needed to the framework. Our code is publicly available at ttps://github.com/clear-nus/fdrl.
GRADE: Generating Realistic And Dynamic Environments for Robotics Research with Isaac Sim
Venue: arXiv cs.RO 2024
Year: 2024

Synthetic data and novel rendering techniques have greatly influenced computer vision research in tasks like target tracking and human pose estimation. However, robotics research has lagged behind in leveraging it due to the limitations of most simulation frameworks, including the lack of low-level software control and flexibility, Robot Operating System integration, realistic physics, or photorealism. This hindered progress in (visual-)perception research, e.g. in autonomous robotics, especially in dynamic environments. Visual Simultaneous Localization and Mapping (V-SLAM), for instance, has been mostly developed passively, in static environments, and evaluated on few pre-recorded dynamic datasets due to the difficulties of realistically simulating dynamic worlds and the huge sim-to-real gap. To address these challenges, we present GRADE (Generating Realistic and Dynamic Environments), a highly customizable framework built upon NVIDIA Isaac Sim. We leverage Isaac's rendering capabilities and low-level APIs to populate and control the simulation, collect ground-truth data, and test online and offline approaches. Importantly, we introduce a new way to precisely repeat a recorded experiment within a physically enabled simulation while allowing environmental and simulation changes. Next, we collect a synthetic dataset of richly annotated videos in dynamic environments with a flying drone. Using that, we train detection and segmentation models for humans, closing the syn-to-real gap. Finally, we benchmark state-of-the-art dynamic V-SLAM algorithms, revealing their short tracking times and low generalization capabilities. We also show for the first time that the top-performing deep learning models do not achieve the best SLAM performance. Code and data are provided as open-source at https://grade.is.tue.mpg.de.
MenuCraft: Interactive Menu System Design with Large Language Models
Venue: arXiv cs.CL 2024
Year: 2024

Menu system design for user interfaces is a challenging task involving many design options and various human factors. For example, one crucial factor that designers need to consider is the semantic and systematic relation of menu commands. However, capturing these relations can be challenging due to limited available resources. Large language models can be helpful in this regard, using their pre-training knowledge to design and refine menu systems. In this paper, we propose MenuCraft, an AI-assisted designer for menu design that enables collaboration between the designer and a dialogue system to design menus. MenuCraft offers an interactive language-based menu design tool that simplifies the menu design process and enables easy customization of design options. MenuCraft supports a variety of interactions through dialog that allows performing in-context learning.
Efficient Visuo-Haptic Object Shape Completion for Robot Manipulation
Venue: arXiv cs.RO 2024
Year: 2024

For robot manipulation, a complete and accurate object shape is desirable. Here, we present a method that combines visual and haptic reconstruction in a closed-loop pipeline. From an initial viewpoint, the object shape is reconstructed using an implicit surface deep neural network. The location with highest uncertainty is selected for haptic exploration, the object is touched, the new information from touch and a new point cloud from the camera are added, object position is re-estimated and the cycle is repeated. We extend Rustler et al. (2022) by using a new theoretically grounded method to determine the points with highest uncertainty, and we increase the yield of every haptic exploration by adding not only the contact points to the point cloud but also incorporating the empty space established through the robot movement to the object. Additionally, the solution is compact in that the jaws of a closed two-finger gripper are directly used for exploration. The object position is re-estimated after every robot action and multiple objects can be present simultaneously on the table. We achieve a steady improvement with every touch using three different metrics and demonstrate the utility of the better shape reconstruction in grasping experiments on the real robot. On average, grasp success rate increases from 63.3% to 70.4% after a single exploratory touch and to 82.7% after five touches. The collected data and code are publicly available (https://osf.io/j6rkd/, https://github.com/ctu-vras/vishac)
Soft insoles for estimating 3D ground reaction forces using 3D printed foam-like sensors
Venue: arXiv cs.RO 2024
Year: 2024

Sensorized insoles provide a tool for gait studies and health monitoring during daily life. For users to accept such insoles they need to be comfortable and lightweight. Previous work has already demonstrated that estimation of ground reaction forces (GRFs) is possible with insoles. However, these are often assemblies of commercial components restricting design freedom and customization. Within this work, we investigate using four 3D-printed soft foam-like sensors to sensorize an insole. These sensors were combined with system identification of Hammerstein-Wiener models to estimate the 3D GRFs, which were compared to values from an instrumented treadmill as the golden standard. It was observed that the four sensors behaved in line with the expected change in pressure distribution during the gait cycle. In addition, the identified (personalized) Hammerstein-Wiener models showed the best estimation performance (on average RMS error 9.3%, R^2=0.85 and mean absolute error (MAE) 7%) of the vertical, mediolateral, and anteroposterior GRFs. Thereby showing that these sensors can estimate the resulting 3D force reasonably well. These results for nine participants were comparable to or outperformed other works that used commercial FSRs with machine learning. The identified models did decrease in estimation performance over time but stayed on average 11.35% RMS and 8.6% MAE after a week with the Hammerstein-Wiener model seeming consistent between days two and seven. These results show promise for using 3D-printed soft piezoresistive foam-like sensors with system identification to be a viable approach for applications that require softness, lightweight, and customization such as wearable (force) sensors.
Multimodal Parameter-Efficient Few-Shot Class Incremental Learning
Venue: arXiv cs.CV 2024
Year: 2024

Few-Shot Class Incremental Learning (FSCIL) is a challenging continual learning task, where limited training examples are available during several learning sessions. To succeed in this task, it is necessary to avoid over-fitting new classes caused by biased distributions in the few-shot training sets. The general approach to address this issue involves enhancing the representational capability of a pre-defined backbone architecture by adding special modules for backward compatibility with older classes. However, this approach has not yet solved the dilemma of ensuring high classification accuracy over time while reducing the gap between the performance obtained on larger training sets and the smaller ones. In this work, we propose an alternative approach called Continual Parameter-Efficient CLIP (CPE-CLIP) to reduce the loss of information between different learning sessions. Instead of adapting additional modules to address information loss, we leverage the vast knowledge acquired by CLIP in large-scale pre-training and its effectiveness in generalizing to new concepts. Our approach is multimodal and parameter-efficient, relying on learnable prompts for both the language and vision encoders to enable transfer learning across sessions. We also introduce prompt regularization to improve performance and prevent forgetting. Our experimental results demonstrate that CPE-CLIP significantly improves FSCIL performance compared to state-of-the-art proposals while also drastically reducing the number of learnable parameters and training costs.
Breaking Symmetries Leads to Diverse Quadrupedal Gaits
Venue: arXiv cs.RO 2024
Year: 2024

Symmetry manifests itself in legged locomotion in a variety of ways. No matter where a legged system begins to move periodically, the torso and limbs coordinate with each other's movements in a similar manner. Also, in many gaits observed in nature, the legs on both sides of the torso move in exactly the same way, sometimes they are just half a period out of phase. Furthermore, when some animals move forward and backward, their movements are strikingly similar as if the time had been reversed. This work aims to generalize these phenomena and propose formal definitions of symmetries in legged locomotion using group theory terminology. Symmetries in some common quadrupedal gaits such as pronking, bounding, half-bounding, and galloping have been discussed. Moreover, a spring-mass model has been used to demonstrate how breaking symmetries can alter gaits in a legged system. Studying the symmetries may provide insight into which gaits may be suitable for a particular robotic design, or may enable roboticists to design more agile and efficient robot controllers by using certain gaits.
Non-aligned supervision for Real Image Dehazing
Venue: arXiv cs.CV 2024
Year: 2024

Removing haze from real-world images is challenging due to unpredictable weather conditions, resulting in the misalignment of hazy and clear image pairs. In this paper, we propose an innovative dehazing framework that operates under non-aligned supervision. This framework is grounded in the atmospheric scattering model, and consists of three interconnected networks: dehazing, airlight, and transmission networks. In particular, we explore a non-alignment scenario that a clear reference image, unaligned with the input hazy image, is utilized to supervise the dehazing network. To implement this, we present a multi-scale reference loss that compares the feature representations between the referred image and the dehazed output. Our scenario makes it easier to collect hazy/clear image pairs in real-world environments, even under conditions of misalignment and shift views. To showcase the effectiveness of our scenario, we have collected a new hazy dataset including 415 image pairs captured by mobile Phone in both rural and urban areas, called "Phone-Hazy". Furthermore, we introduce a self-attention network based on mean and variance for modeling real infinite airlight, using the dark channel prior as positional guidance. Additionally, a channel attention network is employed to estimate the three-channel transmission. Experimental results demonstrate the superior performance of our framework over existing state-of-the-art techniques in the real-world image dehazing task. Phone-Hazy and code will be available at https://fanjunkai1.github.io/projectpage/NSDNet/index.html.
Scalable Stochastic Gradient Riemannian Langevin Dynamics in Non-Diagonal Metrics
Venue: arXiv cs.LG 2024
Year: 2024

Stochastic-gradient sampling methods are often used to perform Bayesian inference on neural networks. It has been observed that the methods in which notions of differential geometry are included tend to have better performances, with the Riemannian metric improving posterior exploration by accounting for the local curvature. However, the existing methods often resort to simple diagonal metrics to remain computationally efficient. This loses some of the gains. We propose two non-diagonal metrics that can be used in stochastic-gradient samplers to improve convergence and exploration but have only a minor computational overhead over diagonal metrics. We show that for fully connected neural networks (NNs) with sparsity-inducing priors and convolutional NNs with correlated priors, using these metrics can provide improvements. For some other choices the posterior is sufficiently easy also for the simpler metrics.
SpyroPose: SE(3) Pyramids for Object Pose Distribution Estimation
Venue: arXiv cs.CV 2024
Year: 2024

Object pose estimation is a core computer vision problem and often an essential component in robotics. Pose estimation is usually approached by seeking the single best estimate of an object's pose, but this approach is ill-suited for tasks involving visual ambiguity. In such cases it is desirable to estimate the uncertainty as a pose distribution to allow downstream tasks to make informed decisions. Pose distributions can have arbitrary complexity which motivates estimating unparameterized distributions, however, until now they have only been used for orientation estimation on SO(3) due to the difficulty in training on and normalizing over SE(3). We propose a novel method for pose distribution estimation on SE(3). We use a hierarchical grid, a pyramid, which enables efficient importance sampling during training and sparse evaluation of the pyramid at inference, allowing real time 6D pose distribution estimation. Our method outperforms state-of-the-art methods on SO(3), and to the best of our knowledge, we provide the first quantitative results on pose distribution estimation on SE(3). Code will be available at spyropose.github.io
Improving Masked Autoencoders by Learning Where to Mask
Venue: arXiv cs.CV 2024
Year: 2024

Masked image modeling is a promising self-supervised learning method for visual data. It is typically built upon image patches with random masks, which largely ignores the variation of information density between them. The question is: Is there a better masking strategy than random sampling and how can we learn it? We empirically study this problem and initially find that introducing object-centric priors in mask sampling can significantly improve the learned representations. Inspired by this observation, we present AutoMAE, a fully differentiable framework that uses Gumbel-Softmax to interlink an adversarially-trained mask generator and a mask-guided image modeling process. In this way, our approach can adaptively find patches with higher information density for different images, and further strike a balance between the information gain obtained from image reconstruction and its practical training difficulty. In our experiments, AutoMAE is shown to provide effective pretraining models on standard self-supervised benchmarks and downstream tasks.
Modular Quantization-Aware Training for 6D Object Pose Estimation
Venue: arXiv cs.CV 2024
Year: 2024

Edge applications, such as collaborative robotics and spacecraft rendezvous, demand efficient 6D object pose estimation on resource-constrained embedded platforms. Existing 6D pose estimation networks are often too large for such deployments, necessitating compression while maintaining reliable performance. To address this challenge, we introduce Modular Quantization-Aware Training (MQAT), an adaptive and mixed-precision quantization-aware training strategy that exploits the modular structure of modern 6D pose estimation architectures. MQAT guides a systematic gradated modular quantization sequence and determines module-specific bit precisions, leading to quantized models that outperform those produced by state-of-the-art uniform and mixed-precision quantization techniques. Our experiments showcase the generality of MQAT across datasets, architectures, and quantization algorithms. Remarkably, MQAT-trained quantized models achieve a significant accuracy boost (>7%) over the baseline full-precision network while reducing model size by a factor of 4x or more. Our project website is at: https://saqibjaved1.github.io/MQAT_/
Asynchronous Decentralized Federated Lifelong Learning for Landmark Localization in Medical Imaging
Venue: arXiv cs.LG 2024
Year: 2024

Federated learning is a recent development in the machine learning area that allows a system of devices to train on one or more tasks without sharing their data to a single location or device. However, this framework still requires a centralized global model to consolidate individual models into one, and the devices train synchronously, which both can be potential bottlenecks for using federated learning. In this paper, we propose a novel method of asynchronous decentralized federated lifelong learning (ADFLL) method that inherits the merits of federated learning and can train on multiple tasks simultaneously without the need for a central node or synchronous training. Thus, overcoming the potential drawbacks of conventional federated learning. We demonstrate excellent performance on the brain tumor segmentation (BRATS) dataset for localizing the left ventricle on multiple image sequences and image orientation. Our framework allows agents to achieve the best performance with a mean distance error of 7.81, better than the conventional all-knowing agent's mean distance error of 11.78, and significantly (p=0.01) better than a conventional lifelong learning agent with a distance error of 15.17 after eight rounds of training. In addition, all ADFLL agents have comparable or better performance than a conventional LL agent. In conclusion, we developed an ADFLL framework with excellent performance and speed-up compared to conventional RL agents.
Gaussian Process on the Product of Directional Manifolds
Venue: arXiv cs.LG 2024
Year: 2024

We present a principled study on defining Gaussian processes (GPs) with inputs on the product of directional manifolds. A circular kernel is first presented according to the von Mises distribution. Based thereon, the hypertoroidal von Mises (HvM) kernel is proposed to establish GPs on hypertori with consideration of correlated circular components. The proposed HvM kernel is demonstrated with multi-output GP regression for learning vector-valued functions on hypertori using the intrinsic coregionalization model. Analytic derivatives for hyperparameter optimization are provided for runtime-critical applications. For evaluation, we synthesize a ranging-based sensor network and employ the HvM-based GPs for data-driven recursive localization. Numerical results show that the HvM-based GP achieves superior tracking accuracy compared to parametric model and GPs of conventional kernel designs.