Reviewerly/exHarmony huggingface dataset:
FATROP: A Fast Constrained Optimal Control Problem Solver for Robot Trajectory Optimization and Control

Trajectory optimization is a powerful tool for robot motion planning and control. State-of-the-art general-purpose nonlinear programming solvers are versatile, handle constraints effectively and provide a high numerical robustness, but they are slow because they do not fully exploit the optimal control problem structure at hand. Existing structure-exploiting solvers are fast, but they often lack techniques to deal with nonlinearity or rely on penalty methods to enforce (equality or inequality) path constraints. This work presents FATROP: a trajectory optimization solver that is fast and benefits from the salient features of general-purpose nonlinear optimization solvers. The speed-up is mainly achieved through the integration of a specialized linear solver, based on a Riccati recursion that is generalized to also support stagewise equality constraints. To demonstrate the algorithm's potential, it is bench-marked on a set of robot problems that are challenging from a numerical perspective, including problems with a minimum-time objective and no-collision constraints. The solver is shown to solve problems for trajectory generation of a quadrotor, a robot manipulator and a truck-trailer problem in a few tens of milliseconds. The algorithm's C <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">++</sup> -code implementation accompanies this work as open source software, released under the GNU Lesser General Public License (LGPL). This software framework may encourage and enable the robotics community to use trajectory optimization in more challenging applications.
HELSA: Hierarchical Reinforcement Learning with Spatiotemporal Abstraction for Large-Scale Multi-Agent Path Finding

The Multi-Agent Path Finding (MAPF) problem is a critical challenge in dynamic multi-robot systems. Recent studies have revealed that multi-agent reinforcement learning (MARL) is a promising approach to solving MAPF problems in a fully decentralized manner. However, as the size of the multi-robot system increases, sample inefficiency becomes a major impediment to learning-based methods. This paper presents a hierarchical reinforcement learning (HRL) framework for large-scale multi-agent path finding, featuring applying spatial and temporal abstraction to capture intermediate reward and thus encourage efficient exploration. Specifically, we introduce a meta controller that partitions the map into interconnected regions and optimizes agents' region-wise paths towards globally better solutions. Additionally, we design a lower-level controller that efficiently solves each sub-problem by incorporating heuristic guidance and an inter-agent communication mechanism with RL-based policies. Our empirical results on test instances of various scales demonstrate that our method outperforms existing approaches in terms of both success rate and makespan.
Hyperspectral Imaging based Characterization and Identification of Sintered UO<sub>2</sub> Fuel Pellets

Hyperspectral Imaging (HSI) is a well-established technology able to capture the same spatial scene or image in hundreds of different wavelengths across the electromagnetic spectrum, covering not only the visible but also the shortwave infrared range (1000-2500nm), potentially revealing information otherwise invisible to the human eye. In this work, we explored whether HSI technology can be used for the fast-non-invasive characterization and identification of sintered UO <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</inf> fuel pellets. Preliminary experiments included the imaging of the pellets, revealing their spectral responses. These show promising features that could be used for their identification, where two different groups of pellets, pure and doped, seem to be easily recognized based on their spectral response. The experiments included a pixel-wise classification map generated via the Spectral Angle Mapper (SAM) technique in which the pure and doped pellets are successfully identified.
ShiVaNet: Shift Variant Image Deconvolution using Deep Learning

Image Deconvolution is a well-studied problem that seeks to restore the original sharp image from a blurry image formed in the imaging system. The Point Spread function(PSF) of a particular system can be used to infer the original sharp image given the blurred image. However, such a problem is usually simplified by making the shift-invariant assumption over the field of view(FOV).Realistic systems are shift variant; the optical system’s point spread function depends on the position of the object point from the principal axis. For example, asymmetrical lenses can cause space variant aberration.In this paper, we first simulate our space-variant aberrations by generating PSFs using the Seidel Aberration polynomial and use a space-variant forward blur model to generate our shift variant blurred image pairs. We then introduce, ShiVaNet. It is a two-stage architecture that builds upon the Learnable Wiener Deconvolution block as described in [1] by introducing Simplified Channel Attention and Transpose Attention to improve the performance of the module. We also devise a novel UNet refinement block by fusing a ConvNext-V2 block with Channel Attention and coupling with Transposed Attention [2]. Our model performs better than state-of-the-art restoration models by a factor of 0.2 dB.
Early or Late Fusion Matters: Efficient RGB-D Fusion in Vision Transformers for 3D Object Recognition

The Vision Transformer (ViT) architecture has established its place in computer vision literature, however, training ViTs for RGB-D object recognition remains an understudied topic, viewed in recent literature only through the lens of multi-task pretraining in multiple vision modalities. Such approaches are often computationally intensive, relying on the scale of multiple pretraining datasets to align RGB with 3D information. In this work, we propose a simple yet strong recipe for transferring pretrained ViTs in RGB-D domains for 3D object recognition, focusing on fusing RGB and depth representations encoded jointly by the ViT. Compared to previous works in multimodal Transformers, the key challenge here is to use the attested flexibility of ViTs to capture cross-modal interactions at the downstream and not the pretraining stage. We explore which depth representation is better in terms of resulting accuracy and compare early and late fusion techniques for aligning the RGB and depth modalities within the ViT architecture. Experimental results in the Washington RGB-D Objects dataset (ROD) demonstrate that in such RGB → RGB-D scenarios, late fusion techniques work better than most popularly employed early fusion. With our transfer baseline, fusion ViTs score up to 95.4% top-1 accuracy in ROD, achieving new state-of-the-art results in this benchmark. We further show the benefits of using our multimodal fusion baseline over unimodal feature extractors in a synthetic-to-real visual adaptation as well as in an open-ended lifelong learning scenario in the ROD benchmark, where our model outperforms previous works by a margin of >8%. Finally, we integrate our method with a robot framework and demonstrate how it can serve as a perception utility in an interactive robot learning scenario, both in simulation and with a real robot.
Learning from Symmetry: Meta-Reinforcement Learning with Symmetrical Behaviors and Language Instructions

Meta-reinforcement learning (meta-RL) is a promising approach that enables the agent to learn new tasks quickly. However, most meta-RL algorithms show poor generalization in multi-task scenarios due to the insufficient task information provided only by rewards. Language-conditioned meta-RL improves the generalization capability by matching language instructions with the agent's behaviors. While both behaviors and language instructions have symmetry, which can speed up human learning of new knowledge. Thus, combining symmetry and language instructions into meta-RL can help improve the algorithm's generalization and learning efficiency. We propose a dual-MDP meta-reinforcement learning method that enables learning new tasks efficiently with symmetrical behav-iors and language instructions. We evaluate our method in mul-tiple challenging manipulation tasks, and experimental results show that our method can greatly improve the generalization and learning efficiency of meta-reinforcement learning. Videos are available at https://tumi6robot.wixsite.com/symmetry/.
ARMP: Autoregressive Motion Planning for Quadruped Locomotion and Navigation in Complex Indoor Environments

Generating natural and physically feasible motions for legged robots has been a challenging problem due to its complex dynamics. In this work, we introduce a novel learning-based framework of autoregressive motion planner (ARMP) for quadruped locomotion and navigation. Our method can generate motion plans with an arbitrary length in an autore-gressive fashion, unlike most offline trajectory optimization algorithms for a fixed trajectory length. To this end, we first construct the motion library by solving a dense set of trajectory optimization problems for diverse scenarios and parameter settings. Then we learn the motion manifold from the dataset in a supervised learning fashion. We show that the proposed ARMP can generate physically plausible motions for various tasks and situations. We also showcase that our method can be successfully integrated with the recent robot navigation frameworks as a low-level controller and unleash the full capability of legged robots for complex indoor navigation.
Audit of Significant Objects of Critical Information Infrastructure

The increasing importance of protecting critical information infrastructures has emerged as a top concern in the realm of information security. This heightened focus stems from the potential for an intruder's successful attack to disrupt essential functions within various sectors of a nation. To preemptively mitigate the repercussions of such attacks on information security systems, prioritizing the timely detection of vulnerabilities is paramount. Conducting audits represent the most effective approach in identifying potential weaknesses. However, the current standard offered by regulators for audit outcomes remains limited to a binary "pass/fail" result. Nevertheless, the audit methodology expounded upon in this article introduces a novel visualization technique empowering stakeholders to discern all discernible gaps present within the information protection system of a critical information infrastructure entity. By expanding upon the conventional binary framework, this approach enables a more nuanced understanding of security vulnerabilities and enhances overall preparedness efforts. Also, this audit methodology empowers organizations to proactively improve their security posture.
Generating Scenarios from High-Level Specifications for Object Rearrangement Tasks

Rearranging objects is an essential skill for robots. To quickly teach robots new rearrangements tasks, we would like to generate training scenarios from high-level specifications that define the relative placement of objects for the task at hand. Ideally, to guide the robot's learning we also want to be able to rank these scenarios according to their difficulty. Prior work has shown how generating diverse scenario from specifications and providing the robot with easy-to-difficult samples can improve the learning. Yet, existing scenario generation methods typically cannot generate diverse scenarios while controlling their difficulty. We address this challenge by conditioning generative models on spatial logic specifications to generate spatially-structured scenarios that meet the specification and desired difficulty level. Our experiments showed that generative models are more effective and data-efficient than rejection sam-pling and that the spatially-structured scenarios can drastically improve training of downstream tasks by orders of magnitude.
Robust Bayesian Vision Transformer for Image Analysis and Classification

The tremendous success of the Transformer neural networks in natural language processing (NLP) boosts the interest in integrating and applying Transformer models to computer vision applications. The Vision Transformer (ViT) model adeptly captures extensive inter-dependencies among input sequences by employing the self-attention mechanism, thereby transforming picture data into semantically significant representations. In recent times, ViT has demonstrated superior performance in image classification tasks by implementing the transformer architecture, surpassing the capabilities of convolutional neural networks. Nevertheless, these deterministic architectures cannot evaluate the uncertainty associated with predictions, a crucial aspect in divergent and noisy situations. In order to guarantee the effectiveness and reliability of ViT in critical applications, the Bayesian Inference facilitates the process of making probabilistic predictions. Estimating the Bayesian posterior distribution of the network parameters enables a systematic method for reasoning about predictive uncertainty. The major difficulty in this process lies in propagating the posterior distribution through numerous non-linear layers of ViT architecture, which is mathematically cumbersome. In this paper, we propose a Bayesian Vision Transformer (Bayes-ViT) model, which seeks to make predictions as well as quantify the uncertainty associated with the output decision. The variational optimization approximates the posterior distribution over the unknown model parameters by minimizing the evidence lower bound (ELBO) loss function. The variational moments are propagated through the sequential, non-linear layers of Bayes-ViT by employing the first-order Taylor approximation. The covariance matrix of the predictive distribution effectively manifests the uncertainty associated with the output prediction. Extensive experiments on benchmark datasets (MNIST and Fashion-MNIST) exhibit (1) the superior robustness against noise and adversarial attacks compared to the deterministic ViT and (2) the self-evaluation ability based on the prediction uncertainty that becomes more evident when noise levels increase.
Are Industrial ML Image Classifiers Robust to Withstand Adversarial Attacks on Videos?

We investigate the impact of adversarial attacks against videos on the object detection and classification performance of industrial Machine Learning (ML) application. Specifically, we design the use case with the Intelligent Transportation System that processes real videos recorded by the vehicles’ dash cams and detects traffic lights and road signs in these videos. As the ML system, we employed Rekognition cloud service from Amazon, which is a commercial tool for on-demand object detection in the data of various modalities. To study Rekognition robustness to adversarial attacks, we manipulate the videos by adding the noise to them. We vary the intensity of the added noise by setting the ratio of randomly selected pixels affected by this noise. We then process the videos affected by the noise of various intensity and evaluate the performance demonstrated by Rekognition. As the evaluation metrics, we employ confidence scores provided by Rekognition, and the ratio of correct decisions that shows how successful is Rekognition in recognizing the patterns of interest in the frame. According to our results, even simple adversarial attacks of low intensity (up to 2% of the affected pixels in a single frame) result in a significant Rekognition performance decrease and require additional measures to improve the robustness and satisfy the industrial ML applications’ demands.
Virtual Reality for Enhancing Engagement with Net Zero Transitions

Immersive experiences can increase engagement and improve data understanding. We explore the use of virtual reality to visualise the complexities of electric networks and facilitate a greater understanding of net zero initiatives. We propose that an abstract metaphorical immersive experience can provide an intuitive overview of the complex components of energy networks and how they behave over time in response to fluctuating supply and demand in order to achieve stability and sustainability. This work aims to enable a variety of stakeholders to better understand and associate the complexities in the systems and be aware of the connections over time and space. Using the Monash University electric network system as a case study, we explore the network dynamics and fluctuations over time. The virtual reality experience allows users to view campus electricity assets and local and remote energy sources, as well as witness the dynamics of the network. Thus, the complexities of the energy system and its impact on sustainability can be more readily understood.
CEFHRI: A Communication Efficient Federated Learning Framework for Recognizing Industrial Human-Robot Interaction

Human-robot interaction (HRI) is a rapidly growing field that encompasses social and industrial applications. Machine learning plays a vital role in industrial HRI by enhancing the adaptability and autonomy of robots in complex environments. However, data privacy is a crucial concern in the interaction between humans and robots, as companies need to protect sensitive data while machine learning algorithms require access to large datasets. Federated Learning (FL) offers a solution by enabling the distributed training of models without sharing raw data. Despite extensive research on Federated learning (FL) for tasks such as natural language processing (NLP) and image classification, the question of how to use FL for HRI remains an open research problem. The traditional FL approach involves transmitting large neural network parameter matrices between the server and clients, which can lead to high communication costs and often becomes a bottleneck in FL. This paper proposes a communication-efficient FL framework for human-robot interaction (CEFHRI) to address the challenges of data heterogeneity and communication costs. The framework leverages pre-trained models and introduces a trainable spatiotemporal adapter for video understanding tasks in HRI. Experimental results on three human-robot interaction benchmark datasets: HRI30, InHARD, and COIN demonstrate the superiority of CEFHRI over full fine-tuning in terms of communication costs. The proposed methodology provides a secure and efficient approach to HRI federated learning, particularly in industrial environments with data privacy concerns and limited communication bandwidth. Our code is available at https://github.com/umarkhalidAI/CEFHRI-Efficient-Federated-Learning.
LiDAR-SGMOS: Semantics-Guided Moving Object Segmentation with 3D LiDAR

Most of the existing moving object segmentation (MOS) methods regard MOS as an independent task, in this paper, we associate the MOS task with semantic segmentation, and propose a semantics-guided network for moving object segmentation (LiDAR-SGMOS). We first transform the range image and semantic features of the past scan into the range view of current scan based on the relative pose between scans. The residual image is obtained by calculating the normalized absolute difference between the current and transformed range images. Then, we apply a Meta-Kernel based cross scan fusion (CSF) module to adaptively fuse the range images and semantic features of current scan, the residual image and transformed features. Finally, the fused features with rich motion and semantic information are processed to obtain reliable MOS results. We also introduce a residual image augmentation method to further improve the MOS performance. Our method outperforms most LiDAR-MOS methods with only two sequential LiDAR scans as inputs on the SemanticKITTI MOS dataset.
Automated Testing of Functional Requirements for Virtual Reality Applications

As the virtual reality (VR) application landscape continues to expand exponentially, the software industry faces an increasingly critical need for advanced tools to ensure the quality of these applications. This paper meticulously assesses the strengths and weaknesses of existing solutions designed for the automated testing of VR applications. It introduces an innovative approach that combines traditional test coverage parameters, including GUI, API, performance requirements, load requirements, and user scenarios, with a specialized focus on usability testing. This usability testing involves simulating real-world interactions within a VR application using commonly available VR input devices. Notably, the paper outlines the development of a program rooted in this methodology, significantly enhancing the testing process by eliminating the requirement for a VR headset. Instead, standard input devices such as a keyboard and mouse are utilized as controllers, providing a practical and comprehensive means of ensuring the quality and functionality of VR applications in a rapidly evolving industry.
Collective Decision-Making and Change Detection with Bayesian Robots in Dynamic Environments

Solving complex problems collectively with simple entities is a challenging task for swarm robotics. For the task of collective decision-making, robots decide based on local observations on the microscopic level to achieve consensus on the macroscopic level. We study this problem for a common benchmark of classifying distributed features in a binary dynamic environment. Our special focus is on environmental features that are dynamic as they change during the experiment. We present a control algorithm that uses sophisticated statistical change detection in combination with Bayesian robots to classify dynamic environments. The main profit is to reduce false positives allowing for improved speed and accuracy in decision-making. Supported by results from various simulated experiments, we introduce three feedback loops to balance speed and accuracy. In our benchmarks, we show the superiority of our new approach over previous works on Bayesian robots. Our approach of using change detection shows a more reliable detection of environmental changes. This enables the swarm to successfully classify even difficult environments (i.e., hard to detect differences between the binary features), while achieving faster and more accurate results in simpler environments.
ITIRRT: A Decoupled Framework for the Integration of Machine Learning Into Path Planning

Machine learning models have been successfully applied in many separate domains in the recent years. Yet, there is still a large disconnect with the path planning community mainly due to implementation difficulty. In this paper, we identify the issues and present a path planning framework that aims to ease the integration of machine learning models. The framework is divided into two phases: prediction and planning. During the prediction phase machine learning models are used to predict a path solution which, while not perfect, is enough to jump-start our proposed planner. The planning phase begins with the initialization of multiple independent tree data structures and, during execution, our planner merges the different trees until a path is found. Due to the architecture of our framework, the prediction and the planning phases are totally decoupled, resulting in a planner that treats the information given by the prediction model as a guess and not as a path to be corrected. This also allows our planner to maintain its probabilistic completeness. Our planner was evaluated on simulated environments and on a small robotic arm. We show that our planner can successfully leverage predicted paths to significantly improve the planning performances. We also demonstrate the flexibility of our framework by integrating multiple machine learning models of varying complexity.