Reviewerly/exHarmony huggingface dataset:
Learning Whom to Trust in Navigation: Dynamically Switching Between Classical and Neural Planning

Navigation of terrestrial robots is typically addressed either with localization and mapping (SLAM) followed by classical planning on the dynamically created maps, or by machine learning (ML), often through end-to-end training with reinforcement learning (RL) or imitation learning (IL). Recently, modular designs have achieved promising results, and hybrid algorithms that combine ML with classical planning have been proposed. Existing methods implement these combinations with handcrafted functions, which cannot fully exploit the complementary nature of the policies and the complex regularities between scene structure and planning performance. Our work builds on the hypothesis that the strengths and weaknesses of neural planners and classical planners follow some regularities, which can be learned from training data, in particular from interactions. This is grounded on the assumption that, both, trained planners and the mapping algorithms underlying classical planning are subject to failure cases depending on the semantics of the scene and that this dependence is learnable: for instance, certain areas, objects or scene structures can be reconstructed easier than others. We propose a hierarchical method composed of a high-level planner dynamically switching between a classical and a neural planner. We fully train all neural policies in simulation and evaluate the method in both simulation and real experiments with a LoCoBot robot, showing significant gains in performance, in particular in the real environment. We also qualitatively conjecture on the nature of data regularities exploited by the high-level planner.
VADER: Vector-Quantized Generative Adversarial Network for Motion Prediction

Human motion prediction is an essential component for enabling close-proximity human-robot collaboration. The task of accurately predicting human motion is non-trivial and is compounded by the variability of human motion and the presence of multiple humans in proximity. To address some of the open challenges in motion prediction, in this work, we propose VADER, a novel sequence learning algorithm that models past observed poses using a flexible discrete latent space. VADER introduces the concept of Vector Quantization for human motion prediction, enabling the learning of a discrete latent space without being restricted by any static prior. In addition, we propose a new objective function that uses the discriminator objective to penalize deviation of predicted motion from the ground-truth. Finally, to explicitly model interaction in multiple humans, we introduce a lightweight attention mechanism to condition per-agent prediction on the previous hidden states of all the agents. Our evaluation across three scenarios: single-agent, multi-agent, and human-robot collaboration shows that VADER outperformed all the state-of-the-art approaches, resulting in more feasible human poses that align better with the ground-truth. Finally, we conducted extensive ablation studies to emphasize the importance of the proposed modules.
VL-Grasp: a 6-Dof Interactive Grasp Policy for Language-Oriented Objects in Cluttered Indoor Scenes

Robotic grasping faces new challenges in human-robot-interaction scenarios. We consider the task that the robot grasps a target object designated by human's language directives. The robot not only needs to locate a target based on vision-and-language information, but also needs to predict the reasonable grasp pose candidate at various views and postures. In this work, we propose a novel interactive grasp policy, named Visual-Lingual-Grasp (VL-Grasp), to grasp the target specified by human language. First, we build a new challenging visual grounding dataset to provide functional training data for robotic interactive perception in indoor environments. Second, we propose a 6- Dof interactive grasp policy combined with visual grounding and 6- Dof grasp pose detection to extend the universality of interactive grasping. Third, we design a grasp pose filter module to enhance the performance of the policy. Experiments demonstrate the effectiveness and extendibility of the VL-Grasp in real world. The VL-Grasp achieves a success rate of 72.5 % in different indoor scenes. The code and dataset is available at https://github.com/luyh20/VL-Grasp.
Gender Specificity in Exergaming for Therapy and Rehabilitation

As IT products' developers seek to enhance the level of user experience, more attention is given to consideration the personal characteristics and preferences of the target audience. For exergaming software that is increasingly employed for medical therapy and rehabilitation, motivation and satisfaction of its users is of particular importance, due to the need to ensure its regular usage. The novelty of the VR/AR environment is seen as a positive factor for the above, but the potentially negative reactions to the VR immersion do not gain enough research attention. In our paper, we explore gender differences in users' experience with VR-based exergaming (VRE). We describe the experimental study that we performed with 54 subjects who engaged with Beat Saber game in VR, tasked with crashing various flying blocks with two light sabers. The game score reflected the objective exergaming performance, while we also registered the subjects' self-assessment emotional state data during the VRE experience and their personal characteristics. Our results suggest that the gender-related differences were rather minor: for men, frustration affected the coordination of sensory and motor activities; while for women frustration, personal anxiety and rigidity were related to the decrease in anxiety caused by exergaming. We believe that our findings could be of usage to VR/AR content designers seeking to cater more personalized products target audience and may help to improve VR programs of neurorehabilitation due to mental disorders or cardiovascular diseases.
Medium-Term Forecast of Water Inflow to the Sayano-Shushenskaya Hydroelectric Power Plant Reservoir by Autoregressive Machine Learning Models

Hydropower is an important economic sector. It provides for the generation of green power and proves key in power-frequency control in the energy system. Its peculiarity is its strong dependence on hydrological conditions. Therefore, the task of forecasting and planning the operation of hydroelectric power plants in conditions of changing river flow is relevant. Accurate forecasting of the inflow is used to make the most rational regulation of the reservoir, therefore, increases economic efficiency. A study on forecasting the water inflow into the reservoir of the Sayano-Shushenskaya hydroelectric power plant to improve the efficiency of water-power regimes was performed. The machine learning approach was chosen as a method of medium-term inflow forecasting. Such models as linear regression, polynomial regression and the k-nearest neighbor model are compared in the paper. The purpose of study is to determine the depth of retrospective data and the sampling step based on autoregressive machine learning models. The most efficient result on the test sample was obtained for k-nearest neighbor model with a depth of retrospective data of 24 months and sample step of 1 months. The coefficient of determination was 0,749.
Weakly Supervised Referring Expression Grounding via Dynamic Self-Knowledge Distillation

Weakly supervised referring expression grounding (WREG) is an attractive and challenging task for grounding target regions in images by understanding given referring expressions. WREG learns to ground target objects without the manual annotations between image regions and referring expressions during the model training phase. Different from the predominant grounding pattern of existing models, which locates target objects by reconstructing the region-expression correspondence, we investigate WREG from a novel perspective and enrich the prevailing pattern with self-knowledge distillation. Specifically, we propose a target-guided self-knowledge distillation approach that adopts the target prediction knowledge learned from the previous training iterations as the teacher to guide the subsequent training procedure. In order to avoid the misleading caused by the teacher knowledge with low prediction confidence, we present an uncertaintyaware knowledge refinement strategy to adaptively rectify the teacher knowledge by learning dynamic threshold values based on the model prediction uncertainty. To validate the proposed approach, we implement extensive experiments on three benchmark datasets, i.e., Ref Coco, RefCOCO+, and RefCOCOg. Our approach achieves new state-of-the-art results on several splits of the benchmark datasets, showcasing the advantage of the proposed framework for WREG. The implementation codes and trained models are available at: https://github.com/dami23IWREG.sar_KD.
Efficient Heuristics for Multi-Robot Path Planning in Crowded Environments

Optimal Multi-Robot Path Planning (MRPP) has garnered significant attention due to its many applications in domains including warehouse automation, transportation, and swarm robotics. Current MRPP solvers can be divided into reduction-based, search-based, and rule-based categories, each with their strengths and limitations. Regardless of the methodology, however, the issue of handling dense MRPP instances remains a significant challenge, where existing approaches generally demonstrate a dichotomy regarding solution optimality and efficiency. This study seeks to bridge the gap in optimal MRPP resolution for dense, highly-entangled scenarios, with potential applications to high-density storage systems and traffic congestion control. Toward that goal, we analyze the behaviors of SOTA MRPP algorithms in dense settings and develop two hybrid algorithms leveraging the strengths of existing SOTA algorithms: DCBS (database-accelerated enhanced conflict-based search) and SCBS (sparsified enhanced conflict-based search). Experimental validations demonstrate that DCBS and SCBS deliver a significant reduction in computational time compared to existing bounded-suboptimal methods and improve solution quality compared to existing rule-based methods, achieving a desirable balance between computational efficiency and solution optimality. As a result, DCBS and SCBS are particularly suitable for quickly computing good-quality solutions for multi-robot routing in dense settings. Simulation video https://youtu.be/dZxMPUr7Bqg Upon the publication of the manuscript source code and data will be released at https://github.com/arc-l/dcbs
Graph Neural Networks for Radiological/Nuclear Detection with Static Detector Networks

To address the threat posed by the nefarious use of radiological/nuclear sources, there is a growing interest in deploying connected static detector networks in cities. Connected detector networks can have improved detection sensitivity over unconnected networks by using the correlations between detectors. We explore the use of graph neural networks (GNNs) to exploit such correlations in a minimal detector network of two nodes and one edge in downtown Chicago. We compare a GNN model with and without message passing, and find that the model with message passing performs significantly better (p = 0.03), suggesting that it is leveraging spatiotemporal relationships to improve detection performance. Future work includes assessing the performance of GNN models on larger detector networks and benchmarking them against existing algorithms.
Self-Supervised Object Goal Navigation with In-Situ Finetuning

A household robot should be able to navigate to target objects without requiring users to first annotate everything in their home. Most current approaches to object navigation do not test on real robots and rely solely on reconstructed scans of houses and their expensively labeled semantic 3D meshes. In this work, our goal is to build an agent that builds self-supervised models of the world via exploration, the same as a child might - thus we (1) eschew the expense of labeled 3D mesh and (2) enable self-supervised in-situ finetuning in the real world. We identify a strong source of self-supervision (Location Consistency - LocCon) that can train all components of an ObjectNav agent, using unannotated simulated houses. Our key insight is that embodied agents can leverage location consistency as a self-supervision signal - collecting images from different views/angles and applying contrastive learning. We show that our agent can perform competitively in the real world and simulation. Our results also indicate that supervised training with 3D mesh annotations causes models to learn simulation artifacts, which are not transferrable to the real world. In contrast, our LocCon shows the most robust transfer in the real world among the set of models we compare to, and that the real-world performance of all models can be further improved with self-supervised LocCon in-situ training.
Is Weakly-Supervised Action Segmentation Ready for Human-Robot Interaction? No, Let's Improve It with Action-Union Learning

Action segmentation plays an important role in enabling robots to automatically understand human activities. To train the action recognition model, while obtaining action labels for all frames is costly, annotating timestamp labels for weak supervision is cost-effective. However, existing methods may not fully utilize timestamp labels, which leads to insufficient performance. To alleviate this issue, we proposed a novel learning pattern in our training stage, which maximizes the probability of action union of surrounding timestamps for unlabeled frames. In our inference stage, we provided a new refinement solution to generate better hard-assigned action classes from soft-assigned predictions. Importantly, our methods are model-agnostic and can be applied to existing frameworks. On three commonly used action-segmentation data, our method outperforms previous timestamp-supervision methods and achieves new state-of-the-art performance. More-over, our method uses less than 1% of fully-supervised labels to obtain comparable or even better results.
DAMON: Dynamic Amorphous Obstacle Navigation using Topological Manifold Learning and Variational Autoencoding

DAMON leverages manifold learning and variational autoencoding to achieve obstacle avoidance, allowing for motion planning through adaptive graph traversal in a pre-learned low-dimensional hierarchically-structured manifold graph that captures intricate motion dynamics between a robotic arm and its obstacles. This versatile and reusable approach is applicable to various collaboration scenarios. The primary advantage of DAMON is its ability to embed information in a low-dimensional graph, eliminating the need for repeated computation required by current sampling-based methods. As a result, it offers faster and more efficient motion planning with significantly lower computational overhead and memory footprint. In summary, DAMON is a breakthrough methodology that addresses the challenge of dynamic obstacle avoidance in robotic systems and offers a promising solution for safe and efficient human-robot collaboration. Our approach has been experimentally validated on a 7-DoF robotic manipulator in both simulation and physical settings. DAMON enables the robot to learn and generate skills for avoiding previously-unseen obstacles while achieving predefined objectives. We also optimize DAMON's design parameters and performance using an analytical framework. Our approach outperforms mainstream methodologies, including RRT, RRT*, Dynamic RRT*, L2RRT, and MpNet, with 40% more trajectory smoothness and over 65% improved latency performance, on average.
Predicting Energy Consumption and Traversal Time of Ground Robots for Outdoor Navigation on Multiple Types of Terrain

The outdoor navigation capabilities of ground robots have improved significantly in recent years, opening up new potential applications in a variety of settings. Cost-based representations of the environment are frequently used in the path planning domain to obtain an optimized path based on various objectives, such as traversal time or energy consumption. However, obtaining such cost representations is still cumbersome, particularly in outdoor settings with diverse terrain types and slope angles. In this paper, we address this problem by using a data-driven approach to develop a cost representation for various outdoor terrain types that supports two optimization objectives, namely energy consumption and traversal time. We train a supervised machine learning model whose inputs consists of extracted environment data along a path and whose outputs are the predicted energy consumption and traversal time. The model is based on a ResNet neural network architecture and trained using field-recorded data. The error of the proposed method on different types of terrain is within 11% of the ground truth data. To show that it performs and generalizes better than currently existing approaches on various types of terrain, a comparison to a baseline method is made.
SemanticBEVFusion: Rethinking LiDAR-Camera Fusion in Unified Bird's-Eye View Representation for 3D Object Detection

LiDAR and cameras are two essential sensors for 3D object detection in autonomous driving. LiDAR provides accurate and reliable 3D geometry information while the camera provides rich texture with color. Despite the increasing popularity of fusing these two complementary sensors, the challenge remains in how to effectively fuse 3D LiDAR point cloud with 2D camera images. Recent methods focus on point-level fusion which paints the LiDAR point cloud with camera features in the perspective view or bird's-eye view (BEV)-level fusion which unifies multi-modality features in the BEV representation. In this paper, we rethink these previous fusion strategies and analyze their information loss and influences on geometric and semantic features. We present SemanticBEVFusion to deeply fuse camera features with LiDAR features in a unified BEV representation while maintaining per-modality strengths for 3D object detection. Our method achieves state-of-the-art performance on the large-scale nuScenes dataset, especially for challenging distant objects. The code will be made publicly available.
Viewpoint Push Planning for Mapping of Unknown Confined Spaces

Viewpoint planning is an important task in any application where objects or scenes need to be viewed from different angles to achieve sufficient coverage. The mapping of confined spaces such as shelves is an especially challenging task since objects occlude each other and the scene can only be observed from the front, posing limitations on the possible viewpoints. In this paper, we propose a deep reinforcement learning framework that generates promising views aiming at reducing the map entropy. Additionally, the pipeline extends standard viewpoint planning by predicting adequate minimally invasive push actions to uncover occluded objects and increase the visible space. Using a 2.5D occupancy height map as state representation that can be efficiently updated, our system decides whether to plan a new viewpoint or perform a push. To learn feasible pushes, we use a neural network to sample push candidates on the map based on training data provided by human experts. As simulated and real-world experimental results with a robotic arm show, our system is able to significantly increase the mapped space compared to different baselines, while the executed push actions highly benefit the viewpoint planner with only minor changes to the object configuration.
navlie: A Python Package for State Estimation on Lie Groups

The ability to rapidly test a variety of algorithms for an arbitrary state estimation task is valuable in the prototyping phase of navigation systems. Lie group theory is now mainstream in the robotics community, and hence estimation prototyping tools should allow state definitions that belong to manifolds. A new package, called navlie, provides a framework that allows a user to model a large class of problems by implementing a set of classes complying with a generic interface. Once accomplished, navlie provides a variety of on-manifold estimation algorithms that can run directly on these classes. The package also provides a built-in library of common models, as well as many useful utilities. The open-source project can be found at https://github.com/decargroup/navlie
Autonomous Exploration and Mapping for Mobile Robots via Cumulative Curriculum Reinforcement Learning

Deep reinforcement learning (DRL) has been widely applied in autonomous exploration and mapping tasks, but often struggles with the challenges of sampling efficiency, poor adaptability to unknown map sizes, and slow simulation speed. To speed up convergence, we combine curriculum learning (CL) with DRL, and first propose a Cumulative Curriculum Reinforcement Learning (CCRL) training framework to alleviate the issue of catastrophic forgetting faced by general CL. Besides, we present a novel state representation, which considers a local egocentric map and a global exploration map resized to the fixed dimension, so as to flexibly adapt to environments with various sizes and shapes. Additionally, for facilitating the fast training of DRL models, we develop a lightweight grid-based simulator, which can substantially accelerate simulation compared to popular robot simulation platforms such as Gazebo. Based on the customized simulator, comprehensive experiments have been conducted, and the results show that the CCRL framework not only mitigates the catastrophic forgetting problem, but also improves the sample efficiency and generalization of DRL models, compared to general CL as well as without a curriculum. Our code is available at https://github.com/BeamanLi/CCRL_Exploration.
A Dynamic Programming Algorithm for Grid-Based Formation Planning of Multiple Vehicles

A common operation in multirobot systems is to generate a motion plan for multiple robots such that the robots can move in formation to achieve some desired effects. For example, in autonomous parking lots, a group of vehicles can be asked to move to another location when they block another vehicle that needs to leave the parking lot. In this paper, we present a novel grid-based planning approach for motion planning that minimizes the makespan of moving multiple vehicles from one location to another in a safe manner. Unlike most existing multirobot planning algorithms, our algorithm uses dynamic programming to compute a nearly-optimal motion plan for a large group of vehicles in polynomial time with the help of a given set of intermediate vehicle patterns. Our experimental results show that our algorithm is much faster than an exact algorithm but does not increase the minimum makespans tremendously.
Addressing the Scale Shrinkage Problem in Learning-based Binocular Depth Estimation

Binocular depth estimation is a fundamental problem in computer vision. Learning-based models have achieved significant performance improvements on public datasets in recent years. Our study finds that the performance of the current state-of-the-art deep learning-based models deteriorates significantly in distant areas. We point out that these deep learning-based models suffer from a scale shrinkage problem. Specifically, the predicted depth value ratio to the ground truth decreases as depth increases. Such a phenomenon is not conducive to the path planning and navigation of intelligent agents in outdoor scenes. We analyze the reasons for the scale shrinkage problem and give a simple and effective method. Our method employs a two-stage fine-tuning strategy and appropriately fuses the predictions of the two-stage models. The method does not reduce the prediction accuracy in close areas and significantly improves the accuracy of the models in distant areas. On the KITTI stereo 2015 dataset, our method can reduce the absolute relative difference by about 6% and the root-mean-square error (RMSE) by about 10%.