OpenRaiser/Intern-Atlas huggingface dataset:
Constraint-based Task Specification and Trajectory Optimization for Sequential Manipulation
Venue: Conference IROS 2022
Year: 2022

To economically deploy robotic manipulators the programming and execution of robot motions must be swift. To this end, we propose a novel, constraint-based method to intuitively specify sequential manipulation tasks and to compute time-optimal robot motions for such a task specification. Our approach follows the ideas of constraint-based task specification by aiming for a minimal and object-centric task description that is largely independent of the underlying robot kinematics. We transform this task description into a non-linear optimization problem. By solving this problem we obtain a (locally) time-optimal robot motion, not just for a single motion, but for an entire manipulation sequence. We demonstrate the capabilities of our approach in a series of experiments involving five distinct robot models, including a highly redundant mobile manipulator.
Robust Contact State Estimation in Humanoid Walking Gaits
Venue: Conference IROS 2022
Year: 2022

In this article, we propose a deep learning framework that provides a unified approach to the problem of leg contact detection in humanoid robot walking gaits. Our formulation accomplishes to accurately and robustly estimate the contact state probability for each leg (i.e., stable or slip/no contact). The proposed framework employs solely proprioceptive sensing and although it relies on simulated ground-truth contact data for the classification process, we demonstrate that it generalizes across varying friction surfaces and different legged robotic platforms and, at the same time, is readily transferred from simulation to practice. The framework is quantitatively and qualitatively assessed in simulation via the use of ground-truth contact data and is contrasted against state of-the-art methods with an ATLAS, a NAO, and a TALOS humanoid robot. Furthermore, its efficacy is demonstrated in base estimation with a real TALOS humanoid. To reinforce further research endeavors, our implementation is offered as an open-source ROS/Python package, coined Legged Contact Detection (LCD).
Quantity over Quality: Training an AV Motion Planner with Large Scale Commodity Vision Data
Venue: Conference IROS 2022
Year: 2022

With the Autonomous Vehicle (AV) industry shifting towards machine-learned approaches for motion planning, the performance of self-driving systems is starting to rely heavily on large quantities of expert driving demonstrations. However, collecting this demonstration data typically involves expensive HD sensor suites (LiDAR + RADAR + cameras), which quickly becomes financially infeasible at the scales required. This motivates the use of commodity sensors like cameras for data collection, which are an order of magnitude cheaper than HD sensor suites, but offer lower fidelity. Leveraging these sensors for training an AV motion planner opens a financially viable path to observe the `long tail' of driving events. As our main contribution we show it is possible to train a high-performance motion planner using commodity vision data which outperforms planners trained on HD-sensor data for a fraction of the cost. To the best of our knowledge, we are the first to demonstrate this using real-world data. We compare the performance of the autonomy system on these two different sensor configurations, and show that we can compensate for the lower sensor fidelity by means of increased quantity: a planner trained on 100h of commodity vision data outperforms the one with 25h of expensive HD data. We also share the engineering challenges we had to tackle to make this work.
Multi-View Guided Multi-View Stereo
Venue: Conference IROS 2022
Year: 2022

This paper introduces a novel deep framework for dense 3D reconstruction from multiple image frames, leveraging a sparse set of depth measurements gathered jointly with image acquisition. Given a deep multi-view stereo network, our framework uses sparse depth hints to guide the neural network by modulating the plane-sweep cost volume built during the forward step, enabling us to infer constantly much more accurate depth maps. Moreover, since multiple viewpoints can provide additional depth measurements, we propose a multi-view guidance strategy that increases the density of the sparse points used to guide the network, thus leading to even more accurate results. We evaluate our Multi-View Guided framework within a variety of state-of-the-art deep multi-view stereo networks, demonstrating its effectiveness at improving the results achieved by each of them on BlendedMVG and DTU datasets.
Estimation of Soft Robotic Bladder Compression for Smart Helmets using IR Range Finding and Hall Effect Magnetic Sensing
Venue: Conference IROS 2022
Year: 2022

This research focuses on soft robotic bladders that are used to monitor and control the interaction between a user's head and the shell of a Smart Helmet. Compression of these bladders determines impact dissipation; hence the focus of this paper is sensing and estimation of bladder compression. An IR rangefinder-based solution is evaluated using regression techniques as well as a Neural Network to estimate bladder compression. A Hall-Effect (HE) magnetic sensing system is also examined where HE sensors embedded in the base of the bladder sense the position of a magnet in the top of the bladder. The paper presents the HE sensor array, signal processing of HE voltage data, and then a Neural Network (NN) for predicting bladder compression. Efficacy of different training data sets on NN performance is studied. Different NN configurations are examined to determine a configuration that provides accurate estimates with as few nodes as possible. Different bladder compression profiles are evaluated to characterize IR range finding and HE based techniques in application scenarios.
Efficient Extrinsic Calibration of Multi-Sensor 3D LiDAR Systems for Autonomous Vehicles using Static Objects Information
Venue: Conference IROS 2022
Year: 2022

For an autonomous vehicle, the ability to sense its surroundings and to build an overall representation of the environment by fusing different sensor data streams is fundamental. To this end, the poses of all sensors need to be accurately determined. Traditional calibration methods are based on: 1) using targets specifically designed for calibration purposes in controlled environments, 2) optimizing a quality metric of the point clouds collected while traversing an unknown but static environment, or 3) optimizing the match among per-sensor incremental motion observations along a motion path fulfilling special requirements. In real scenarios, however, the online applicability of these methods can be limited, as they are typically highly dynamic, contain degenerate paths, and require fast computations. In this paper, we propose an approach that tackles some of these challenges by formulating the calibration problem as a joint but structured optimization problem of all sensor calibrations that takes as input a summary of the point cloud information consisting of ground points and pole detections. We demonstrate the efficiency and quality of the results of the proposed approach in a set of experiments with LiDAR simulation and real data from an urban trip.
Colonoscopy Navigation using End-to-End Deep Visuomotor Control: A User Study
Venue: Conference IROS 2022
Year: 2022

Flexible endoscopes for colonoscopy present several limitations due to their inherent complexity, resulting in patient discomfort and lack of intuitiveness for clinicians. Robotic devices together with autonomous control represent a viable solution to reduce the workload of endoscopists and the training time while improving the overall procedure outcome. Prior works on autonomous endoscope control use heuristic policies that limit their generalisation to the unstructured and highly deformable colon environment and require frequent human intervention. This work proposes an image-based control of the endoscope using Deep Reinforcement Learning, called Deep Visuomotor Control (DVC), to exhibit adaptive behaviour in convoluted sections of the colon tract. DVC learns a mapping between the endoscopic images and the control signal of the endoscope. A first user study of 20 expert gastrointestinal endoscopists was carried out to compare their navigation performance with DVC policies using a realistic virtual simulator. The results indicate that DVC shows equivalent performance on several assessment parameters, being more safer. Moreover, a second user study with 20 novice participants was performed to demonstrate easier human supervision compared to a state-of-the-art heuristic control policy. Seamless supervision of colonoscopy procedures would enable interventionists to focus on the medical decision rather than on the control problem of the endoscope.
BoxGraph: Semantic Place Recognition and Pose Estimation from 3D LiDAR
Venue: Conference IROS 2022
Year: 2022

This paper is about extremely robust and lightweight localisation using LiDAR point clouds based on instance segmentation and graph matching. We model 3D point clouds as fully-connected graphs of semantically identified components where each vertex corresponds to an object instance and encodes its shape. Optimal vertex association across graphs allows for full 6-Degree-of-Freedom (DoF) pose estimation and place recognition by measuring similarity. This representation is very concise, condensing the size of maps by a factor of 25 against the state-of-the-art, requiring only 3kB to represent a 1.4MB laser scan. We verify the efficacy of our system on the SemanticKITTI dataset, where we achieve a new state-of-the-art in place recognition, with an average of 88.4% recall at 100% precision where the next closest competitor follows with 64.9%. We also show accurate metric pose estimation performance - estimating 6-DoF pose with median errors of 10 cm and 0.33 deg.
Online 3D Bin Packing Reinforcement Learning Solution with Buffer
Venue: Conference IROS 2022
Year: 2022

The 3D Bin Packing Problem (3D-BPP) is one of the most demanded yet challenging problems in industry, where an agent must pack variable size items delivered in sequence into a finite bin with the aim to maximize the space utilization. It represents a strongly NP-Hard optimization problem such that no solution has been offered to date with high performance in space utilization. In this paper, we present a new reinforcement learning (RL) framework for a 3D-BPP solution for improving performance. First, a buffer is introduced to allow multi-item action selection. By increasing the degree of freedom in action selection, a more complex policy that results in better packing performance can be derived. Second, we propose an agnostic data augmentation strategy that exploits both bin item symmetries for improving sample efficiency. Third, we implement a model-based RL method adapted from the popular algorithm AlphaGo, which has shown superhuman performance in zero-sum games. Our adaptation is capable of working in single-player and score based environments. In spite of the fact that AlphaGo versions are known to be computationally heavy, we manage to train the proposed framework with a single thread and GPU, while obtaining a solution that outperforms the state-of-the-art results in space utilization.
UAV-miniUGV Hybrid System for Hidden Area Exploration and Manipulation
Venue: Conference IROS 2022
Year: 2022

We propose a novel hybrid system (both hardware and software) of an Unmanned Aerial Vehicle (UAV) carrying a miniature Unmanned Ground Vehicle (miniUGV) to perform a complex search and manipulation task. This system leverages heterogeneous robots to accomplish a task that cannot be done using a single robot system. It enables the UAV to explore a hidden space with a narrow opening through which the miniUGV can easily enter and escape. The hidden space is assumed to be navigable for the miniUGV. The miniUGV uses Infrared (IR) sensors and a monocular camera to search for an object in the hidden space. The proposed system takes advantage of a wider field of view (fov) of the camera as well as the stochastic nature of the object detection algorithms to guide the miniUGV in the hidden space to find the object. Upon finding the object the miniUGV grabs it using visual servoing and then returns back to its start point from where the UAV retracts it back and transports the object to a safe place. In case there is no object found in the hidden space, UAV continues the aerial search. The tethered miniUGV gives the UAV an ability to act beyond its reach and perform a search and manipulation task which was not possible before for any of the robots individually. The system has a wide range of applications and we have demonstrated its feasibility through repetitive experiments.
Quantifying Safety of Learning-based Self-Driving Control Using Almost-Barrier Functions
Venue: Conference IROS 2022
Year: 2022

Path-tracking control of self-driving vehicles can benefit from deep learning for tackling longstanding challenges such as nonlinearity and uncertainty. However, deep neural controllers lack safety guarantees, restricting their practical use. We propose a new approach of learning almost-barrier functions, which approximately characterizes the forward invariant set for the system under neural controllers, to quantitatively analyze the safety of deep neural controllers for path-tracking. We design sampling-based learning procedures for constructing candidate neural barrier functions, and certification procedures that utilize robustness analysis for neural networks to identify regions where the barrier conditions are fully satisfied. We use an adversarial training loop between learning and certification to optimize the almost-barrier functions. The learned barrier can also be used to construct online safety monitors through reachability analysis. We demonstrate effectiveness of our methods in quantifying safety of neural controllers in various simulation environments, ranging from simple kinematic models to the TORCS simulator with high-fidelity vehicle dynamics simulation.
DSOL: A Fast Direct Sparse Odometry Scheme
Venue: Conference IROS 2022
Year: 2022

In this paper, we describe Direct Sparse Odometry Lite (DSOL), an improved version of Direct Sparse Odometry (DSO). We propose several algorithmic and implementation enhancements which speed up computation by a significant factor (on average 5x) even on resource constrained platforms. The increase in speed allows us to process images at higher frame rates, which in turn provides better results on rapid motions. Our open-source implementation is available at https://github.com/versatran01/dsol.
Back to the Manifold: Recovering from Out-of-Distribution States
Venue: Conference IROS 2022
Year: 2022

Learning from previously collected datasets of expert data offers the promise of acquiring robotic policies without unsafe and costly online explorations. However, a major challenge is a distributional shift between the states in the training dataset and the ones visited by the learned policy at the test time. While prior works mainly studied the distribution shift caused by the policy during the offline training, the problem of recovering from out-of-distribution states at the deployment time is not very well studied yet. We alleviate the distributional shift at the deployment time by introducing a recovery policy that brings the agent back to the training manifold whenever it steps out of the in-distribution states, e.g., due to an external perturbation. The recovery policy relies on an approximation of the training data density and a learned equivariant mapping that maps visual observations into a latent space in which translations correspond to the robot actions. We demonstrate the effectiveness of the proposed method through several manipulation experiments on a real robotic platform. Our results show that the recovery policy enables the agent to complete tasks while the behavioral cloning alone fails because of the distributional shift problem.
Human-to-Robot Manipulability Domain Adaptation with Parallel Transport and Manifold-Aware ICP
Venue: Conference IROS 2022
Year: 2022

Manipulability ellipsoids efficiently capture the human pose and reveal information about the task at hand. Their use in task-dependent robot teaching - particularly their transfer from a teacher to a learner - can advance emulation of human-like motion. Although in recent literature focus is shifted towards manipulability transfer between two robots, the adaptation to the capabilities of the other kinematic system is to date not addressed and research in transfer from human to robot is still in its infancy. This work presents a novel manipulability domain adaptation method for the transfer of manipulability information to the domain of another kinematic system. As manipulability matrices/ellipsoids are symmetric positive-definite (SPD) they can be viewed as points on the Riemannian manifold of SPD matrices. We are the first to address the problem of manipulability transfer from the perspective of point cloud registration. We propose a manifold-aware Iterative Closest Point algorithm (ICP) with parallel transport initialization. Furthermore, we introduce a correspondence matching heuristic for manipulability ellipsoids based on inherent geometric features. We confirm our method in simulation experiments with 2-DoF manipulators as well as 7-DoF models representing the human-arm kinematics.
Visual-Inertial Multi-Instance Dynamic SLAM with Object-level Relocalisation
Venue: Conference IROS 2022
Year: 2022

In this paper, we present a tightly-coupled visual-inertial object-level multi-instance dynamic SLAM system. Even in extremely dynamic scenes, it can robustly optimise for the camera pose, velocity, IMU biases and build a dense 3D reconstruction object-level map of the environment. Our system can robustly track and reconstruct the geometries of arbitrary objects, their semantics and motion by incrementally fusing associated colour, depth, semantic, and foreground object probabilities into each object model thanks to its robust sensor and object tracking. In addition, when an object is lost or moved outside the camera field of view, our system can reliably recover its pose upon re-observation. We demonstrate the robustness and accuracy of our method by quantitatively and qualitatively testing it in real-world data sequences.
Rearrangement-Based Manipulation via Kinodynamic Planning and Dynamic Planning Horizons
Venue: Conference IROS 2022
Year: 2022

Robot manipulation in cluttered environments often requires complex and sequential rearrangement of multiple objects in order to achieve the desired reconfiguration of the target objects. Due to the sophisticated physical interactions involved in such scenarios, rearrangement-based manipulation is still limited to a small range of tasks and is especially vulnerable to physical uncertainties and perception noise. This paper presents a planning framework that leverages the efficiency of sampling-based planning approaches, and closes the manipulation loop by dynamically controlling the planning horizon. Our approach interleaves planning and execution to progressively approach the manipulation goal while correcting any errors or path deviations along the process. Meanwhile, our framework allows the definition of manipulation goals without requiring explicit goal configurations, enabling the robot to flexibly interact with all objects to facilitate the manipulation of the target ones. With extensive experiments both in simulation and on a real robot, we evaluate our framework on three manipulation tasks in cluttered environments: grasping, relocating, and sorting. In comparison with two baseline approaches, we show that our framework can significantly improve planning efficiency, robustness against physical uncertainties, and task success rate under limited time budgets.
Bubble Planner: Planning High-speed Smooth Quadrotor Trajectories using Receding Corridors
Venue: Conference IROS 2022
Year: 2022

Quadrotors are agile platforms. With human experts, they can perform extremely high-speed flights in cluttered environments. However, fully autonomous flight at high speed remains a significant challenge. In this work, we propose a motion planning algorithm based on the corridor-constrained minimum control effort trajectory optimization (MINCO) framework. Specifically, we use a series of overlapping spheres to represent the free space of the environment and propose two novel designs that enable the algorithm to plan high-speed quadrotor trajectories in real-time. One is a sampling-based corridor generation method that generates spheres with large overlapped areas (hence overall corridor size) between two neighboring spheres. The second is a Receding Horizon Corridors (RHC) strategy, where part of the previously generated corridor is reused in each replan. Together, these two designs enlarge the corridor spaces in accordance with the quadrotor's current state and hence allow the quadrotor to maneuver at high speeds. We benchmark our algorithm against other state-of-the-art planning methods to show its superiority in simulation. Comprehensive ablation studies are also conducted to show the necessity of the two designs. The proposed method is finally evaluated on an autonomous LiDAR-navigated quadrotor UAV in woods environments, achieving flight speeds over 13.7 m/s without any prior map of the environment or external localization facility.
A Whole-Body Controller Based on a Simplified Template for Rendering Impedances in Quadruped Manipulators
Venue: Conference IROS 2022
Year: 2022

Quadrupedal manipulators require to be compliant when dealing with external forces during autonomous manipulation, tele-operation or physical human-robot interaction. This paper presents a whole-body controller that allows for the implementation of a Cartesian impedance control to coordinate tracking performance and desired compliance for the robot base and manipulator arm. The controller is formulated through an optimization problem using Quadratic Programming (QP) to impose a desired behavior for the system while satisfying friction cone constraints, unilateral force constraints, joint and torque limits. The presented strategy decouples the arm and the base of the platform, enforcing the behavior of a linear double-mass spring damper system, and allows to independently tune their inertia, stiffness and damping properties. The control architecture is validated through an extensive simulation study using the 90kg HyQ robot equipped with a 7-DoF manipulator arm. Simulation results show the impedance rendering performance when external forces are applied at the arm's end-effector. The paper presents results for full stance condition (all legs on the ground) and, for the first time, also shows how the impedance rendering is affected by the contact conditions during a dynamic gait.
Tracking monocular camera pose and deformation for SLAM inside the human body
Venue: Conference IROS 2022
Year: 2022

Monocular SLAM in deformable scenes will open the way to multiple medical applications like computer-assisted navigation in endoscopy, automatic drug delivery or autonomous robotic surgery. In this paper we propose a novel method to simultaneously track the camera pose and the 3D scene deformation, without any assumption about environment topology or shape. The method uses an illumination-invariant photometric method to track image features and estimates camera motion and deformation combining reprojection error with spatial and temporal regularization of deformations. Our results in simulated colonoscopies show the method's accuracy and robustness in complex scenes under increasing levels of deformation. Our qualitative results in human colonoscopies from Endomapper dataset show that the method is able to successfully cope with the challenges of real endoscopies: deformations, low texture and strong illumination changes. We also compare with previous tracking methods in simpler scenarios from Hamlyn dataset where we obtain competitive performance, without needing any topological assumption.