Reviewerly/exHarmony huggingface dataset:
A MySQL Database for the Systematic Configuration Selection of Redundant Manipulators when Path Planning in Confined Spaces

Redundant manipulators offer a continuum of joint configurations which satisfy a specific end-effector pose, an advantage when operating within confined spaces. This, how-ever, challenges a controller to select a single goal configuration from a wide range when path planning. This paper outlines the use of the MySQL database management system for systematic goal selection during redundant manipulator path planning in confined spaces. We outline a sampling method to envelope all configurations of a redundant manipulator and utilise it to generate a complete database of configurations. We demonstrate the application of this method to generate a large data-set of (1 billion) manipulator configurations for a KUKA LBR iiwa 14 equipped with a Robotiq 2F-85 gripper. With this database, the controller systematically selects goal configurations during 50 path planning scenarios within the confined space of a glovebox. We compare this to an iterative method using existing kinematic solvers to select goal configurations as a baseline. The database method achieves a 100% success rate in 42% of the scenarios attempted. In comparison, the baseline method achieves >50% success rate in just 6% of the scenarios attempted. Our proposed method also produces repeatable paths, which are similar in length and link swept area for each attempt of the same scenario, whereas the baseline method generates a different path in every attempt.
ResiPlan: Closing the Planning-Acting Loop for Safe Underwater Navigation

Autonomous operation in underwater environ-ments is, arguably, one of the most complex domains. It requires safe operations under the presence of unpredictable surge, currents, uncertainty, and dynamic obstacles that challenges to the highest degree real-time motion planning; the primary focus of this paper. Although previous work addressed the problem of safe real-time 3D navigation in cluttered underwater environments, it did not account explicitly for disturbances, currents, dynamic obstacles, or uncertainty growth. This paper presents ResiPlan, a novel motion planning framework that utilizes past information of errors monitoring the path follower's performance, along with estimation of dynamic obstacles and uncertainty, to produce adaptive paths by adjusting the safety margins accordingly. Extensive numerical experiments and simulations validate the safety guarantees of the technique, in a variety of different environments with various types of disturbance, showcasing the strong potential to be utilized for operations in challenging underwater environments.
SoLo T-DIRL: Socially-Aware Dynamic Local Planner based on Trajectory-Ranked Deep Inverse Reinforcement Learning

This work proposes a novel framework for socially-aware robot navigation in dynamic, crowded environments using a Deep Inverse Reinforcement Learning. To address the social navigation problem, our multi-modal learning based planner explicitly considers social interaction factors, as well as social-awareness factors, into the DIRL pipeline to learn a reward function from human demonstrations. Moreover, we propose a novel trajectory ranking score using the sudden velocity change of pedestrians around the robot to address the sub-optimality in human demonstrations. Our evaluation shows that this method can successfully make a robot navigate in a crowded social environment and outperforms the state-of-art social navigation methods in terms of the success rate, navigation time, and invasion rate.
Small-shot Multi-modal Distillation for Vision-based Autonomous Steering

In this paper, we propose a novel learning framework for autonomous systems that uses a small amount of “auxiliary information” that complements the learning of the main modality, called “small-shot auxiliary modality distillation network (AMD-S-Net)”. The AMD-S-Net contains a two-stream framework design that can fully extract information from different types of data (i.e., paired/unpaired multi-modality data) to distill knowledge more effectively. We also propose a novel training paradigm based on the “reset operation” that enables the teacher to explore the local loss landscape near the student domain iteratively, providing local landscape information and potential directions to discover better solutions by the student, thus achieving higher learning performance. Our experiments show that AMD-S-Net and our training paradigm outperform other SOTA methods by up to 12.7% and 18.1% improvement in autonomous steering, respectively.
TODE-Trans: Transparent Object Depth Estimation with Transformer

Transparent objects are widely used in industrial automation and daily life. However, robust visual recognition and perception of transparent objects have always been a major challenge. Currently, most commercial-grade depth cameras are still not good at sensing the surfaces of transparent objects due to the refraction and reflection of light. In this work, we present a transformer-based transparent object depth estimation approach from a single RGB-D input. We observe that the global characteristics of the transformer make it easier to extract contextual information to perform depth estimation of transparent areas. In addition, to better enhance the fine-grained features, a feature fusion module (FFM) is designed to assist coherent prediction. Our empirical evidence demonstrates that our model delivers significant improvements in recent popular datasets, e.g., 25% gain on RMSE and 21% gain on REL compared to previous state-of-the-art convolutional-based counterparts in ClearGrasp dataset. Extensive results show that our transformer-based model enables better aggregation of the object's RGB and inaccurate depth information to obtain a better depth representation. Our code and the pre-trained model are available at https://github.com/yuchendoudou/TODE.
Low-level controller in response to changes in quadrotor dynamics

The dynamics of all real quadrotors inevitably differ even if they are the same product. In particular, the dynamics can change significantly during the flight due to additional device attachments or overheating motors. In this study, we focus on training a low-level controller, which operates in response to dynamics-changes without prior knowledge or fine-tuning of the parameters, using reinforcement learning. We randomize the dynamics of quadrotors in the simulator and train the policy based on dynamics information extracted from the state-action history through recurrent neural networks (RNNs). In addition, our experiment demonstrates the difficulties in applying existing actor-critic structures that extract dynamics information using end-to-end RNNs for unstable quadrotors; hence, we propose a novel structure with better performance. Finally, the excellent performance of the proposed controller is verified by testing experiments that stabilize quadrotors with different dynamics. The experiment videos and the code can be found at https://github.com/jackyoung96/RNN-Quadrotor-controller.
Robot explanatory narratives of collaborative and adaptive experiences

In the future, robots are expected to autonomously interact and/or collaborate with humans, who will increase the uncertainty during the execution of tasks, provoking online adaptations of robots' plans. Hence, trustworthy robots must be able to store, retrieve and narrate important knowledge about their collaborations and adaptations. In this article, it is proposed a sound methodology that integrates three main elements. First, an ontology for collaborative robotics and adaptation to model the domain knowledge. Second, an episodic memory for time-indexed knowledge storage and retrieval. Third, a novel algorithm to extract the relevant knowledge and generate textual explanatory narratives. The algorithm produces three different types of outputs, varying the specificity, for diverse uses and preferences. A pilot study was conducted to evaluate the usefulness of the narratives, obtaining promising results. Finally, we discuss how the methodology can be generalized to other ontologies and experiences. This work boosts robot explainability, especially in cases where robots need to narrate the details of their short and long-term past experiences.
E-VFIA: Event-Based Video Frame Interpolation with Attention

Video frame interpolation (VFI) is a fundamental vision task that aims to synthesize several frames between two consecutive original video images. Most algorithms aim to accomplish VFI by using only keyframes, which is an ill-posed problem since the keyframes usually do not yield any accurate precision about the trajectories of the objects in the scene. On the other hand, event-based cameras provide more precise information between the keyframes of a video. Some recent state-of-the-art event-based methods approach this problem by utilizing event data for better optical flow estimation to interpolate for video frame by warping. Nonetheless, those methods heavily suffer from the ghosting effect. On the other hand, some of kernel-based VFI methods that only use frames as input, have shown that deformable convolutions, when backed up with transformers, can be a reliable way of dealing with long-range dependencies. We propose event-based video frame interpolation with attention (E-VFIA), as a lightweight kernelbased method. E-VFIA fuses event information with standard video frames by deformable convolutions to generate high quality interpolated frames. The proposed method represents events with high temporal resolution and uses a multi-head selfattention mechanism to better encode event-based information, while being less vulnerable to blurring and ghosting artifacts; thus, generating crispier frames. The simulation results show that the proposed technique outperforms current state-of-the-art methods (both frame and event-based) with a significantly smaller model size. Multimedia material: The code is available at https://github.com/ahmetakman/E-VFIA
A Contextual Bandit Approach for Learning to Plan in Environments with Probabilistic Goal Configurations

Object-goal navigation (Object-nav) entails searching, recognizing and navigating to a target object. Object-nav has been extensively studied by the Embodied-AI community, but most solutions are often restricted to considering static objects (e.g., television, fridge, etc.), We propose a modular framework for object-nav that is able to efficiently search indoor environments for not just static objects but also movable objects (e.g. fruits, glasses, phones, etc.) that frequently change their positions due to human intervention. Our contextual-bandit agent efficiently explores the environment by showing optimism in the face of uncertainty and learns a model of the likelihood of spotting different objects from each navigable location. The likelihoods are used as rewards in a weighted minimum latency solver to deduce a trajectory for the robot. We evaluate our algorithms in two simulated environments and a real-world setting, to demonstrate high sample efficiency and reliability.
Intermittent diffusion-based path planning for heterogeneous groups of mobile sensors in cluttered environments

This paper presents a method for task-oriented path planning and collision avoidance for a group of heterogeneous holonomic mobile sensors. It is a generalization of the authors' prior work on diffusion-based path planning. The proposed variant allows one to plan paths in environments cluttered with obstacles. The agents follow flow dynamics, i.e., the negative gradient of a function that is the sum of two functions: the first minimizes the distance from desired target regions and the second captures distance from other agents within a field of view. When it becomes necessary to steer around an obstacle, this function is augmented by a projection term that is carefully designed in terms of obstacle boundaries. More importantly, a diffusion term is added intermittently so that agents can exit local minima. In addition, the new approach skips the offline planning phase in the prior approach to improve computational performance and handle collision avoidance with a completely decentralized method. This approach also provably finds collision-free paths under certain conditions. Numerical simulations of three deployment missions further support the performance of ID-based diffusion.
Safe and Efficient Navigation in Extreme Environments using Semantic Belief Graphs

To achieve autonomy in unknown and unstruc-tured environments, we propose a method for semantic-based planning under perceptual uncertainty. This capability is cru-cial for safe and efficient robot navigation in environment with mobility-stressing elements that require terrain-specific locomotion policies. We propose the Semantic Belief Graph (SBG), a geometric- and semantic-based representation of a robot's probabilistic roadmap in the environment. The SBG nodes comprise of the robot geometric state and the semantic-knowledge of the terrains in the environment. The SBG edges represent local semantic-based controllers that drive the robot between the nodes or invoke an information gathering action to reduce semantic belief uncertainty. We formulate a semantic-based planning problem on SBG that produces a policy for the robot to safely navigate to the target location with min-imal traversal time. We analyze our method in simulation and present real-world results with a legged robotic platform navigating multi-level outdoor environments.
Sample-Efficient Goal-Conditioned Reinforcement Learning via Predictive Information Bottleneck for Goal Representation Learning

We propose Predictive Information bottleneck for Goal representation learning (PI-Goal), a self-supervised method for sample-efficient goal-conditioned reinforcement learning (RL). Goal-conditioned RL learns to reach commanded goals with reward signals. A goal could be given in a noisy or abstract form, and thus jeopardizes sample efficiency. Previous methods usually assume that the agent can map a state to an achievable goal. In this work, we consider a setting in which the goal space is unknown to the agent and the agent cannot recognize a goal in a specific state (referred to as a goal state) until the goal is commanded. Our PI-Goal learns a goal representation which contains only the predictive information of a goal state, i.e., the mutual information between a current state and a future state, and guarantees the optimality of the learned policy. Experimental results show that PI-Goal consistently outperforms the baseline methods in tasks with unknown goal spaces, e.g., object manipulation, object search, and embodied question answering.
Robot Mimicry Attack on Keystroke-Dynamics User Identification and Authentication System

Future robots will be very advanced with high flexibility and accurate control performance. They will have the ability to mimic human behaviours or even perform better, which raises the significant risk of robot attack. In this work, we study the robot mimic attack on the current keystroke-dynamic user authentication system. Specifically, we proposed a robot mimicry attack framework for keystroke-dynamics systems. We collected keyboard logging data and acoustical signal data from real users and extracted the timing pattern of keystrokes to understand victim's behaviour for robot imitation attacks. Furthermore, we develop a deep Q-Network (DQN) algorithm to control the velocity of robot which is one of the key challenges of forging the human typing timing features. We tested and evaluated our approach on the real-life robotic testbed. We presented our results considering user identification and user authentication performance. We achieved a 90.3% user identification accuracy with genuine keyboard logging data samples and 89.6% accuracy with robot-forged data samples. Furthermore, we achieved 11.1%, and 36.6% EER for user authentication performance with zero-effort attack, and robot mimicry attack, respectively.
Unseen Object Instance Segmentation with Fully Test-time RGB-D Embeddings Adaptation

Segmenting unseen objects is a crucial ability for the robot since it may encounter new environments during the operation. Recently, a popular solution is leveraging RGB-D features of large-scale synthetic data and directly applying the model to unseen real-world scenarios. However, the domain shift caused by the sim2real gap is inevitable, posing a crucial challenge to the segmentation model. In this paper, we em-phasize the adaptation process across sim2real domains and model it as a learning problem on the BatchNorm param-eters of a simulation-trained model. Specifically, we propose a novel non-parametric entropy objective, which formulates the learning objective for the test-time adaptation in an open-world manner. Then, a cross-modality knowledge distillation objective is further designed to encourage the test-time knowledge transfer for feature enhancement. Our approach can be efficiently implemented with only test images, without requiring annotations or revisiting the large-scale synthetic training data. Besides significant time savings, the proposed method consistently improves segmentation results on the overlap and boundary metrics, achieving state-of-the-art performance on unseen object instance segmentation.
Robot-Assisted Eye-Hand Coordination Training System by Estimating Motion Direction Using Smooth-Pursuit Eye Movements

Robot-assisted eye-hand coordination rehabilitation training system is extremely urgent to study since recent evidence suggests that eye-hand coordination can be brutally disturbed by stroke with critical consequences on motor behavior. In this paper, we develop a robot-assisted eye-hand coordination training system by estimating motion direction using smooth-pursuit eye movements. Firstly, we design a Pong Game, which requires users to extrapolate the direction of a linearly moving ball and to predict whether this ball would be hit. Secondly, the motion direction of the ball is estimated via smooth-pursuit eye movements, allowing the robot quickly establish an assistive force field to hit the ball. Thirdly, adding haptic feedback technology into this training system to make users more immersive. Finally, we conduct a feasibility study with eight healthy subjects to verify the effectiveness of the proposed system. The experimental results show that the mean success rate for hitting the pong ball of the experiment group (assistance turn-on) is 28.33% higher than that of the control group (assistance turn-off), and the mean interception time of the experiment group is 0.35s shorter than that of the control group. Therefore, the developed system may be promising for transferring to the robot-assisted eye-hand coordination rehabilitation training for post-stroke patients.
Dynamic Interactive Relation Capturing via Scene Graph Learning for Robotic Surgical Report Generation

For robot-assisted surgery, an accurate surgical report reflects clinical operations during surgery and helps document entry tasks, post-operative analysis and follow-up treatment. It is a challenging task due to many complex and diverse interactions between instruments and tissues in the surgical scene. Although existing surgical report generation methods based on deep learning have achieved large success, they often ignore the interactive relation between tissues and instrumental tools, thereby degrading the report generation performance. This paper presents a neural network to boost surgical report generation by explicitly exploring the interactive relation between tissues and surgical instruments. To do so, we first devise a relational exploration (RE) module to model the interactive relation via graph learning, and an interaction perception (IP) module to assist the graph learning in RE module. In our IP module, we first devise a node tracking system to identify and append missing graph nodes of the current video frame for constructing graphs at RE module. Moreover, the IP module generates a global attention model to indicate the existence of the interactive relation on the whole scene of the current video frame to eliminate the graph learning at the current video frame. Furthermore, our IP module predicts a local attention model to more accurately identify the interaction relation of each graph node for assisting the graph updating at the RE module. After that, we concatenate features of all graph nodes of RE module and pass concatenated features into a transformer for generating the output surgical report. We validate the effectiveness of our method on a widely-used robotic surgery benchmark dataset, and experimental results show that our network can significantly outperform existing state-of-the-art surgical report generation methods (e.g., 7.48% and 5.43% higher for BLEU-1 and ROUGE).
Mapping Waves with an Uncrewed Surface Vessel via Gaussian Process Regression

Mobile robots are well suited for environmental surveys because they can travel to any area of interest and react to observations without the need for pre-existing infrastructure or significant setup time. However, vehicle motion constraints limit where and when measurements occur. This is challenging for a single vehicle observing a time-varying phenomenon, such as coastal waves, but the ability to generate a spatiotemporal map would have immediate scientific and engineering applications. In this paper, an uncrewed surface vessel (USV) was used to measure waves on the coast of Lake Ontario, Canada. Data were collected from a low-cost inertial measurement system onboard the USV and processed in an offline Gaussian process regression (GPR) workflow to create a spatiotemporal wave model. Frequency analysis of raw sensor data was used to best select and design kernel functions, and to initialize hyperparameters. The relative speed of the waves limited the ability to make complete wave reconstructions, but GPR captured the dominant periodic components of the waves despite irregularities in the signals. After optimization, the hyperparameters indicate a dominant signal with a wave period of 0.87 <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathbf{s}$</tex> , which concurs with ground truth estimates.
Stochastic Traveling Salesperson Problem with Neighborhoods for Object Detection

We introduce a new route-finding problem which considers perception and travel costs simultaneously. Specifically, we consider the problem of finding the shortest tour such that all objects of interest can be detected successfully. To represent a viable detection region for each object, we propose to use an entropy-based viewing score that generates a diameter-bounded region as a viewing neighborhood. We formulate the detection-based trajectory planning problem as a stochastic traveling salesperson problem with neighborhoods and propose a center-visit method that obtains an approximation ratio of <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$O(\frac{D_{max}}{D_{min}})$</tex> for disjoint regions. For non-disjoint regions, our method -provides a novel finite detour in 3D, which utilizes the region's minimum curvature property. Finally, we show that our method can generate efficient trajectories compared to a baseline method in a photo-realistic simulation environment.