Reviewerly/exHarmony huggingface dataset:
Learning Models of Adversarial Agent Behavior Under Partial Observability

The need for opponent modeling and tracking arises in several real-world scenarios, such as professional sports, video game design, and drug-trafficking interdiction. In this work, we present Graph based Adversarial Modeling with Mutual Information (GrAMMI) for modeling the behavior of an adversarial opponent agent. GrAMMI is a novel graph neural network (GNN) based approach that uses mutual information maximization as an auxiliary objective to predict the current and future states of an adversarial opponent with partial observability. To evaluate GrAMMI, we design two large-scale, pursuit-evasion domains inspired by real-world scenarios, where a team of heterogeneous agents is tasked with tracking and interdicting a single adversarial agent, and the adversarial agent must evade detection while achieving its own objectives. With the mutual information formulation, GrAMMI outperforms all baselines in both domains and achieves 31.68% higher log-likelihood on average for future adversarial state predictions across both domains.
Improved Inference of Human Intent by Combining Plan Recognition and Language Feedback

Conversational assistive robots can aid people, especially those with cognitive impairments, to accomplish various tasks such as cooking meals, performing exercises, or operating machines. However, to interact with people effectively, robots must recognize human plans and goals from noisy observations of human actions, even when the user acts sub-optimally. Previous works on Plan and Goal Recognition (PGR) as planning have used hierarchical task networks (HTN) to model the actor/human. However, these techniques are insufficient as they do not have user engagement via natural modes of interaction such as language. Moreover, they have no mechanisms to let users, especially those with cognitive impairments, know of a deviation from their original plan or about any sub-optimal actions taken towards their goal. We propose a novel framework for plan and goal recognition in partially observable domains—Dialogue for Goal Recognition (D4GR) enabling a robot to rectify its belief in human progress by asking clarification questions about noisy sensor data and sub-optimal human actions. We evaluate the performance of D4GR over two simulated domains—kitchen and blocks domain. With language feedback and the world state information in a hierarchical task model, we show that D4GR framework for the highest sensor noise performs 1% better than HTN in goal accuracy in both domains. For plan accuracy, D4GR outperforms by 4% in the kitchen domain and 2% in the blocks domain in comparison to HTN. The ALWAYS-ASK oracle outperforms our policy by 3% in goal recognition and 7% in plan recognition. D4GR does so by asking 68% fewer questions than an oracle baseline. We also demonstrate a real-world robot scenario in the kitchen domain, validating the improved plan and goal recognition of D4GR in a realistic setting.
CompUDA: Compositional Unsupervised Domain Adaptation for Semantic Segmentation Under Adverse Conditions

In autonomous driving, performing robust semantic segmentation under adverse weather conditions is a long-standing challenge. Imperfect camera observations under adverse conditions result in images with reduced visibility, which hinders label annotation and semantic scene understanding based on these images. A common solution is to adopt semantic segmentation models trained in a source domain with ground truth labels and perform unsupervised domain adaptation (UDA) from the source domain to an unlabeled target domain that has adverse conditions. Due to imperfect visual observations in the target domain, such adaptation needs special treatment to achieve good performance. In this paper, we propose a new compositional unsupervised domain adaptation (CompUDA) method that disentangles the domain gap based on multiple factors including style, visibility, and image quality. The domain gaps caused by these individual factors can then be addressed separately by introducing the intermediate domains. Specifically, 1) to address the style gap, we perform source-to-intermediate domain adaptation and generate pseudo-labels for self-training in the target domain; 2) to address the visibility gap, we perform a geometry-aligned normal-to-adverse image translation and introduce a synthetic domain; 3) finally, to address the image quality gap between the synthetic and target domain, we perform a synthetic-to-real adaptation based on the generated pseudo-labels. Our compositional unsupervised domain adaptation can be used in conjunction with a wide variety of semantic segmentation methods and result in significant performance improvement across datasets. The codes are available at https://github.com/zhengziqiang/CompUDA.
FISS+: Efficient and Focused Trajectory Generation and Refinement Using Fast Iterative Search and Sampling Strategy

Trajectory planning plays a crucial role in autonomous driving systems, as it is tasked to generate feasible trajectories under highly dynamic scenarios within the time constraint. This paper proposes a novel two-stage coarse-to-fine framework for efficient sampling-based trajectory planning. The proposed method is designed to iteratively generate new trajectory samples focused on the low-cost regions in the sampling space. Two trajectory exploration algorithms are well-designed for efficient search in discretized coarse global space and continuous fine local space, respectively. Experimental results on the first-of-its-kind planning benchmark tool CommonRoad show that our method significantly outperforms the baseline methods both in optimality and computational efficiency. Overall, our approach offers a promising solution for efficient and effective trajectory planning in more autonomous vehicle applications.
Autocomplete of 3D Motions for UAV Teleoperation

Tele-operating aerial vehicles without any automated assistance is challenging due to various limitations, especially for inexperienced users. Autocomplete addresses this problem by automatically identifying and completing the user's intended motion. Such a framework uses machine learning to recognize and classify human inputs as one of a set of motion primitives, and then, if the human operator accepts, synthesizes the motion in order to complete the desired motion. This has been shown to improve the performance of the system and reduce operator workload. Previous Autocomplete systems focused on different 2D motions (line, arc, sine,..). However, since most UAVs tasks are in a 3D world, this paper introduces 3D Autocomplete for 3D motions. Moreover, the proposed framework presents just-in-time prediction of the 3D motions by proposing a change point detection technique, which allows the framework to autonomously identify when to conduct a prediction. Also, it deals with variable motion sizes. Real time simulation results show that the proposed framework is capable of predicting the user intentions after change point detection.
IDA: Informed Domain Adaptive Semantic Segmentation

Mixup-based data augmentation has been validated to be a critical stage in the self-training framework for unsupervised domain adaptive semantic segmentation (UDASS), which aims to transfer knowledge from a well-annotated (source) domain to an unlabeled (target) domain. Existing self-training methods usually adopt the popular region-based mixup techniques with a random sampling strategy, which unfortunately ignores the dynamic evolution of different semantics across various domains as training proceeds. To improve the UDA-SS performance, we propose an Informed Domain Adaptation (IDA) model, a self-training framework that mixes the data based on class-level segmentation performance, which aims to emphasize small-region semantics during mixup. In our IDA model, the class-level performance is tracked by an expected confidence score (ECS). We then use a dynamic schedule to determine the mixing ratio for data in different domains. Extensive experimental results reveal that our proposed method is able to outperform the state-of-the-art UDA-SS method by a margin of 1.1 mIoU in the adaptation of GTA-V to Cityscapes and of 0.9 mIoU in the adaptation of SYNTHIA to Cityscapes. Code link: https://github.com/ArlenCHEN/IDA.git
STL: Surprisingly Tricky Logic (for System Validation)

Much of the recent work developing formal methods techniques to specify or learn the behavior of autonomous systems is predicated on a belief that formal specifications are interpretable and useful for humans when checking systems. Though frequently asserted, this assumption is rarely tested. We performed a human experiment <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$(\mathbf{N}=62)$</tex> with a mix of people who were and were not familiar with formal methods beforehand, asking them to validate whether a set of signal temporal logic (STL) constraints would keep an agent out of harm and allow it to complete a task in a gridworld capture-the-ftag setting. Validation accuracy was 45% <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\pm$</tex> 20% (mean <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\pm$</tex> standard deviation). The ground-truth validity of a specification, subjects' familiarity with formal methods, and subjects' level of education were found to be significant factors in determining validation correctness. Participants exhibited an affirmation bias, causing significantly increased accuracy on valid specifications, but significantly decreased accuracy on invalid specifications. Additionally, participants, particularly those familiar with formal methods, tended to be overconfident in their answers, and be similarly confident regardless of actual correctness. Our data do not support the belief that formal specifications are inherently human-interpretable to a meaningful degree for system validation. We recommend ergonomic improvements to data presentation and validation training, which should be tested before claims of interpretability make their way back into the formal methods literature.
CAR-DESPOT: Causally-Informed Online POMDP Planning for Robots in Confounded Environments

Robots operating in real-world environments must reason about possible outcomes of stochastic actions and make decisions based on partial observations of the true world state. A major challenge for making accurate and robust action predictions is the problem of confounding, which if left untreated can lead to prediction errors. The partially observable Markov decision process (POMDP) is a widely-used framework to model these stochastic and partially-observable decision-making problems. However, due to a lack of explicit causal semantics, POMDP planning methods are prone to confounding bias and thus in the presence of unobserved confounders may produce underperforming policies. This paper presents a novel causally-informed extension of “anytime regularized determinized sparse partially observable tree” (AR-DESPOT), a modern anytime online POMDP planner, using causal modelling and inference to eliminate errors caused by unmeasured confounder variables. We further propose a method to learn offline the partial parameterisation of the causal model for planning, from ground truth model data. We evaluate our methods on a toy problem with an unobserved confounder and show that the learned causal model is highly accurate, while our planning method is more robust to confounding and produces overall higher performing policies than AR-DESPOT.
Data-Based MHE for Agile Quadrotor Flight

This paper develops a data-based moving horizon estimation (MHE) method for agile quadrotors. Accurate state estimation of the system is paramount for precise trajectory control for agile quadrotors; however, the high level of aerodynamic forces experienced by the quadrotors during high-speed flights make this task extremely challenging. These complex turbulent effects are difficult to model and the unmodelled dynamics introduce inaccuracies in the state estimation. In this work, we propose a method to model these aerodynamic effects using Gaussian Processes which we integrate into the MHE to achieve efficient and accurate state estimation with minimal computational burden. Through extensive simulation and experimental studies, this method has demonstrated significant improvement in state estimation performance displaying superior robustness to poor state measurements.
Graph-Based View Motion Planning for Fruit Detection

Crop monitoring is crucial for maximizing agricultural productivity and efficiency. However, monitoring large and complex structures such as sweet pepper plants presents significant challenges, especially due to frequent occlusions of the fruits. Traditional next-best view planning can lead to unstructured and inefficient coverage of the crops. To address this, we propose a novel view motion planner that builds a graph network of viable view poses and trajectories between nearby poses, thereby considering robot motion constraints. The planner searches the graphs for view sequences with the highest accumulated information gain, allowing for efficient pepper plant monitoring while minimizing occlusions. The generated view poses aim at both sufficiently covering already detected and discovering new fruits. The graph and the corresponding best view pose sequence are computed with a limited horizon and are adaptively updated in fixed time intervals as the system gathers new information. We demonstrate the effectiveness of our approach through simulated and real-world experiments using a robotic arm equipped with an RGB-D camera and mounted on a trolley. As the experimental results show, our planner produces view pose sequences to systematically cover the crops and leads to increased fruit coverage when given a limited time in comparison to a state-of-the-art single next-best view planner.
A Hybrid-State Path Planner for ASV Formations with Full-Scale Experiments

We present a hybrid-state path planner for autonomous surface vehicles (ASVs) constrained by a min-imum turning radius. The work is motivated by the future Norwegian naval mine countermeasures (NMCM) concept, which includes mine sweeping operations with ASVs that operate alone or in a formation of two, with and without mine sweeping equipment attached. Our path-planning approach is a variant of hybrid A* search, with the goal of finding the shortest path for transit in an environment consisting of an a priori map, operator-defined avoid areas and static obstacles detected by the vehicle. The planner is capable of finding traversable paths for ASV formations and single vehicles using a simple geometric model based on an estimated minimum turning radius for a given configuration. The method is modular as it is uncoupled from the path-following control method, making it applicable to many different systems, including our ASV configurations for mine sweeping operations. We present results from path planning in simulation for a single vehicle and two vehicles in formation. The method has been validated through a full-scale experiment with one of our ASVs in our testing area for autonomous platforms.
Language Guided Robotic Grasping with Fine-Grained Instructions

Given a single RGB image and the attribute-rich language instructions, this paper investigates the novel problem of using Fine-grained instructions for the Language guided robotic Grasping (FLarG). This problem is made challenging by learning fine-grained language descriptions to ground target objects. Recent advances have been made in visually grounding the objects simply by several coarse attributes [1]. However, these methods have poor performance as they cannot well align the multi-modal features, and do not make the best of recent powerful large pre-trained vision and language models, e.g., CLIP. To this end, this paper proposes a FLarG pipeline including stages of CLIP-guided object localization, and 6-DoF category-level object pose estimation for grasping. Specially, we first take the CLIP-based segmentation model CRIS as the backbone and propose an end-to-end DyCRIS model that uses a novel dynamic mask strategy to well fuse the multi-level language and vision features. Then, the well-trained instance segmentation backbone Mask R-CNN is adopted to further improve the predicted mask of our DyCRIS. Finally, the target object pose is inferred for the robotics grasping by using the recent 6-DoF object pose estimation method. To validate our CLIP-enhanced pipeline, we also construct a validation dataset for our FLarG task and name it RefNOCS. Extensive results on RefNOCS have shown the utility and effectiveness of our proposed method. The project homepage is available at https://sunqiang85.github.ioIFLarG/.
LQR-Trees with Sampling Based Exploration of the State Space

This paper introduces an extension of the LQR-tree algorithm, which is a feedback-motion-planning algorithm for stabilizing a system of ordinary differential equations from a bounded set of initial conditions to a goal. The constructed policies are represented by a tree of exemplary system trajec-tories, so called demonstrations, and linear-quadratic regulator (LQR) feedback controllers. Consequently, the crucial component of any LQR-tree algorithm is a demonstrator that provides suitable demonstrations. In previous work, such a demonstrator was given by a local trajectory optimizer. However, these require appropriate initial guesses of solutions to provide valid results, which was pointed out, but largely unresolved in previous implementations. In this paper, we augment the LQR-tree algorithm with a randomized motion-planning procedure to discover new valid demonstration candidates to initialize the demonstrator in parts of state space not yet covered by the LQR-tree. In comparison to the previous versions of the LQR-tree algorithm, the resulting exploring LQR-tree algorithm reliably synthesizes feedback control laws for a far more general set of problems.
Adversarial Object Rearrangement in Constrained Environments with Heterogeneous Graph Neural Networks

Adversarial object rearrangement in the real world (e.g., previously unseen or oversized items in kitchens and stores) could benefit from understanding task scenes, which inherently entail heterogeneous components such as current objects, goal objects, and environmental constraints. The semantic relationships among these components are distinct from each other and crucial for multi-skilled robots to perform efficiently in everyday scenarios. We propose a hierarchical robotic manipulation system that learns the underlying relationships and maximizes the collaborative power of its diverse skills (e.g., PICK-PLACE, PUSH) for rearranging adversarial objects in constrained environments. The high-level coordinator employs a heterogeneous graph neural network (HetGNN), which reasons about the current objects, goal objects, and environmental constraints; the low-level 3D Convolutional Neural Network-based actors execute the action primitives. Our approach is trained entirely in simulation, and achieved an average success rate of 87.88% and a planning cost of 12.82 in real-world experiments, surpassing all baseline methods. Supplementary material is available at https://sites.google.com/umn.edu/versatile-rearrangement.
Bang-Bang Boosting of RRTs

This paper presents methods for dramatically improving the performance of sampling-based kinodynamic planners. The key component is a complete, exact steering method that produces a time-optimal trajectory between any states for a vector of synchronized double integrators. This method is applied in three ways: 1) to generate RRT edges that quickly solve the two-point boundary-value problems, 2) to produce a (quasi)metric for more accurate Voronoi bias in RRTs, and 3) to iteratively time-optimize a given collision-free trajectory. Experiments are performed for state spaces with up to 2000 dimensions, resulting in improved computed trajectories and orders of magnitude computation time improvements over using ordinary metrics and constant controls.
Subtask Aware End-to-End Learning for Visual Room Rearrangement

The goal of intelligent embodied agents is to learn how to explore within the environment, interact with objects, and understand the environment in order to achieve task objectives. There are two main approaches to training such agents: one is to train an action policy that performs the task goal through end-to-end learning, and the other is to construct a policy by implementing the necessary abilities according to the task goal in a modular manner. For complex and long-horizon tasks, such as visual room rearrangement, a modular approach that infers task sequence by identifying the causality of actions through prior knowledge shows higher performance. Based on this insight, we propose an Online Subtask Prediction Network (OSPNet) that determines the subtask to be performed at each moment based on the environment information and past subtask inference history to train an embodied agent for long-horizon tasks through an end-to-end manner, and also propose a Subtask Aware Policy Network (SAPNet) as the action policy that decides actions based on the reasoning of the OSPNet. We implement an embodied agent that performs visual room rearrangement using the proposed SAPNet and train it through imitation learning, demonstrating similar or better performance with much fewer training steps than previous works.
Leveraging Single-Goal Predictions to Improve the Efficiency of Multi-Goal Motion Planning with Dynamics

Multi-goal motion planning requires a robot to plan collision-free and dynamically-feasible motions to reach multiple goals, often in unstructured, obstacle-rich environments. This is challenging due to the complex dependencies between navigation and high-level reasoning, requiring the robot to explore a vast space of feasible motions and goal sequences. Our approach combines machine learning and Traveling Salesman Problem (TSP) solvers with sampling-based motion planning. Machine learning predicts distances and directions between locations, considering obstacles and robot dynamics, which the TSP solver uses to compute promising tours. Sampling-based motion planning expands a motion tree to follow the tours along the predicted directions. We demonstrate the effectiveness of our approach through experiments with vehicle and snake-like robot models operating in unstructured environments with multiple goals.
Mapless Urban Robot Navigation by Following Pedestrians

Navigating effectively and safely in unknown urban environments is a crucial ability for service robot applications such as last-mile package delivery. To reach the entrance of its target destination, the robot must make informed local and global path planning decisions. We present a mapless global planning strategy based on pedestrian following. Our method allows the robot to exploit natural routes taken by surrounding pedestrians to make informed and efficient path planning decisions for reaching its goal. The algorithm also includes a recovery system to assist the robot when insufficient progress is made (i.e. robot stuck in dead end). Once the robot is within the vicinity of the target building, a wall following behaviour is used to reach the entrance of the target building. Simulated experiments and a proof-of-concept demonstration on a real robot were shown to validate the approach.