OpenRaiser/Intern-Atlas huggingface dataset:
Learning and Leveraging Features in Flow-Like Environments to Improve Situational Awareness
Venue: arXiv cs.RO 2022
Year: 2022

This paper studies how global dynamics and knowledge of high-level features can inform decision-making for robots in flow-like environments. Specifically, we investigate how coherent sets, an environmental feature found in these environments, inform robot awareness within these scenarios. The proposed approach is an online environmental feature generator which can be used for robot reasoning. We compute coherent sets online with techniques from machine learning and design frameworks for robot behavior that leverage coherent set features. We demonstrate the effectiveness of online methods over offline methods. Notably, we apply these online methods for robot monitoring of pedestrian behaviors and robot navigation through water. Environmental features such as coherent sets provide rich context to robots for smarter, more efficient behavior.
SPARQLing Database Queries from Intermediate Question Decompositions
Venue: arXiv cs.CL 2022
Year: 2022

To translate natural language questions into executable database queries, most approaches rely on a fully annotated training set. Annotating a large dataset with queries is difficult as it requires query-language expertise. We reduce this burden using grounded in databases intermediate question representations. These representations are simpler to collect and were originally crowdsourced within the Break dataset (Wolfson et al., 2020). Our pipeline consists of two parts: a neural semantic parser that converts natural language questions into the intermediate representations and a non-trainable transpiler to the SPARQL query language (a standard language for accessing knowledge graphs and semantic web). We chose SPARQL because its queries are structurally closer to our intermediate representations (compared to SQL). We observe that the execution accuracy of queries constructed by our model on the challenging Spider dataset is comparable with the state-of-the-art text-to-SQL methods trained with annotated SQL queries. Our code and data are publicly available (see https://github.com/yandex-research/sparqling-queries).
STraTA: Self-Training with Task Augmentation for Better Few-shot Learning
Venue: arXiv cs.CL 2022
Year: 2022

Despite their recent successes in tackling many NLP tasks, large-scale pre-trained language models do not perform as well in few-shot settings where only a handful of training examples are available. To address this shortcoming, we propose STraTA, which stands for Self-Training with Task Augmentation, an approach that builds on two key ideas for effective leverage of unlabeled data. First, STraTA uses task augmentation, a novel technique that synthesizes a large amount of data for auxiliary-task fine-tuning from target-task unlabeled texts. Second, STraTA performs self-training by further fine-tuning the strong base model created by task augmentation on a broad distribution of pseudo-labeled data. Our experiments demonstrate that STraTA can substantially improve sample efficiency across 12 few-shot benchmarks. Remarkably, on the SST-2 sentiment dataset, STraTA, with only 8 training examples per class, achieves comparable results to standard fine-tuning with 67K training examples. Our analyses reveal that task augmentation and self-training are both complementary and independently effective.
Learning-to-defer for sequential medical decision-making under uncertainty
Venue: arXiv cs.LG 2022
Year: 2022

Learning-to-defer is a framework to automatically defer decision-making to a human expert when ML-based decisions are deemed unreliable. Existing learning-to-defer frameworks are not designed for sequential settings. That is, they defer at every instance independently, based on immediate predictions, while ignoring the potential long-term impact of these interventions. As a result, existing frameworks are myopic. Further, they do not defer adaptively, which is crucial when human interventions are costly. In this work, we propose Sequential Learning-to-Defer (SLTD), a framework for learning-to-defer to a domain expert in sequential decision-making settings. Contrary to existing literature, we pose the problem of learning-to-defer as model-based reinforcement learning (RL) to i) account for long-term consequences of ML-based actions using RL and ii) adaptively defer based on the dynamics (model-based). Our proposed framework determines whether to defer (at each time step) by quantifying whether a deferral now will improve the value compared to delaying deferral to the next time step. To quantify the improvement, we account for potential future deferrals. As a result, we learn a pre-emptive deferral policy (i.e. a policy that defers early if using the ML-based policy could worsen long-term outcomes). Our deferral policy is adaptive to the non-stationarity in the dynamics. We demonstrate that adaptive deferral via SLTD provides an improved trade-off between long-term outcomes and deferral frequency on synthetic, semi-synthetic, and real-world data with non-stationary dynamics. Finally, we interpret the deferral decision by decomposing the propagated (long-term) uncertainty around the outcome, to justify the deferral decision.
safe-control-gym: a Unified Benchmark Suite for Safe Learning-based Control and Reinforcement Learning in Robotics
Venue: arXiv cs.RO 2022
Year: 2022

In recent years, both reinforcement learning and learning-based control -- as well as the study of their safety, which is crucial for deployment in real-world robots -- have gained significant traction. However, to adequately gauge the progress and applicability of new results, we need the tools to equitably compare the approaches proposed by the controls and reinforcement learning communities. Here, we propose a new open-source benchmark suite, called safe-control-gym, supporting both model-based and data-based control techniques. We provide implementations for three dynamic systems -- the cart-pole, the 1D, and 2D quadrotor -- and two control tasks -- stabilization and trajectory tracking. We propose to extend OpenAI's Gym API -- the de facto standard in reinforcement learning research -- with (i) the ability to specify (and query) symbolic dynamics and (ii) constraints, and (iii) (repeatably) inject simulated disturbances in the control inputs, state measurements, and inertial properties. To demonstrate our proposal and in an attempt to bring research communities closer together, we show how to use safe-control-gym to quantitatively compare the control performance, data efficiency, and safety of multiple approaches from the fields of traditional control, learning-based control, and reinforcement learning.
Neural Networks with Physics-Informed Architectures and Constraints for Dynamical Systems Modeling
Venue: arXiv cs.LG 2022
Year: 2022

Effective inclusion of physics-based knowledge into deep neural network models of dynamical systems can greatly improve data efficiency and generalization. Such a-priori knowledge might arise from physical principles (e.g., conservation laws) or from the system's design (e.g., the Jacobian matrix of a robot), even if large portions of the system dynamics remain unknown. We develop a framework to learn dynamics models from trajectory data while incorporating a-priori system knowledge as inductive bias. More specifically, the proposed framework uses physics-based side information to inform the structure of the neural network itself, and to place constraints on the values of the outputs and the internal states of the model. It represents the system's vector field as a composition of known and unknown functions, the latter of which are parametrized by neural networks. The physics-informed constraints are enforced via the augmented Lagrangian method during the model's training. We experimentally demonstrate the benefits of the proposed approach on a variety of dynamical systems -- including a benchmark suite of robotics environments featuring large state spaces, non-linear dynamics, external forces, contact forces, and control inputs. By exploiting a-priori system knowledge during training, the proposed approach learns to predict the system dynamics two orders of magnitude more accurately than a baseline approach that does not include prior knowledge, given the same training dataset.
Robust Inverse Framework using Knowledge-guided Self-Supervised Learning: An application to Hydrology
Venue: arXiv cs.LG 2022
Year: 2022

Machine Learning is beginning to provide state-of-the-art performance in a range of environmental applications such as streamflow prediction in a hydrologic basin. However, building accurate broad-scale models for streamflow remains challenging in practice due to the variability in the dominant hydrologic processes, which are best captured by sets of process-related basin characteristics. Existing basin characteristics suffer from noise and uncertainty, among many other things, which adversely impact model performance. To tackle the above challenges, in this paper, we propose a novel Knowledge-guided Self-Supervised Learning (KGSSL) inverse framework to extract system characteristics from driver and response data. This first-of-its-kind framework achieves robust performance even when characteristics are corrupted. We show that KGSSL achieves state-of-the-art results for streamflow modeling for CAMELS (Catchment Attributes and MEteorology for Large-sample Studies) which is a widely used hydrology benchmark dataset. Specifically, KGSSL outperforms other methods by up to 16 \% in reconstructing characteristics. Furthermore, we show that KGSSL is relatively more robust to distortion than baseline methods, and outperforms the baseline model by 35\% when plugging in KGSSL inferred characteristics.
Structure-Enhanced Pop Music Generation via Harmony-Aware Learning
Venue: arXiv cs.SD 2022
Year: 2022

Pop music generation has always been an attractive topic for both musicians and scientists for a long time. However, automatically composing pop music with a satisfactory structure is still a challenging issue. In this paper, we propose to leverage harmony-aware learning for structure-enhanced pop music generation. On the one hand, one of the participants of harmony, chord, represents the harmonic set of multiple notes, which is integrated closely with the spatial structure of music, the texture. On the other hand, the other participant of harmony, chord progression, usually accompanies the development of the music, which promotes the temporal structure of music, the form. Moreover, when chords evolve into chord progression, the texture and form can be bridged by the harmony naturally, which contributes to the joint learning of the two structures. Furthermore, we propose the Harmony-Aware Hierarchical Music Transformer (HAT), which can exploit the structure adaptively from the music, and make the musical tokens interact hierarchically to enhance the structure in multi-level musical elements. Experimental results reveal that compared to the existing methods, HAT owns a much better understanding of the structure and it can also improve the quality of generated music, especially in the form and texture.
Spiking Neural Networks for Visual Place Recognition via Weighted Neuronal Assignments
Venue: arXiv cs.CV 2022
Year: 2022

Spiking neural networks (SNNs) offer both compelling potential advantages, including energy efficiency and low latencies and challenges including the non-differentiable nature of event spikes. Much of the initial research in this area has converted deep neural networks to equivalent SNNs, but this conversion approach potentially negates some of the advantages of SNN-based approaches developed from scratch. One promising area for high-performance SNNs is template matching and image recognition. This research introduces the first high-performance SNN for the Visual Place Recognition (VPR) task: given a query image, the SNN has to find the closest match out of a list of reference images. At the core of this new system is a novel assignment scheme that implements a form of ambiguity-informed salience, by up-weighting single-place-encoding neurons and down-weighting "ambiguous" neurons that respond to multiple different reference places. In a range of experiments on the challenging Nordland, Oxford RobotCar, SPEDTest, Synthia, and St Lucia datasets, we show that our SNN achieves comparable VPR performance to state-of-the-art and classical techniques, and degrades gracefully in performance with an increasing number of reference places. Our results provide a significant milestone towards SNNs that can provide robust, energy-efficient, and low latency robot localization.
Formal Methods for Quantum Programs: A Survey
Venue: arXiv cs.PL 2022
Year: 2022

While recent progress in quantum hardware open the door for significant speedup in certain key areas (cryptography, biology, chemistry, optimization, machine learning, etc), quantum algorithms are still hard to implement right, and the validation of such quantum programs is achallenge. Moreover, importing the testing and debugging practices at use in classical programming is extremely difficult in the quantum case, due to the destructive aspect of quantum measurement. As an alternative strategy, formal methods are prone to play a decisive role in the emerging field of quantum software. Recent works initiate solutions for problems occurring at every stage of the development process: high-level program design, implementation, compilation, etc. We review the induced challenges for an efficient use of formal methods in quantum computing and the current most promising research directions.
Foreground Object Structure Transfer for Unsupervised Domain Adaptation
Venue: arXiv cs.CV 2022
Year: 2022

Unsupervised domain adaptation aims to train a classification model from the labeled source domain for the unlabeled target domain. Since the data distributions of the two domains are different, the model often performs poorly on the target domain. Existing methods align the feature distributions of the source and target domains and learn domain-invariant features to improve the performance of the model. However, the features are usually aligned as a whole, and the domain adaptation task fails to serve the classification, which will ignore the class information and lead to misalignment.In this paper, we investigate those features that should be used for domain alignment, introduce prior knowledge to extract foreground features to guide the domain adaptation task for classification tasks, and perform alignment in the local structure of objects. We propose a method called Foreground Object Structure Transfer(FOST). The key to FOST is the new clustering based condition, which combines the relative position relationship of foreground objects. Based on this conditions, FOST makes the data distribution of the same class more compact in geometry. In practice, since the label of the target domain is not available, we use the clustering information of the source domain to assign pseudo labels to the target domain samples, and then according to the source domain data prior knowledge guides those positive features to maximum the inter-class distance between different classes and mimimum the intra-class distance. Extensive experimental results on various benchmarks ($i.e.$ ImageCLEF-DA, Office-31, Office-Home, Visda-2017) under different domain adaptation settings prove that our FOST compares favorably against the existing state-of-the-art domain adaptation methods.
Optimizing the ecological connectivity of landscapes with generalized flow models and preprocessing
Venue: arXiv cs.DS 2022
Year: 2022

In this paper we consider the problem of optimizing the ecological connectivity of a landscape under a budget constraint by improving habitat areas and ecological corridors between them. We consider a formulation of this problem in terms of graphs in which vertices represent the habitat areas and arcs represent a probability of connection between two areas that depend on the quality of the respective corridor. We propose a new generalized flow model that improves existing models for this problem and an efficient preprocessing algorithm that reduces the size of the graphs on which generalized flows is computed. Reported numerical experiments highlight the benefice of these two contributions on computation times and show that larger problems can be solved using them. Our experiments also show that several variants of greedy algorithms perform relatively well on practical instances while they return arbitrary bad solutions in the worst case.
Investigation of condominium building collapse in Surfside, Florida: A video feature tracking approach
Venue: arXiv cs.CV 2022
Year: 2022

On June 24, 2021, a 12-story condominium building (Champlain Towers South) in Surfside, Florida partially collapsed, resulting in one of the deadliest building collapses in United States history with 98 people confirmed deceased. In this work, we analyze the collapse event using a video clip that is publicly available from social media. In our analysis, we apply computer vision algorithms to corroborate new information from the video clip that may not be readily interpreted by human eyes. By comparing the differential features against different video frames, our proposed method is used to quantify the falling structural components by mapping the directions and magnitudes of their movements. We demonstrate the potential of this video processing methodology in investigations of catastrophic structural failures and hope our approach may serve as a basis for further investigations into structure collapse events.
Multihop: Leveraging Complex Models to Learn Accurate Simple Models
Venue: arXiv cs.LG 2022
Year: 2022

Knowledge transfer from a complex high performing model to a simpler and potentially low performing one in order to enhance its performance has been of great interest over the last few years as it finds applications in important problems such as explainable artificial intelligence, model compression, robust model building and learning from small data. Known approaches to this problem (viz. Knowledge Distillation, Model compression, ProfWeight, etc.) typically transfer information directly (i.e. in a single/one hop) from the complex model to the chosen simple model through schemes that modify the target or reweight training examples on which the simple model is trained. In this paper, we propose a meta-approach where we transfer information from the complex model to the simple model by dynamically selecting and/or constructing a sequence of intermediate models of decreasing complexity that are less intricate than the original complex model. Our approach can transfer information between consecutive models in the sequence using any of the previously mentioned approaches as well as work in 1-hop fashion, thus generalizing these approaches. In the experiments on real data, we observe that we get consistent gains for different choices of models over 1-hop, which on average is more than 2\% and reaches up to 8\% in a particular case. We also empirically analyze conditions under which the multi-hop approach is likely to be beneficial over the traditional 1-hop approach, and report other interesting insights. To the best of our knowledge, this is the first work that proposes such a multi-hop approach to perform knowledge transfer given a single high performing complex model, making it in our opinion, an important methodological contribution.
Data Hunches: Incorporating Personal Knowledge into Visualizations
Venue: arXiv cs.HC 2022
Year: 2022

The trouble with data is that it frequently provides only an imperfect representation of a phenomenon of interest. Experts who are familiar with their datasets will often make implicit, mental corrections when analyzing a dataset, or will be cautious not to be over-confident in any findings if caveats are present. However, the implicit knowledge about the caveats of a dataset are typically not collected in a structured way, which is problematic especially when teams work together who might have knowledge about different aspects of a dataset. In this work, we define such analyst's knowledge about datasets as data hunches. We discuss the implications of data hunches and propose a set of techniques for recording and communicating data hunches through data visualization. Furthermore, we provide guidelines for designing visualizations that support recording and visualizing data hunches. We envision that data hunches will empower analysts to externalize their knowledge, facilitate collaboration and communication, and support the ability to learn from others' data hunches.
Efficient and Probabilistic Adaptive Voxel Mapping for Accurate Online LiDAR Odometry
Venue: arXiv cs.RO 2022
Year: 2022

This paper proposes an efficient and probabilistic adaptive voxel mapping method for LiDAR odometry. The map is a collection of voxels; each contains one plane (or edge) feature that enables the probabilistic representation of the environment and accurate registration of a new LiDAR scan. We further analyze the need for coarse-to-fine voxel mapping and then use a novel voxel map organized by a Hash table and octrees to build and update the map efficiently. We apply the proposed voxel map to an iterated extended Kalman filter and construct a maximum a posteriori probability problem for pose estimation. Experiments on the open KITTI dataset show the high accuracy and efficiency of our method compared to other state-of-the-art methods. Outdoor experiments on unstructured environments with non-repetitive scanning LiDARs further verify the adaptability of our mapping method to different environments and LiDAR scanning patterns. Our codes and dataset are open-sourced on Github
Complementary Feature Enhanced Network with Vision Transformer for Image Dehazing
Venue: arXiv cs.CV 2022
Year: 2022

Conventional CNNs-based dehazing models suffer from two essential issues: the dehazing framework (limited in interpretability) and the convolution layers (content-independent and ineffective to learn long-range dependency information). In this paper, firstly, we propose a new complementary feature enhanced framework, in which the complementary features are learned by several complementary subtasks and then together serve to boost the performance of the primary task. One of the prominent advantages of the new framework is that the purposively chosen complementary tasks can focus on learning weakly dependent complementary features, avoiding repetitive and ineffective learning of the networks. We design a new dehazing network based on such a framework. Specifically, we select the intrinsic image decomposition as the complementary tasks, where the reflectance and shading prediction subtasks are used to extract the color-wise and texture-wise complementary features. To effectively aggregate these complementary features, we propose a complementary features selection module (CFSM) to select the more useful features for image dehazing. Furthermore, we introduce a new version of vision transformer block, named Hybrid Local-Global Vision Transformer (HyLoG-ViT), and incorporate it within our dehazing networks. The HyLoG-ViT block consists of the local and the global vision transformer paths used to capture local and global dependencies. As a result, the HyLoG-ViT introduces locality in the networks and captures the global and long-range dependencies. Extensive experiments on homogeneous, non-homogeneous, and nighttime dehazing tasks reveal that the proposed dehazing network can achieve comparable or even better performance than CNNs-based dehazing models.