OpenRaiser/Intern-Atlas huggingface dataset:
Towards Robust Semantic Segmentation of Accident Scenes via Multi-Source Mixed Sampling and Meta-Learning
Venue: arXiv cs.CV 2022
Year: 2022

Autonomous vehicles utilize urban scene segmentation to understand the real world like a human and react accordingly. Semantic segmentation of normal scenes has experienced a remarkable rise in accuracy on conventional benchmarks. However, a significant portion of real-life accidents features abnormal scenes, such as those with object deformations, overturns, and unexpected traffic behaviors. Since even small mis-segmentation of driving scenes can lead to serious threats to human lives, the robustness of such models in accident scenarios is an extremely important factor in ensuring safety of intelligent transportation systems. In this paper, we propose a Multi-source Meta-learning Unsupervised Domain Adaptation (MMUDA) framework, to improve the generalization of segmentation transformers to extreme accident scenes. In MMUDA, we make use of Multi-Domain Mixed Sampling to augment the images of multiple-source domains (normal scenes) with the target data appearances (abnormal scenes). To train our model, we intertwine and study a meta-learning strategy in the multi-source setting for robustifying the segmentation results. We further enhance the segmentation backbone (SegFormer) with a HybridASPP decoder design, featuring large window attention spatial pyramid pooling and strip pooling, to efficiently aggregate long-range contextual dependencies. Our approach achieves a mIoU score of 46.97% on the DADA-seg benchmark, surpassing the previous state-of-the-art model by more than 7.50%. Code will be made publicly available at https://github.com/xinyu-laura/MMUDA.
Mechanism, measurement, and quantification of stress in decision process: a model based systematic-review protocol
Venue: arXiv cs.HC 2022
Year: 2022

Every human action begins with decision-making. Stress is a significant source of biases that can influence human decision-making. In order to understand the relationship between stress and decision-making, stress quantification is fundamental. Different methods of measuring and quantifying stress in decision-making have been described in the literature while an up-to-date systematic review of the existing methods is lacking. Moreover, mental stress, mental effort, cognitive workload, and workload are often used interchangeably but should be distinguished to enable in-depth investigations of decision-mechanisms. Our objectives are to clarify stress related concepts and review the measurement, quantification, and application of stress during decision making activities.
CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigation
Venue: arXiv cs.CV 2022
Year: 2022

For robots to be generally useful, they must be able to find arbitrary objects described by people (i.e., be language-driven) even without expensive navigation training on in-domain data (i.e., perform zero-shot inference). We explore these capabilities in a unified setting: language-driven zero-shot object navigation (L-ZSON). Inspired by the recent success of open-vocabulary models for image classification, we investigate a straightforward framework, CLIP on Wheels (CoW), to adapt open-vocabulary models to this task without fine-tuning. To better evaluate L-ZSON, we introduce the Pasture benchmark, which considers finding uncommon objects, objects described by spatial and appearance attributes, and hidden objects described relative to visible objects. We conduct an in-depth empirical study by directly deploying 21 CoW baselines across Habitat, RoboTHOR, and Pasture. In total, we evaluate over 90k navigation episodes and find that (1) CoW baselines often struggle to leverage language descriptions, but are proficient at finding uncommon objects. (2) A simple CoW, with CLIP-based object localization and classical exploration -- and no additional training -- matches the navigation efficiency of a state-of-the-art ZSON method trained for 500M steps on Habitat MP3D data. This same CoW provides a 15.6 percentage point improvement in success over a state-of-the-art RoboTHOR ZSON model.
Smoothing Advantage Learning
Venue: arXiv cs.LG 2022
Year: 2022

Advantage learning (AL) aims to improve the robustness of value-based reinforcement learning against estimation errors with action-gap-based regularization. Unfortunately, the method tends to be unstable in the case of function approximation. In this paper, we propose a simple variant of AL, named smoothing advantage learning (SAL), to alleviate this problem. The key to our method is to replace the original Bellman Optimal operator in AL with a smooth one so as to obtain more reliable estimation of the temporal difference target. We give a detailed account of the resulting action gap and the performance bound for approximate SAL. Further theoretical analysis reveals that the proposed value smoothing technique not only helps to stabilize the training procedure of AL by controlling the trade-off between convergence rate and the upper bound of the approximation errors, but is beneficial to increase the action gap between the optimal and sub-optimal action value as well.
Over-parameterization: A Necessary Condition for Models that Extrapolate
Venue: arXiv cs.LG 2022
Year: 2022

In this work, we study over-parameterization as a necessary condition for having the ability for the models to extrapolate outside the convex hull of training set. We specifically, consider classification models, e.g., image classification and other applications of deep learning. Such models are classification functions that partition their domain and assign a class to each partition \cite{strang2019linear}. Partitions are defined by decision boundaries and so is the classification model/function. Convex hull of training set may occupy only a subset of the domain, but trained model may partition the entire domain and not just the convex hull of training set. This is important because many of the testing samples may be outside the convex hull of training set and the way in which a model partitions its domain outside the convex hull would be influential in its generalization. Using approximation theory, we prove that over-parameterization is a necessary condition for having control over the partitioning of the domain outside the convex hull of training set. We also propose a more clear definition for the notion of over-parametrization based on the learning task and the training set at hand. We present empirical evidence about geometry of datasets, both image and non-image, to provide insights about the extent of extrapolation performed by the models. We consider a 64-dimensional feature space learned by a ResNet model and investigate the geometric arrangements of convex hulls and decision boundaries in that space. We also formalize the notion of extrapolation and relate it to the scope of the model. Finally, we review the rich extrapolation literature in pure and applied mathematics, e.g., the Whitney's Extension Problem, and place our theory in that context.
Partitioning Image Representation in Contrastive Learning
Venue: arXiv cs.CV 2022
Year: 2022

In contrastive learning in the image domain, the anchor and positive samples are forced to have as close representations as possible. However, forcing the two samples to have the same representation could be misleading because the data augmentation techniques make the two samples different. In this paper, we introduce a new representation, partitioned representation, which can learn both common and unique features of the anchor and positive samples in contrastive learning. The partitioned representation consists of two parts: the content part and the style part. The content part represents common features of the class, and the style part represents the own features of each sample, which can lead to the representation of the data augmentation method. We can achieve the partitioned representation simply by decomposing a loss function of contrastive learning into two terms on the two separate representations, respectively. To evaluate our representation with two parts, we take two framework models: Variational AutoEncoder (VAE) and BootstrapYour Own Latent(BYOL) to show the separability of content and style, and to confirm the generalization ability in classification, respectively. Based on the experiments, we show that our approach can separate two types of information in the VAE framework and outperforms the conventional BYOL in linear separability and a few-shot learning task as downstream tasks.
An Adaptive and Scalable ANN-based Model-Order-Reduction Method for Large-Scale TO Designs
Venue: arXiv cs.LG 2022
Year: 2022

Topology Optimization (TO) provides a systematic approach for obtaining structure design with optimum performance of interest. However, the process requires numerical evaluation of objective function and constraints at each iteration, which is computational expensive especially for large-scale design. Deep learning-based models have been developed to accelerate the process either by acting as surrogate models replacing the simulation process, or completely replacing the optimization process. However, most of them require a large set of labelled training data, which are generated mostly through simulations. The data generation time scales rapidly with the design domain size, decreasing the efficiency of the method itself. Another major issue is the weak generalizability of most deep learning models. Most models are trained to work with the design problem similar to that used for data generation and require retraining if the design problem changes. In this work a scalable deep learning-based model-order-reduction method is proposed to accelerate large-scale TO process, by utilizing MapNet, a neural network which maps the field of interest from coarse-scale to fine-scale. The proposed method allows for each simulation of the TO process to be performed at a coarser mesh, thereby greatly reducing the total computational time. Moreover, by using domain fragmentation, the transferability of the MapNet is largely improved. Specifically, it has been demonstrated that the MapNet trained using data from one cantilever beam design with a specific loading condition can be directly applied to other structure design problems with different domain shapes, sizes, boundary and loading conditions.
Stochastic Video Prediction with Structure and Motion
Venue: arXiv cs.CV 2022
Year: 2022

While stochastic video prediction models enable future prediction under uncertainty, they mostly fail to model the complex dynamics of real-world scenes. For example, they cannot provide reliable predictions for scenes with a moving camera and independently moving foreground objects in driving scenarios. The existing methods fail to fully capture the dynamics of the structured world by only focusing on changes in pixels. In this paper, we assume that there is an underlying process creating observations in a video and propose to factorize it into static and dynamic components. We model the static part based on the scene structure and the ego-motion of the vehicle, and the dynamic part based on the remaining motion of the dynamic objects. By learning separate distributions of changes in foreground and background, we can decompose the scene into static and dynamic parts and separately model the change in each. Our experiments demonstrate that disentangling structure and motion helps stochastic video prediction, leading to better future predictions in complex driving scenarios on two real-world driving datasets, KITTI and Cityscapes.
Synergy between 6G and AI: Open Future Horizons and Impending Security Risks
Venue: arXiv cs.CR 2022
Year: 2022

This paper investigates the synergy between 6G and AI. It argues that they can unlock future horizons, by discussing how they can address future challenges in healthcare, transportation, virtual reality, education, resource management, robotics, in addition to public safety and warfare. However, these great opportunities come also with greater risk. Therefore, the paper provides an overview of the security risks and challenges, along with possible mitigation techniques.
MIDAS: Multi-sensorial Immersive Dynamic Autonomous System Improves Motivation of Stroke Affected Patients for Hand Rehabilitation
Venue: arXiv cs.RO 2022
Year: 2022

Majority of stroke survivors are left with poorly functioning paretic hands. Current rehabilitation devices have failed to motivate the patients enough to continue rehabilitation exercises. The objective of this project, MIDAS (Multi-sensorial Immersive Dynamic Autonomous System) is a proof of concept by using an immersive system to improve motivation of stroke patients for hand rehabilitation. MIDAS is intended for stroke patients who suffer from light to mild stroke. MIDAS is lightweight and portable. It consists of a hand exoskeleton subsystem, a Virtual Reality (VR) subsystem, and an olfactory subsystem. Altogether, MIDAS engages four out of five senses during rehabilitation. To evaluate the efficacy of MIDAS a pilot study consisting of three sessions is carried out on five stroke affected patients. Subsystems of MIDAS are added progressively in each session. The game environment, sonic effects, and scent released is carefully chosen to enhance the immersive experience. 60% of the scores of user experience are above 40 (out of 56). 96% Self Rehabilitation Motivation Scale (SRMS) rating shows that the participants are motivated to use MIDAS and 87% rating shows that MIDAS is exciting for rehabilitation. Participants experienced elevated motivation to continue stroke rehabilitation using MIDAS and no undesired side effects were reported.
Human Values Violations in Stack Overflow: An Exploratory Study
Venue: arXiv cs.SE 2022
Year: 2022

A growing number of software-intensive systems are being accused of violating or ignoring human values (e.g., privacy, inclusion, and social responsibility), and this poses great difficulties to individuals and society. Such violations often occur due to the solutions employed and decisions made by developers of such systems that are misaligned with user values. Stack Overflow is the most popular QA website among developers to share their issues, solutions (e.g., code snippets), and decisions during software development. We conducted an exploratory study to investigate the occurrence of human values violations in Stack Overflow posts. As comments under posts are often used to point out the possible issues and weaknesses of the posts, we analyzed 2000 Stack Overflow comments and their corresponding posts (1980 unique questions or answers) to identify the types of human values violations and the reactions of Stack Overflow users to such violations. Our study finds that 315 out of 2000 comments contain concerns indicating their associated posts (313 unique posts) violate human values. Leveraging Schwartz's theory of basic human values as the most widely used values model, we show that hedonism and benevolence are the most violated value categories. We also find the reaction of Stack Overflow commenters to perceived human values violations is very quick, yet the majority of posts (76.35%) accused of human values violation do not get downvoted at all. Finally, we find that the original posters rarely react to the concerns of potential human values violations by editing their posts. At the same time, they usually are receptive when responding to these comments in follow-up comments of their own.
LEReg: Empower Graph Neural Networks with Local Energy Regularization
Venue: arXiv cs.LG 2022
Year: 2022

Researches on analyzing graphs with Graph Neural Networks (GNNs) have been receiving more and more attention because of the great expressive power of graphs. GNNs map the adjacency matrix and node features to node representations by message passing through edges on each convolution layer. However, the message passed through GNNs is not always beneficial for all parts in a graph. Specifically, as the data distribution is different over the graph, the receptive field (the farthest nodes that a node can obtain information from) needed to gather information is also different. Existing GNNs treat all parts of the graph uniformly, which makes it difficult to adaptively pass the most informative message for each unique part. To solve this problem, we propose two regularization terms that consider message passing locally: (1) Intra-Energy Reg and (2) Inter-Energy Reg. Through experiments and theoretical discussion, we first show that the speed of smoothing of different parts varies enormously and the topology of each part affects the way of smoothing. With Intra-Energy Reg, we strengthen the message passing within each part, which is beneficial for getting more useful information. With Inter-Energy Reg, we improve the ability of GNNs to distinguish different nodes. With the proposed two regularization terms, GNNs are able to filter the most useful information adaptively, learn more robustly and gain higher expressiveness. Moreover, the proposed LEReg can be easily applied to other GNN models with plug-and-play characteristics. Extensive experiments on several benchmarks verify that GNNs with LEReg outperform or match the state-of-the-art methods. The effectiveness and efficiency are also empirically visualized with elaborate experiments.
Point3D: tracking actions as moving points with 3D CNNs
Venue: arXiv cs.CV 2022
Year: 2022

Spatio-temporal action recognition has been a challenging task that involves detecting where and when actions occur. Current state-of-the-art action detectors are mostly anchor-based, requiring sensitive anchor designs and huge computations due to calculating large numbers of anchor boxes. Motivated by nascent anchor-free approaches, we propose Point3D, a flexible and computationally efficient network with high precision for spatio-temporal action recognition. Our Point3D consists of a Point Head for action localization and a 3D Head for action classification. Firstly, Point Head is used to track center points and knot key points of humans to localize the bounding box of an action. These location features are then piped into a time-wise attention to learn long-range dependencies across frames. The 3D Head is later deployed for the final action classification. Our Point3D achieves state-of-the-art performance on the JHMDB, UCF101-24, and AVA benchmarks in terms of frame-mAP and video-mAP. Comprehensive ablation studies also demonstrate the effectiveness of each module proposed in our Point3D.
A Dual-Arm Robot that Manipulates Heavy Plates Cooperatively with a Vacuum Lifter
Venue: arXiv cs.RO 2022
Year: 2022

A vacuum lifter is widely used to hold and pick up large, heavy, and flat objects. Conventionally, when using a vacuum lifter, a human worker watches the state of a running vacuum lifter and adjusts the object's pose to maintain balance. In this work, we propose using a dual-arm robot to replace the human workers and develop planning and control methods for a dual-arm robot to raise a heavy plate with the help of a vacuum lifter. The methods help the robot determine its actions by considering the vacuum lifer's suction position and suction force limits. The essence of the methods is two-fold. First, we build a Manipulation State Graph (MSG) to store the weighted logical relations of various plate contact states and robot/vacuum lifter configurations, and search the graph to plan efficient and low-cost robot manipulation sequences. Second, we develop a velocity-based impedance controller to coordinate the robot and the vacuum lifter when lifting an object. With its help, a robot can follow the vacuum lifter's motion and realize compliant robot-vacuum lifter collaboration. The proposed planning and control methods are investigated using real-world experiments. The results show that a robot can effectively and flexibly work together with a vacuum lifter to manipulate large and heavy plate-like objects with the methods' support.
Does DQN really learn? Exploring adversarial training schemes in Pong
Venue: arXiv cs.LG 2022
Year: 2022

In this work, we study two self-play training schemes, Chainer and Pool, and show they lead to improved agent performance in Atari Pong compared to a standard DQN agent -- trained against the built-in Atari opponent. To measure agent performance, we define a robustness metric that captures how difficult it is to learn a strategy that beats the agent's learned policy. Through playing past versions of themselves, Chainer and Pool are able to target weaknesses in their policies and improve their resistance to attack. Agents trained using these methods score well on our robustness metric and can easily defeat the standard DQN agent. We conclude by using linear probing to illuminate what internal structures the different agents develop to play the game. We show that training agents with Chainer or Pool leads to richer network activations with greater predictive power to estimate critical game-state features compared to the standard DQN agent.
Hierarchical Reinforcement Learning of Locomotion Policies in Response to Approaching Objects: A Preliminary Study
Venue: arXiv cs.RO 2022
Year: 2022

Animals such as rabbits and birds can instantly generate locomotion behavior in reaction to a dynamic, approaching object, such as a person or a rock, despite having possibly never seen the object before and having limited perception of the object's properties. Recently, deep reinforcement learning has enabled complex kinematic systems such as humanoid robots to successfully move from point A to point B. Inspired by the observation of the innate reactive behavior of animals in nature, we hope to extend this progress in robot locomotion to settings where external, dynamic objects are involved whose properties are partially observable to the robot. As a first step toward this goal, we build a simulation environment in MuJoCo where a legged robot must avoid getting hit by a ball moving toward it. We explore whether prior locomotion experiences that animals typically possess benefit the learning of a reactive control policy under a proposed hierarchical reinforcement learning framework. Preliminary results support the claim that the learning becomes more efficient using this hierarchical reinforcement learning method, even when partial observability (radius-based object visibility) is taken into account.
Immersive Text Game and Personality Classification
Venue: arXiv cs.CL 2022
Year: 2022

We designed and built a game called \textit{Immersive Text Game}, which allows the player to choose a story and a character, and interact with other characters in the story in an immersive manner of dialogues. The game is based on several latest models, including text generation language model, information extraction model, commonsense reasoning model, and psychology evaluation model. In the past, similar text games usually let players choose from limited actions instead of answering on their own, and not every time what characters said are determined by the player. Through the combination of these models and elaborate game mechanics and modes, the player will find some novel experiences as driven through the storyline.