OpenRaiser/Intern-Atlas huggingface dataset:
Learning-based Inverse Rendering of Complex Indoor Scenes with Differentiable Monte Carlo Raytracing
Venue: arXiv cs.CV 2022
Year: 2022

Indoor scenes typically exhibit complex, spatially-varying appearance from global illumination, making inverse rendering a challenging ill-posed problem. This work presents an end-to-end, learning-based inverse rendering framework incorporating differentiable Monte Carlo raytracing with importance sampling. The framework takes a single image as input to jointly recover the underlying geometry, spatially-varying lighting, and photorealistic materials. Specifically, we introduce a physically-based differentiable rendering layer with screen-space ray tracing, resulting in more realistic specular reflections that match the input photo. In addition, we create a large-scale, photorealistic indoor scene dataset with significantly richer details like complex furniture and dedicated decorations. Further, we design a novel out-of-view lighting network with uncertainty-aware refinement leveraging hypernetwork-based neural radiance fields to predict lighting outside the view of the input photo. Through extensive evaluations on common benchmark datasets, we demonstrate superior inverse rendering quality of our method compared to state-of-the-art baselines, enabling various applications such as complex object insertion and material editing with high fidelity. Code and data will be made available at \url{https://jingsenzhu.github.io/invrend}.
Bridging Speech and Textual Pre-trained Models with Unsupervised ASR
Venue: arXiv cs.CL 2022
Year: 2022

Spoken language understanding (SLU) is a task aiming to extract high-level semantics from spoken utterances. Previous works have investigated the use of speech self-supervised models and textual pre-trained models, which have shown reasonable improvements to various SLU tasks. However, because of the mismatched modalities between speech signals and text tokens, previous methods usually need complex designs of the frameworks. This work proposes a simple yet efficient unsupervised paradigm that connects speech and textual pre-trained models, resulting in an unsupervised speech-to-semantic pre-trained model for various tasks in SLU. To be specific, we propose to use unsupervised automatic speech recognition (ASR) as a connector that bridges different modalities used in speech and textual pre-trained models. Our experiments show that unsupervised ASR itself can improve the representations from speech self-supervised models. More importantly, it is shown as an efficient connector between speech and textual pre-trained models, improving the performances of five different SLU tasks. Notably, on spoken question answering, we reach the state-of-the-art result over the challenging NMSQA benchmark.
Efficient Traffic State Forecasting using Spatio-Temporal Network Dependencies: A Sparse Graph Neural Network Approach
Venue: arXiv cs.LG 2022
Year: 2022

Traffic state prediction in a transportation network is paramount for effective traffic operations and management, as well as informed user and system-level decision-making. However, long-term traffic prediction (beyond 30 minutes into the future) remains challenging in current research. In this work, we integrate the spatio-temporal dependencies in the transportation network from network modeling, together with the graph convolutional network (GCN) and graph attention network (GAT). To further tackle the dramatic computation and memory cost caused by the giant model size (i.e., number of weights) caused by multiple cascaded layers, we propose sparse training to mitigate the training cost, while preserving the prediction accuracy. It is a process of training using a fixed number of nonzero weights in each layer in each iteration. We consider the problem of long-term traffic speed forecasting for a real large-scale transportation network data from the California Department of Transportation (Caltrans) Performance Measurement System (PeMS). Experimental results show that the proposed GCN-STGT and GAT-STGT models achieve low prediction errors on short-, mid- and long-term prediction horizons, of 15, 30 and 45 minutes in duration, respectively. Using our sparse training, we could train from scratch with high sparsity (e.g., up to 90%), equivalent to 10 times floating point operations per second (FLOPs) reduction on computational cost using the same epochs as dense training, and arrive at a model with very small accuracy loss compared with the original dense training
Learning body models: from humans to humanoids
Venue: arXiv cs.RO 2022
Year: 2022

Humans and animals excel in combining information from multiple sensory modalities, controlling their complex bodies, adapting to growth, failures, or using tools. These capabilities are also highly desirable in robots. They are displayed by machines to some extent. Yet, the artificial creatures are lagging behind. The key foundation is an internal representation of the body that the agent - human, animal, or robot - has developed. The mechanisms of operation of body models in the brain are largely unknown and even less is known about how they are constructed from experience after birth. In collaboration with developmental psychologists, we conducted targeted experiments to understand how infants acquire first "sensorimotor body knowledge". These experiments inform our work in which we construct embodied computational models on humanoid robots that address the mechanisms behind learning, adaptation, and operation of multimodal body representations. At the same time, we assess which of the features of the "body in the brain" should be transferred to robots to give rise to more adaptive and resilient, self-calibrating machines. We extend traditional robot kinematic calibration focusing on self-contained approaches where no external metrology is needed: self-contact and self-observation. Problem formulation allowing to combine several ways of closing the kinematic chain simultaneously is presented, along with a calibration toolbox and experimental validation on several robot platforms. Finally, next to models of the body itself, we study peripersonal space - the space immediately surrounding the body. Again, embodied computational models are developed and subsequently, the possibility of turning these biologically inspired representations into safe human-robot collaboration is studied.
On the Specialization of FDRL Agents for Scalable and Distributed 6G RAN Slicing Orchestration
Venue: arXiv cs.NI 2022
Year: 2022

Network slicing enables multiple virtual networks to be instantiated and customized to meet heterogeneous use case requirements over 5G and beyond network deployments. However, most of the solutions available today face scalability issues when considering many slices, due to centralized controllers requiring a holistic view of the resource availability and consumption over different networking domains. In order to tackle this challenge, we design a hierarchical architecture to manage network slices resources in a federated manner. Driven by the rapid evolution of deep reinforcement learning (DRL) schemes and the Open RAN (O-RAN) paradigm, we propose a set of traffic-aware local decision agents (DAs) dynamically placed in the radio access network (RAN). These federated decision entities tailor their resource allocation policy according to the long-term dynamics of the underlying traffic, defining specialized clusters that enable faster training and communication overhead reduction. Indeed, aided by a traffic-aware agent selection algorithm, our proposed Federated DRL approach provides higher resource efficiency than benchmark solutions by quickly reacting to end-user mobility patterns and reducing costly interactions with centralized controllers.
MiddleGAN: Generate Domain Agnostic Samples for Unsupervised Domain Adaptation
Venue: arXiv cs.CV 2022
Year: 2022

In recent years, machine learning has achieved impressive results across different application areas. However, machine learning algorithms do not necessarily perform well on a new domain with a different distribution than its training set. Domain Adaptation (DA) is used to mitigate this problem. One approach of existing DA algorithms is to find domain invariant features whose distributions in the source domain are the same as their distribution in the target domain. In this paper, we propose to let the classifier that performs the final classification task on the target domain learn implicitly the invariant features to perform classification. It is achieved via feeding the classifier during training generated fake samples that are similar to samples from both the source and target domains. We call these generated samples domain-agnostic samples. To accomplish this we propose a novel variation of generative adversarial networks (GAN), called the MiddleGAN, that generates fake samples that are similar to samples from both the source and target domains, using two discriminators and one generator. We extend the theory of GAN to show that there exist optimal solutions for the parameters of the two discriminators and one generator in MiddleGAN, and empirically show that the samples generated by the MiddleGAN are similar to both samples from the source domain and samples from the target domain. We conducted extensive evaluations using 24 benchmarks; on the 24 benchmarks, we compare MiddleGAN against various state-of-the-art algorithms and outperform the state-of-the-art by up to 20.1\% on certain benchmarks.
Balancing graph Voronoi diagrams with one more vertex
Venue: arXiv cs.DS 2022
Year: 2022

Let $G=(V,E)$ be a graph with unit-length edges and nonnegative costs assigned to its vertices. Being given a list of pairwise different vertices $S=(s_1,s_2,\ldots,s_p)$, the {\em prioritized Voronoi diagram} of $G$ with respect to $S$ is the partition of $G$ in $p$ subsets $V_1,V_2,\ldots,V_p$ so that, for every $i$ with $1 \leq i \leq p$, a vertex $v$ is in $V_i$ if and only if $s_i$ is a closest vertex to $v$ in $S$ and there is no closest vertex to $v$ in $S$ within the subset $\{s_1,s_2,\ldots,s_{i-1}\}$. For every $i$ with $1 \leq i \leq p$, the {\em load} of vertex $s_i$ equals the sum of the costs of all vertices in $V_i$. The load of $S$ equals the maximum load of a vertex in $S$. We study the problem of adding one more vertex $v$ at the end of $S$ in order to minimize the load. This problem occurs in the context of optimally locating a new service facility ({\it e.g.}, a school or a hospital) while taking into account already existing facilities, and with the goal of minimizing the maximum congestion at a site. There is a brute-force algorithm for solving this problem in ${\cal O}(nm)$ time on $n$-vertex $m$-edge graphs. We prove a matching time lower bound for the special case where $m=n^{1+o(1)}$ and $p=1$, assuming the so called Hitting Set Conjecture of Abboud et al. On the positive side, we present simple linear-time algorithms for this problem on cliques, paths and cycles, and almost linear-time algorithms for trees, proper interval graphs and (assuming $p$ to be a constant) bounded-treewidth graphs.
KGTN-ens: Few-Shot Image Classification with Knowledge Graph Ensembles
Venue: arXiv cs.CV 2022
Year: 2022

We propose KGTN-ens, a framework extending the recent Knowledge Graph Transfer Network (KGTN) in order to incorporate multiple knowledge graph embeddings at a small cost. We evaluate it with different combinations of embeddings in a few-shot image classification task. We also construct a new knowledge source - Wikidata embeddings - and evaluate it with KGTN and KGTN-ens. Our approach outperforms KGTN in terms of the top-5 accuracy on the ImageNet-FS dataset for the majority of tested settings.
Fast Key Points Detection and Matching for Tree-Structured Images
Venue: arXiv cs.CV 2022
Year: 2022

This paper offers a new authentication algorithm based on image matching of nano-resolution visual identifiers with tree-shaped patterns. The algorithm includes image-to-tree conversion by greedy extraction of the fractal pattern skeleton along with a custom-built graph matching algorithm that is robust against imaging artifacts such as scaling, rotation, scratch, and illumination change. The proposed algorithm is applicable to a variety of tree-structured image matching, but our focus is on dendrites, recently-developed visual identifiers. Dendrites are entropy rich and unclonable with existing 2D and 3D printers due to their natural randomness, nano-resolution granularity, and 3D facets, making them an appropriate choice for security applications such as supply chain trace and tracking. The proposed algorithm improves upon graph matching with standard image descriptors. For instance, image inconsistency due to the camera sensor noise may cause unexpected feature extraction leading to inaccurate tree conversion and authentication failure. Also, previous tree extraction algorithms are prohibitively slow hindering their scalability to large systems. In this paper, we fix the current issues of [1] and accelerate the key points extraction up to 10-times faster by implementing a new skeleton extraction method, a new key points searching algorithm, as well as an optimized key point matching algorithm. Using minimum enclosing circle and center points, make the algorithm robust to the choice of pattern shape. In contrast to [1] our algorithm handles general graphs with loop connections, therefore is applicable to a wider range of applications such as transportation map analysis, fingerprints, and retina vessel imaging.
Soft Robotic Link with Controllable Transparency for Vision-based Tactile and Proximity Sensing
Venue: arXiv cs.RO 2022
Year: 2022

Robots have been brought to work close to humans in many scenarios. For coexistence and collaboration, robots should be safe and pleasant for humans to interact with. To this end, the robots could be both physically soft with multimodal sensing/perception, so that the robots could have better awareness of the surrounding environment, as well as to respond properly to humans' action/intention. This paper introduces a novel soft robotic link, named ProTac, that possesses multiple sensing modes: tactile and proximity sensing, based on computer vision and a functional material. These modalities come from a layered structure of a soft transparent silicon skin, a polymer dispersed liquid crystal (PDLC) film, and reflective markers. Here, the PDLC film can switch actively between the opaque and the transparent state, from which the tactile sensing and proximity sensing can be obtained by using cameras solely built inside the ProTac link. In this paper, inference algorithms for tactile proximity perception are introduced. Evaluation results of two sensing modalities demonstrated that, with a simple activation strategy, ProTac link could effectively perceive useful information from both approaching and in-contact obstacles. The proposed sensing device is expected to bring in ultimate solutions for design of robots with softness, whole-body and multimodal sensing, and safety control strategies.
Complex Reading Comprehension Through Question Decomposition
Venue: arXiv cs.CL 2022
Year: 2022

Multi-hop reading comprehension requires not only the ability to reason over raw text but also the ability to combine multiple evidence. We propose a novel learning approach that helps language models better understand difficult multi-hop questions and perform "complex, compositional" reasoning. Our model first learns to decompose each multi-hop question into several sub-questions by a trainable question decomposer. Instead of answering these sub-questions, we directly concatenate them with the original question and context, and leverage a reading comprehension model to predict the answer in a sequence-to-sequence manner. By using the same language model for these two components, our best seperate/unified t5-base variants outperform the baseline by 7.2/6.1 absolute F1 points on a hard subset of DROP dataset.
Learning Task Requirements and Agent Capabilities for Multi-agent Task Allocation
Venue: arXiv cs.RO 2022
Year: 2022

This paper presents a learning framework to estimate an agent capability and task requirement model for multi-agent task allocation. With a set of team configurations and the corresponding task performances as the training data, linear task constraints can be learned to be embedded in many existing optimization-based task allocation frameworks. Comprehensive computational evaluations are conducted to test the scalability and prediction accuracy of the learning framework with a limited number of team configurations and performance pairs. A ROS and Gazebo-based simulation environment is developed to validate the proposed requirements learning and task allocation framework in practical multi-agent exploration and manipulation tasks. Results show that the learning process for scenarios with 40 tasks and 6 types of agents uses around 12 seconds, ending up with prediction errors in the range of 0.5-2%.
The Chart Excites Me! Exploring How Data Visualization Design Influences Affective Arousal
Venue: arXiv cs.HC 2022
Year: 2022

As data visualizations have been increasingly applied in mass communication, designers often seek to grasp viewers immediately and motivate them to read more. Such goals, as suggested by previous research, are closely associated with the activation of emotion, namely affective arousal. Given this motivation, this work takes initial steps toward understanding the arousal-related factors in data visualization design. We collected a corpus of 265 data visualizations and conducted a crowdsourcing study with 184 participants during which the participants were asked to rate the affective arousal elicited by data visualization design (all texts were blurred to exclude the influence of semantics) and provide their reasons. Based on the collected data, first, we identified a set of arousal-related design features by analyzing user comments qualitatively. Then, we mapped these features to computable variables and constructed regression models to infer which features are significant contributors to affective arousal quantitatively. Through this exploratory study, we finally identified four design features (e.g., colorfulness, the number of different visual channels) cross-validated as important features correlated with affective arousal.
Instantaneous Channel Oblivious Phase Shift Design for an IRS-Assisted SIMO System with Quantized Phase Shift
Venue: arXiv cs.IT 2022
Year: 2022

We design the phase shifts of an intelligent reflecting surface (IRS)-assisted single-input-multiple-output communication system to minimize the outage probability (OP) and to maximize the ergodic rate. Our phase shifts design uses only statistical channel state information since these depend only on the large-scale fading coefficients; the obtained phase shift design remains valid for a longer time frame. We further assume that one has access to only quantized phase values. The closed-form expressions for OP and ergodic rate are derived for the considered system. Next, two optimization problems are formulated to choose the phase shifts of IRS such that (i) OP is minimized and (ii) the ergodic rate is maximized. We used the multi-valued particle swarm optimization (MPSO) and particle swarm optimization (PSO) algorithms to solve the optimization problems. Numerical simulations are performed to study the impact of various parameters on the OP and ergodic rate. We also discuss signaling overhead between BS and IRS controller. It is shown that the overhead can be reduced up to $99.69 \%$ by using statistical CSI for phase shift design and $5$ bits to represent the phase shifts without significantly compromising on the performance.
Bayesian Disturbance Injection: Robust Imitation Learning of Flexible Policies for Robot Manipulation
Venue: arXiv cs.RO 2022
Year: 2022

Humans demonstrate a variety of interesting behavioral characteristics when performing tasks, such as selecting between seemingly equivalent optimal actions, performing recovery actions when deviating from the optimal trajectory, or moderating actions in response to sensed risks. However, imitation learning, which attempts to teach robots to perform these same tasks from observations of human demonstrations, often fails to capture such behavior. Specifically, commonly used learning algorithms embody inherent contradictions between the learning assumptions (e.g., single optimal action) and actual human behavior (e.g., multiple optimal actions), thereby limiting robot generalizability, applicability, and demonstration feasibility. To address this, this paper proposes designing imitation learning algorithms with a focus on utilizing human behavioral characteristics, thereby embodying principles for capturing and exploiting actual demonstrator behavioral characteristics. This paper presents the first imitation learning framework, Bayesian Disturbance Injection (BDI), that typifies human behavioral characteristics by incorporating model flexibility, robustification, and risk sensitivity. Bayesian inference is used to learn flexible non-parametric multi-action policies, while simultaneously robustifying policies by injecting risk-sensitive disturbances to induce human recovery action and ensuring demonstration feasibility. Our method is evaluated through risk-sensitive simulations and real-robot experiments (e.g., table-sweep task, shaft-reach task and shaft-insertion task) using the UR5e 6-DOF robotic arm, to demonstrate the improved characterisation of behavior. Results show significant improvement in task performance, through improved flexibility, robustness as well as demonstration feasibility.
3D Harmonic Loss: Towards Task-consistent and Time-friendly 3D Object Detection on Edge for V2X Orchestration
Venue: arXiv cs.CV 2022
Year: 2022

Edge computing-based 3D perception has received attention in intelligent transportation systems (ITS) because real-time monitoring of traffic candidates potentially strengthens Vehicle-to-Everything (V2X) orchestration. Thanks to the capability of precisely measuring the depth information on surroundings from LiDAR, the increasing studies focus on lidar-based 3D detection, which significantly promotes the development of 3D perception. Few methods met the real-time requirement of edge deployment because of high computation-intensive operations. Moreover, an inconsistency problem of object detection remains uncovered in the pointcloud domain due to large sparsity. This paper thoroughly analyses this problem, comprehensively roused by recent works on determining inconsistency problems in the image specialisation. Therefore, we proposed a 3D harmonic loss function to relieve the pointcloud based inconsistent predictions. Moreover, the feasibility of 3D harmonic loss is demonstrated from a mathematical optimization perspective. The KITTI dataset and DAIR-V2X-I dataset are used for simulations, and our proposed method considerably improves the performance than benchmark models. Further, the simulative deployment on an edge device (Jetson Xavier TX) validates our proposed model's efficiency.
How to Coordinate Edge Devices for Over-the-Air Federated Learning?
Venue: arXiv cs.IT 2022
Year: 2022

This work studies the task of device coordination in wireless networks for over-the-air federated learning (OTA-FL). For conventional metrics of aggregation error, the task is shown to describe the zero-forcing (ZF) and minimum mean squared error (MMSE) schemes and reduces to the NP-hard problem of subset selection. We tackle this problem by studying properties of the optimal scheme. Our analytical results reveal that this scheme is found by searching among the leaves of a tree with favorable monotonic features. Invoking these features, we develop a low-complexity algorithm that approximates the optimal scheme by tracking a dominant path of the tree sequentially. Our numerical investigations show that the proposed algorithm closely tracks the optimal scheme.