OpenRaiser/Intern-Atlas huggingface dataset:
Behind the Curtain: Learning Occluded Shapes for 3D Object Detection
Venue: arXiv cs.CV 2021
Year: 2021

Advances in LiDAR sensors provide rich 3D data that supports 3D scene understanding. However, due to occlusion and signal miss, LiDAR point clouds are in practice 2.5D as they cover only partial underlying shapes, which poses a fundamental challenge to 3D perception. To tackle the challenge, we present a novel LiDAR-based 3D object detection model, dubbed Behind the Curtain Detector (BtcDet), which learns the object shape priors and estimates the complete object shapes that are partially occluded (curtained) in point clouds. BtcDet first identifies the regions that are affected by occlusion and signal miss. In these regions, our model predicts the probability of occupancy that indicates if a region contains object shapes. Integrated with this probability map, BtcDet can generate high-quality 3D proposals. Finally, the probability of occupancy is also integrated into a proposal refinement module to generate the final bounding boxes. Extensive experiments on the KITTI Dataset and the Waymo Open Dataset demonstrate the effectiveness of BtcDet. Particularly, for the 3D detection of both cars and cyclists on the KITTI benchmark, BtcDet surpasses all of the published state-of-the-art methods by remarkable margins. Code is released (https://github.com/Xharlie/BtcDet}{https://github.com/Xharlie/BtcDet).
Joint Audio-Text Model for Expressive Speech-Driven 3D Facial Animation
Venue: arXiv cs.CV 2021
Year: 2021

Speech-driven 3D facial animation with accurate lip synchronization has been widely studied. However, synthesizing realistic motions for the entire face during speech has rarely been explored. In this work, we present a joint audio-text model to capture the contextual information for expressive speech-driven 3D facial animation. The existing datasets are collected to cover as many different phonemes as possible instead of sentences, thus limiting the capability of the audio-based model to learn more diverse contexts. To address this, we propose to leverage the contextual text embeddings extracted from the powerful pre-trained language model that has learned rich contextual representations from large-scale text data. Our hypothesis is that the text features can disambiguate the variations in upper face expressions, which are not strongly correlated with the audio. In contrast to prior approaches which learn phoneme-level features from the text, we investigate the high-level contextual text features for speech-driven 3D facial animation. We show that the combined acoustic and textual modalities can synthesize realistic facial expressions while maintaining audio-lip synchronization. We conduct the quantitative and qualitative evaluations as well as the perceptual user study. The results demonstrate the superior performance of our model against existing state-of-the-art approaches.
Revisiting $k$-Nearest Neighbor Graph Construction on High-Dimensional Data : Experiments and Analyses
Venue: arXiv cs.DS 2021
Year: 2021

The $k$-nearest neighbor graph (KNNG) on high-dimensional data is a data structure widely used in many applications such as similarity search, dimension reduction and clustering. Due to its increasing popularity, several methods under the same framework have been proposed in the past decade. This framework contains two steps, i.e. building an initial KNNG (denoted as \texttt{INIT}) and then refining it by neighborhood propagation (denoted as \texttt{NBPG}). However, there remain several questions to be answered. First, it lacks a comprehensive experimental comparison among representative solutions in the literature. Second, some recently proposed indexing structures, e.g., SW and HNSW, have not been used or tested for building an initial KNNG. Third, the relationship between the data property and the effectiveness of \texttt{NBPG} is still not clear. To address these issues, we comprehensively compare the representative approaches on real-world high-dimensional data sets to provide practical and insightful suggestions for users. As the first attempt, we take SW and HNSW as the alternatives of \texttt{INIT} in our experiments. Moreover, we investigate the effectiveness of \texttt{NBPG} and find the strong correlation between the huness phenomenon and the performance of \texttt{NBPG}.
Algorithms for Maximum Internal Spanning Tree Problem for Some Graph Classes
Venue: arXiv cs.DS 2021
Year: 2021

For a given graph $G$, a maximum internal spanning tree of $G$ is a spanning tree of $G$ with maximum number of internal vertices. The Maximum Internal Spanning Tree (MIST) problem is to find a maximum internal spanning tree of the given graph. The MIST problem is a generalization of the Hamiltonian path problem. Since the Hamiltonian path problem is NP-hard, even for bipartite and chordal graphs, two important subclasses of graphs, the MIST problem also remains NP-hard for these graph classes. In this paper, we propose linear-time algorithms to compute a maximum internal spanning tree of cographs, block graphs, cactus graphs, chain graphs and bipartite permutation graphs. The optimal path cover problem, which asks to find a path cover of the given graph with maximum number of edges, is also a well studied problem. In this paper, we also study the relationship between the number of internal vertices in maximum internal spanning tree and number of edges in optimal path cover for the special graph classes mentioned above.
Integration of FogBus2 Framework with Container Orchestration Tools in Cloud and Edge Computing Environments
Venue: arXiv cs.DC 2021
Year: 2021

Currently, due to the advantages of light weight, simple deployment, multi-environment support, short startup time, scalability, and easy migration, container technology has been widely used in both cloud and edge/fog computing, and addresses the problem of device heterogeneity in different computing environments. On this basis, as one of the most popular container orchestration and management systems, Kubernetes almost dominates the cloud environment. However, since it is primarily designed for centralized resource management scenarios where computing resources are sufficient, the system is unstable in edge environments due to hardware limitations. Therefore, in order to realize container orchestration in the cloud and edge/fog hybrid computing environment, we propose a feasible approach to build a hybrid clustering based on K3s, which solves the problem that virtual instances in different environments cannot be connected due to IP addresses. We also propose three design patterns for deploying the FogBus2 framework into hybrid environments, including 1) Host Network Mode, 2) Proxy Server, and 3) Environment Variable.
BAANet: Learning Bi-directional Adaptive Attention Gates for Multispectral Pedestrian Detection
Venue: arXiv cs.CV 2021
Year: 2021

Thermal infrared (TIR) image has proven effectiveness in providing temperature cues to the RGB features for multispectral pedestrian detection. Most existing methods directly inject the TIR modality into the RGB-based framework or simply ensemble the results of two modalities. This, however, could lead to inferior detection performance, as the RGB and TIR features generally have modality-specific noise, which might worsen the features along with the propagation of the network. Therefore, this work proposes an effective and efficient cross-modality fusion module called Bi-directional Adaptive Attention Gate (BAA-Gate). Based on the attention mechanism, the BAA-Gate is devised to distill the informative features and recalibrate the representations asymptotically. Concretely, a bi-direction multi-stage fusion strategy is adopted to progressively optimize features of two modalities and retain their specificity during the propagation. Moreover, an adaptive interaction of BAA-Gate is introduced by the illumination-based weighting strategy to adaptively adjust the recalibrating and aggregating strength in the BAA-Gate and enhance the robustness towards illumination changes. Considerable experiments on the challenging KAIST dataset demonstrate the superior performance of our method with satisfactory speed.
BenchML: an extensible pipelining framework for benchmarking representations of materials and molecules at scale
Venue: arXiv cs.LG 2021
Year: 2021

We introduce a machine-learning (ML) framework for high-throughput benchmarking of diverse representations of chemical systems against datasets of materials and molecules. The guiding principle underlying the benchmarking approach is to evaluate raw descriptor performance by limiting model complexity to simple regression schemes while enforcing best ML practices, allowing for unbiased hyperparameter optimization, and assessing learning progress through learning curves along series of synchronized train-test splits. The resulting models are intended as baselines that can inform future method development, next to indicating how easily a given dataset can be learnt. Through a comparative analysis of the training outcome across a diverse set of physicochemical, topological and geometric representations, we glean insight into the relative merits of these representations as well as their interrelatedness.
My(o) Armband Leaks Passwords: An EMG and IMU Based Keylogging Side-Channel Attack
Venue: arXiv cs.CR 2021
Year: 2021

Wearables that constantly collect various sensor data of their users increase the chances for inferences of unintentional and sensitive information such as passwords typed on a physical keyboard. We take a thorough look at the potential of using electromyographic (EMG) data, a sensor modality which is new to the market but has lately gained attention in the context of wearables for augmented reality (AR), for a keylogging side-channel attack. Our approach is based on neural networks for a between-subject attack in a realistic scenario using the Myo Armband to collect the sensor data. In our approach, the EMG data has proven to be the most prominent source of information compared to the accelerometer and gyroscope, increasing the keystroke detection performance. For our end-to-end approach on raw data, we report a mean balanced accuracy of about 76 % for the keystroke detection and a mean top-3 key accuracy of about 32 % on 52 classes for the key identification on passwords of varying strengths. We have created an extensive dataset including more than 310 000 keystrokes recorded from 37 volunteers, which is available as open access along with the source code used to create the given results.
Marching with the Pink Parade: Evaluating Visual Search Recommendations for Non-binary Clothing Items
Venue: arXiv cs.HC 2021
Year: 2021

Fashion, a highly subjective topic is interpreted differently by all individuals. E-commerce platforms, despite these diverse requirements, tend to cater to the average buyer instead of focusing on edge cases like non-binary shoppers. This case study, through participant surveys, shows that visual search on e-commerce platforms like Amazon, Beagle.Vision and Lykdat, is particularly poor for non-binary clothing items. Our comprehensive quantitative analysis shows that these platforms are more robust to binary clothing inputs. The non-binary clothing items are recommended in a haphazard manner, as observed through negative correlation coefficients of the ranking order. The participants also rate the non-binary recommendations lower than the binary ones. Another intriguing observation is that male raters are more inclined to make binary judgements compared to female raters. Thus it is clear that these systems are not inclusive to the minority, disadvantaged communities of society, like LGBTQ+ people. We conclude with a call to action for the e-commerce platforms to take cognizance of our results and be more inclusive.
Fast Graph Neural Tangent Kernel via Kronecker Sketching
Venue: arXiv cs.LG 2021
Year: 2021

Many deep learning tasks have to deal with graphs (e.g., protein structures, social networks, source code abstract syntax trees). Due to the importance of these tasks, people turned to Graph Neural Networks (GNNs) as the de facto method for learning on graphs. GNNs have become widely applied due to their convincing performance. Unfortunately, one major barrier to using GNNs is that GNNs require substantial time and resources to train. Recently, a new method for learning on graph data is Graph Neural Tangent Kernel (GNTK) [Du, Hou, Salakhutdinov, Poczos, Wang and Xu 19]. GNTK is an application of Neural Tangent Kernel (NTK) [Jacot, Gabriel and Hongler 18] (a kernel method) on graph data, and solving NTK regression is equivalent to using gradient descent to train an infinite-wide neural network. The key benefit of using GNTK is that, similar to any kernel method, GNTK's parameters can be solved directly in a single step. This can avoid time-consuming gradient descent. Meanwhile, sketching has become increasingly used in speeding up various optimization problems, including solving kernel regression. Given a kernel matrix of $n$ graphs, using sketching in solving kernel regression can reduce the running time to $o(n^3)$. But unfortunately such methods usually require extensive knowledge about the kernel matrix beforehand, while in the case of GNTK we find that the construction of the kernel matrix is already $O(n^2N^4)$, assuming each graph has $N$ nodes. The kernel matrix construction time can be a major performance bottleneck when the size of graphs $N$ increases. A natural question to ask is thus whether we can speed up the kernel matrix construction to improve GNTK regression's end-to-end running time. This paper provides the first algorithm to construct the kernel matrix in $o(n^2N^3)$ running time.
Anomaly Detection of Wind Turbine Time Series using Variational Recurrent Autoencoders
Venue: arXiv cs.LG 2021
Year: 2021

Ice accumulation in the blades of wind turbines can cause them to describe anomalous rotations or no rotations at all, thus affecting the generation of electricity and power output. In this work, we investigate the problem of ice accumulation in wind turbines by framing it as anomaly detection of multi-variate time series. Our approach focuses on two main parts: first, learning low-dimensional representations of time series using a Variational Recurrent Autoencoder (VRAE), and second, using unsupervised clustering algorithms to classify the learned representations as normal (no ice accumulated) or abnormal (ice accumulated). We have evaluated our approach on a custom wind turbine time series dataset, for the two-classes problem (one normal versus one abnormal class), we obtained a classification accuracy of up to 96$\%$ on test data. For the multiple-class problem (one normal versus multiple abnormal classes), we present a qualitative analysis of the low-dimensional learned latent space, providing insights into the capacities of our approach to tackle such problem. The code to reproduce this work can be found here https://github.com/agrija9/Wind-Turbines-VRAE-Paper.
Enhancing Data-driven Multiscale Topology Optimization with Generalized De-homogenization
Venue: arXiv cs.CE 2021
Year: 2021

De-homogenization is becoming an effective method to significantly expedite the design of high-resolution multiscale structures, but existing methods have thus far been confined to simple static compliance minimization. There are two critical challenges to be addressed in accommodating general cases: enabling the design of unit-cell orientation and using free-form microstructures. In this paper, we propose a data-driven de-homogenization method that allows effective design of the unit-cell orientation angles and conformal mapping of spatially varying, complex microstructures. We devise a parameterized microstructure composed of rods in different directions to provide more diversity in stiffness while retaining geometrical simplicity. The microstructural geometry-property relationship is then surrogated by a neural network to avoid costly homogenization. A Cartesian representation of the unit-cell orientation is incorporated into homogenization-based optimization to design the angles. Corresponding high-resolution multiscale structures are obtained from the homogenization-based designs through a conformal mapping constructed with sawtooth function fields. This allows us to assemble complex microstructures with an oriented and compatible tiling pattern, while preserving the local homogenized properties. To demonstrate our method with a specific application, we optimize the frequency response of structures under harmonic excitations within a given frequency range. It is the first time that a sawtooth function is applied in a de-homogenization framework for complex design scenarios beyond static compliance minimization. The examples illustrate that multiscale structures can be generated with high efficiency and much better dynamic performance compared with the macroscale-only optimization. Beyond frequency response design, our proposed framework can be applied to other general problems.
Neural Photometry-guided Visual Attribute Transfer
Venue: arXiv cs.CV 2021
Year: 2021

We present a deep learning-based method for propagating spatially-varying visual material attributes (e.g. texture maps or image stylizations) to larger samples of the same or similar materials. For training, we leverage images of the material taken under multiple illuminations and a dedicated data augmentation policy, making the transfer robust to novel illumination conditions and affine deformations. Our model relies on a supervised image-to-image translation framework and is agnostic to the transferred domain; we showcase a semantic segmentation, a normal map, and a stylization. Following an image analogies approach, the method only requires the training data to contain the same visual structures as the input guidance. Our approach works at interactive rates, making it suitable for material edit applications. We thoroughly evaluate our learning methodology in a controlled setup providing quantitative measures of performance. Last, we demonstrate that training the model on a single material is enough to generalize to materials of the same type without the need for massive datasets.
Robust Active Learning: Sample-Efficient Training of Robust Deep Learning Models
Venue: arXiv cs.LG 2021
Year: 2021

Active learning is an established technique to reduce the labeling cost to build high-quality machine learning models. A core component of active learning is the acquisition function that determines which data should be selected to annotate. State-of-the-art acquisition functions -- and more largely, active learning techniques -- have been designed to maximize the clean performance (e.g. accuracy) and have disregarded robustness, an important quality property that has received increasing attention. Active learning, therefore, produces models that are accurate but not robust. In this paper, we propose \emph{robust active learning}, an active learning process that integrates adversarial training -- the most established method to produce robust models. Via an empirical study on 11 acquisition functions, 4 datasets, 6 DNN architectures, and 15105 trained DNNs, we show that robust active learning can produce models with the robustness (accuracy on adversarial examples) ranging from 2.35\% to 63.85\%, whereas standard active learning systematically achieves negligible robustness (less than 0.20\%). Our study also reveals, however, that the acquisition functions that perform well on accuracy are worse than random sampling when it comes to robustness. We, therefore, examine the reasons behind this and devise a new acquisition function that targets both clean performance and robustness. Our acquisition function -- named density-based robust sampling with entropy (DRE) -- outperforms the other acquisition functions (including random) in terms of robustness by up to 24.40\% (3.84\% than random particularly), while remaining competitive on accuracy. Additionally, we prove that DRE is applicable as a test selection metric for model retraining and stands out from all compared functions by up to 8.21\% robustness.
Improving Intention Detection in Single-Trial Classification through Fusion of EEG and Eye-tracker Data
Venue: arXiv cs.HC 2021
Year: 2021

Intention decoding is an indispensable procedure in hands-free human-computer interaction (HCI). Conventional eye-tracking system using single-model fixation duration possibly issues commands ignoring users' real expectation. In the current study, an eye-brain hybrid brain-computer interface (BCI) interaction system was introduced for intention detection through fusion of multi-modal eye-track and ERP (a measurement derived from EEG) features. Eye-track and EEG data were recorded from 64 healthy participants as they performed a 40-min customized free search task of a fixed target icon among 25 icons. The corresponding fixation duration of eye-tracking and ERP were extracted. Five previously-validated LDA-based classifiers (including RLDA, SWLDA, BLDA, SKLDA, and STDA) and the widely-used CNN method were adopted to verify the efficacy of feature fusion from both offline and pseudo-online analysis, and optimal approach was evaluated through modulating the training set and system response duration. Our study demonstrated that the input of multi-modal eye-track and ERP features achieved superior performance of intention detection in the single trial classification of active search task. And compared with single-model ERP feature, this new strategy also induced congruent accuracy across different classifiers. Moreover, in comparison with other classification methods, we found that the SKLDA exhibited the superior performance when fusing feature in offline test (ACC=0.8783, AUC=0.9004) and online simulation with different sample amount and duration length. In sum, the current study revealed a novel and effective approach for intention classification using eye-brain hybrid BCI, and further supported the real-life application of hands-free HCI in a more precise and stable manner.
Real-time Informative Surgical Skill Assessment with Gaussian Process Learning
Venue: arXiv cs.LG 2021
Year: 2021

Endoscopic Sinus and Skull Base Surgeries (ESSBSs) is a challenging and potentially dangerous surgical procedure, and objective skill assessment is the key components to improve the effectiveness of surgical training, to re-validate surgeons' skills, and to decrease surgical trauma and the complication rate in operating rooms. Because of the complexity of surgical procedures, the variation of operation styles, and the fast development of new surgical skills, the surgical skill assessment remains a challenging problem. This work presents a novel Gaussian Process Learning-based heuristic automatic objective surgical skill assessment method for ESSBSs. Different with classical surgical skill assessment algorithms, the proposed method 1) utilizes the kinematic features in surgical instrument relative movements, instead of using specific surgical tasks or the statistics to assess skills in real-time; 2) provide informative feedback, instead of a summative scores; 3) has the ability to incrementally learn from new data, instead of depending on a fixed dataset. The proposed method projects the instrument movements into the endoscope coordinate to reduce the data dimensionality. It then extracts the kinematic features of the projected data and learns the relationship between surgical skill levels and the features with the Gaussian Process learning technique. The proposed method was verified in full endoscopic skull base and sinus surgeries on cadavers. These surgeries have different pathology, requires different treatment and has different complexities. The experimental results show that the proposed method reaches 100\% prediction precision for complete surgical procedures and 90\% precision for real-time prediction assessment.