OpenRaiser/Intern-Atlas huggingface dataset:
Multivariate Time Series Classification with Hierarchical Variational Graph Pooling
Venue: arXiv cs.LG 2021
Year: 2021

With the advancement of sensing technology, multivariate time series classification (MTSC) has recently received considerable attention. Existing deep learning-based MTSC techniques, which mostly rely on convolutional or recurrent neural networks, are primarily concerned with the temporal dependency of single time series. As a result, they struggle to express pairwise dependencies among multivariate variables directly. Furthermore, current spatial-temporal modeling (e.g., graph classification) methodologies based on Graph Neural Networks (GNNs) are inherently flat and cannot aggregate hub data in a hierarchical manner. To address these limitations, we propose a novel graph pooling-based framework MTPool to obtain the expressive global representation of MTS. We first convert MTS slices to graphs by utilizing interactions of variables via graph structure learning module and attain the spatial-temporal graph node features via temporal convolutional module. To get global graph-level representation, we design an "encoder-decoder" based variational graph pooling module for creating adaptive centroids for cluster assignments. Then we combine GNNs and our proposed variational graph pooling layers for joint graph representation learning and graph coarsening, after which the graph is progressively coarsened to one node. At last, a differentiable classifier takes this coarsened representation to get the final predicted class. Experiments on ten benchmark datasets exhibit MTPool outperforms state-of-the-art strategies in the MTSC task.
TurboTransformers: An Efficient GPU Serving System For Transformer Models
Venue: arXiv cs.DC 2021
Year: 2021

The transformer is the most critical algorithm innovation of the Nature Language Processing (NLP) field in recent years. Unlike the Recurrent Neural Network (RNN) models, Transformers can process on dimensions of sequence lengths in parallel, therefore leading to better accuracy on long sequences. However, efficient deployments of them for online services in data centers equipped with GPUs are not easy. First, more computation introduced by transformer structures makes it more challenging to meet the latency and throughput constraints of serving. Second, NLP tasks take in sentences of variable length. The variability of input dimensions brings a severe problem to efficient memory management and serving optimization. This paper designed a transformer serving system called TurboTransformers, which consists of a computing runtime and a serving framework to solve the above challenges. Three innovative features make it stand out from other similar works. An efficient parallel algorithm is proposed for GPU-based batch reduction operations, like Softmax and LayerNorm, major hot spots besides BLAS routines. A memory allocation algorithm, which better balances the memory footprint and allocation/free efficiency, is designed for variable-length input situations. A serving framework equipped with a new batch scheduler using dynamic programming achieves the optimal throughput on variable-length requests. The system can achieve the state-of-the-art transformer model serving performance on GPU platforms and can be seamlessly integrated into your PyTorch code with a few lines of code.
New Sublinear Algorithms and Lower Bounds for LIS Estimation
Venue: arXiv cs.DS 2021
Year: 2021

Estimating the length of the longest increasing subsequence (LIS) in an array is a problem of fundamental importance. Despite the significance of the LIS estimation problem and the amount of attention it has received, there are important aspects of the problem that are not yet fully understood. There are no better lower bounds for LIS estimation than the obvious bounds implied by testing monotonicity (for adaptive or nonadaptive algorithms). In this paper, we give the first nontrivial lower bound on the complexity of LIS estimation, and also provide novel algorithms that complement our lower bound. Specifically, for every constant $ε\in (0,1)$, every nonadaptive algorithm that outputs an estimate of the length of the LIS in an array of length $n$ to within an additive error of $ε\cdot n$ has to make $\log^{Ω(\log (1/ε))} n)$ queries. Next, we design nonadaptive LIS estimation algorithms whose complexity decreases as the the number of distinct values, $r$, in the array decreases. We first present a simple algorithm that makes $\tilde{O}(r/ε^3)$ queries and approximates the LIS length with an additive error bounded by $εn$. We then use it to construct a nonadaptive algorithm with query complexity $\tilde{O}(\sqrt{r} \cdot \text{poly}(1/λ))$ that, for an array with LIS length at least $λn$, outputs a multiplicative $Ω(λ)$-approximation to the LIS length. Finally, we describe a nonadaptive erasure-resilient tester for sortedness, with query complexity $O(\log n)$. Our result implies that nonadaptive tolerant testing is strictly harder than nonadaptive erasure-resilient testing for the natural property of monotonicity.
Remote Electrical Tilt Optimization via Safe Reinforcement Learning
Venue: arXiv cs.LG 2021
Year: 2021

Remote Electrical Tilt (RET) optimization is an efficient method for adjusting the vertical tilt angle of Base Stations (BSs) antennas in order to optimize Key Performance Indicators (KPIs) of the network. Reinforcement Learning (RL) provides a powerful framework for RET optimization because of its self-learning capabilities and adaptivity to environmental changes. However, an RL agent may execute unsafe actions during the course of its interaction, i.e., actions resulting in undesired network performance degradation. Since the reliability of services is critical for Mobile Network Operators (MNOs), the prospect of performance degradation has prohibited the real-world deployment of RL methods for RET optimization. In this work, we model the RET optimization problem in the Safe Reinforcement Learning (SRL) framework with the goal of learning a tilt control strategy providing performance improvement guarantees with respect to a safe baseline. We leverage a recent SRL method, namely Safe Policy Improvement through Baseline Bootstrapping (SPIBB), to learn an improved policy from an offline dataset of interactions collected by the safe baseline. Our experiments show that the proposed approach is able to learn a safe and improved tilt update policy, providing a higher degree of reliability and potential for real-world network deployment.
Towards human-level performance on automatic pose estimation of infant spontaneous movements
Venue: arXiv cs.CV 2021
Year: 2021

Assessment of spontaneous movements can predict the long-term developmental disorders in high-risk infants. In order to develop algorithms for automated prediction of later disorders, highly precise localization of segments and joints by infant pose estimation is required. Four types of convolutional neural networks were trained and evaluated on a novel infant pose dataset, covering the large variation in 1 424 videos from a clinical international community. The localization performance of the networks was evaluated as the deviation between the estimated keypoint positions and human expert annotations. The computational efficiency was also assessed to determine the feasibility of the neural networks in clinical practice. The best performing neural network had a similar localization error to the inter-rater spread of human expert annotations, while still operating efficiently. Overall, the results of our study show that pose estimation of infant spontaneous movements has a great potential to support research initiatives on early detection of developmental disorders in children with perinatal brain injuries by quantifying infant movements from video recordings with human-level performance.
Quantifying Membership Privacy via Information Leakage
Venue: arXiv cs.IT 2021
Year: 2021

Machine learning models are known to memorize the unique properties of individual data points in a training set. This memorization capability can be exploited by several types of attacks to infer information about the training data, most notably, membership inference attacks. In this paper, we propose an approach based on information leakage for guaranteeing membership privacy. Specifically, we propose to use a conditional form of the notion of maximal leakage to quantify the information leaking about individual data entries in a dataset, i.e., the entrywise information leakage. We apply our privacy analysis to the Private Aggregation of Teacher Ensembles (PATE) framework for privacy-preserving classification of sensitive data and prove that the entrywise information leakage of its aggregation mechanism is Schur-concave when the injected noise has a log-concave probability density. The Schur-concavity of this leakage implies that increased consensus among teachers in labeling a query reduces its associated privacy cost. Finally, we derive upper bounds on the entrywise information leakage when the aggregation mechanism uses Laplace distributed noise.
Shape-Texture Debiased Neural Network Training
Venue: arXiv cs.CV 2021
Year: 2021

Shape and texture are two prominent and complementary cues for recognizing objects. Nonetheless, Convolutional Neural Networks are often biased towards either texture or shape, depending on the training dataset. Our ablation shows that such bias degenerates model performance. Motivated by this observation, we develop a simple algorithm for shape-texture debiased learning. To prevent models from exclusively attending on a single cue in representation learning, we augment training data with images with conflicting shape and texture information (eg, an image of chimpanzee shape but with lemon texture) and, most importantly, provide the corresponding supervisions from shape and texture simultaneously. Experiments show that our method successfully improves model performance on several image recognition benchmarks and adversarial robustness. For example, by training on ImageNet, it helps ResNet-152 achieve substantial improvements on ImageNet (+1.2%), ImageNet-A (+5.2%), ImageNet-C (+8.3%) and Stylized-ImageNet (+11.1%), and on defending against FGSM adversarial attacker on ImageNet (+14.4%). Our method also claims to be compatible with other advanced data augmentation strategies, eg, Mixup, and CutMix. The code is available here: https://github.com/LiYingwei/ShapeTextureDebiasedTraining.
Deep learning for detection and segmentation of artefact and disease instances in gastrointestinal endoscopy
Venue: arXiv cs.CV 2021
Year: 2021

The Endoscopy Computer Vision Challenge (EndoCV) is a crowd-sourcing initiative to address eminent problems in developing reliable computer aided detection and diagnosis endoscopy systems and suggest a pathway for clinical translation of technologies. Whilst endoscopy is a widely used diagnostic and treatment tool for hollow-organs, there are several core challenges often faced by endoscopists, mainly: 1) presence of multi-class artefacts that hinder their visual interpretation, and 2) difficulty in identifying subtle precancerous precursors and cancer abnormalities. Artefacts often affect the robustness of deep learning methods applied to the gastrointestinal tract organs as they can be confused with tissue of interest. EndoCV2020 challenges are designed to address research questions in these remits. In this paper, we present a summary of methods developed by the top 17 teams and provide an objective comparison of state-of-the-art methods and methods designed by the participants for two sub-challenges: i) artefact detection and segmentation (EAD2020), and ii) disease detection and segmentation (EDD2020). Multi-center, multi-organ, multi-class, and multi-modal clinical endoscopy datasets were compiled for both EAD2020 and EDD2020 sub-challenges. The out-of-sample generalization ability of detection algorithms was also evaluated. Whilst most teams focused on accuracy improvements, only a few methods hold credibility for clinical usability. The best performing teams provided solutions to tackle class imbalance, and variabilities in size, origin, modality and occurrences by exploring data augmentation, data fusion, and optimal class thresholding techniques.
Mixed data Deep Gaussian Mixture Model: A clustering model for mixed datasets
Venue: arXiv cs.LG 2021
Year: 2021

Clustering mixed data presents numerous challenges inherent to the very heterogeneous nature of the variables. A clustering algorithm should be able, despite of this heterogeneity, to extract discriminant pieces of information from the variables in order to design groups. In this work we introduce a multilayer architecture model-based clustering method called Mixed Deep Gaussian Mixture Model (MDGMM) that can be viewed as an automatic way to merge the clustering performed separately on continuous and non-continuous data. This architecture is flexible and can be adapted to mixed as well as to continuous or non-continuous data. In this sense we generalize Generalized Linear Latent Variable Models and Deep Gaussian Mixture Models. We also design a new initialisation strategy and a data driven method that selects the best specification of the model and the optimal number of clusters for a given dataset "on the fly". Besides, our model provides continuous low-dimensional representations of the data which can be a useful tool to visualize mixed datasets. Finally, we validate the performance of our approach comparing its results with state-of-the-art mixed data clustering models over several commonly used datasets.
GreedyFool: Multi-Factor Imperceptibility and Its Application to Designing a Black-box Adversarial Attack
Venue: arXiv cs.LG 2021
Year: 2021

Adversarial examples are well-designed input samples, in which perturbations are imperceptible to the human eyes, but easily mislead the output of deep neural networks (DNNs). Existing works synthesize adversarial examples by leveraging simple metrics to penalize perturbations, that lack sufficient consideration of the human visual system (HVS), which produces noticeable artifacts. To explore why the perturbations are visible, this paper summarizes four primary factors affecting the perceptibility of human eyes. Based on this investigation, we design a multi-factor metric MulFactorLoss for measuring the perceptual loss between benign examples and adversarial ones. In order to test the imperceptibility of the multi-factor metric, we propose a novel black-box adversarial attack that is referred to as GreedyFool. GreedyFool applies differential evolution to evaluate the effects of perturbed pixels on the confidence of a target DNN, and introduces greedy approximation to automatically generate adversarial perturbations. We conduct extensive experiments on the ImageNet and CIFRA-10 datasets and a comprehensive user study with 60 participants. The experimental results demonstrate that MulFactorLoss is a more imperceptible metric than the existing pixelwise metrics, and GreedyFool achieves a 100% success rate in a black-box manner.
Basic principles and concept design of a real-time clinical decision support system for managing medical emergencies on missions to Mars
Venue: arXiv cs.CY 2021
Year: 2021

Space agencies and private companies prepare the beginning of human space exploration for the 2030s with missions to put the first human on the Mars surface. The absence of gravity and radiation, along with distance, isolation and hostile environments, are expected to increase medical events where previously unseen manifestations may arise. The current healthcare strategy based on telemedicine and the possibility to stabilize and transport the injured crewmember to a terrestrial definitive medical facility is not applicable in exploration class missions. Therefore, the need for deploying the full autonomous capability to solve medical emergencies may guide the design of future onboard healthcare systems. We present ten basic principles and concept design of a software suite to bring onboard decision support to help the crew dealing with medical emergencies taking into consideration physiological disturbances in space and spaceflight restrictions. 1) give real-time support for emergency medical decision making, 2) give patient-specific advice for executive problem-solving, 3) take into account available information from life support and monitoring of crewmembers, 4) be fully autonomous from remote facilities, 5) continuously adapt predictions to physiological disturbance and changing conditions, 6) optimize emergency medical decision making in terms of mission fundamental priorities, 7) take into account medical supplies and equipment on board, 8) apply health standards for the level of care V, 9) implement ethics responsibilities for spaceflights, and 10) apply ethical standards for artificial intelligence. Based on these principles, we propose an autonomous clinical decision support system (CDSS) to provide real-time advice for emergency medical interventions on board of space exploration missions.
Peridynamics-based discrete element method (PeriDEM) model of granular systems involving breakage of arbitrarily shaped particles
Venue: arXiv cs.CE 2021
Year: 2021

Usage, manipulation, transport, delivery, and mixing of granular or particulate media, comprised of spherical or polyhedral particles, is commonly encountered in industrial sectors of construction (cement and rock fragments), pharmaceutics (tablets), and transportation (ballast). Elucidating particulate media's behavior in concert with particle attrition (i.e., particle wear and subsequent particle fragmentation) is essential for predicting the performance and increasing the efficiency of engineering systems using such media. Discrete element method (DEM) based techniques can describe the interaction between particles but cannot model intra-particle deformation, especially intra-particle fracture. On the other hand, peridynamics provides the means to account for intra-particle deformation and fracture due to contact forces between particles. The present study proposes a hybrid model referred to as \textit{PeriDEM} that combines the advantages of peridynamics and DEM. The model parameters can be tuned to achieve desired DEM contact forces, damping effects, and intra-particle stiffness. Two particle impacts and compressive behavior of multi-particle systems are thoroughly investigated. The model can account for any arbitrarily shaped particle in general. Spherical, hexagonal, and non-convex particle shapes are simulated in the present study. The effect of mesh resolution on intra-particle peridynamics is explicitly studied. The proposed hybrid model opens a new avenue to explore the complicated interactions encountered in discrete particle dynamics that involve the formation of force chains, particle interlocking, particle attrition, wear, and the eventual breakage.
Discriminating Equivalent Algorithms via Relative Performance
Venue: arXiv cs.PF 2021
Year: 2021

In scientific computing, it is common that a mathematical expression can be computed by many different algorithms (sometimes over hundreds), each identifying a specific sequence of library calls. Although mathematically equivalent, those algorithms might exhibit significant differences in terms of performance. However in practice, due to fluctuations, there is not one algorithm that consistently performs noticeably better than the rest. For this reason, with this work we aim to identify not the one best algorithm, but the subset of algorithms that are reliably faster than the rest. To this end, instead of using the usual approach of quantifying the performance of an algorithm in absolute terms, we present a measurement-based clustering approach to sort the algorithms into equivalence (or performance) classes using pair-wise comparisons. We show that this approach, based on relative performance, leads to robust identification of the fastest algorithms even under noisy system conditions. Furthermore, it enables the development of practical machine learning models for automatic algorithm selection.
Harnessing Uncertainty in Domain Adaptation for MRI Prostate Lesion Segmentation
Venue: arXiv cs.CV 2021
Year: 2021

The need for training data can impede the adoption of novel imaging modalities for learning-based medical image analysis. Domain adaptation methods partially mitigate this problem by translating training data from a related source domain to a novel target domain, but typically assume that a one-to-one translation is possible. Our work addresses the challenge of adapting to a more informative target domain where multiple target samples can emerge from a single source sample. In particular we consider translating from mp-MRI to VERDICT, a richer MRI modality involving an optimized acquisition protocol for cancer characterization. We explicitly account for the inherent uncertainty of this mapping and exploit it to generate multiple outputs conditioned on a single input. Our results show that this allows us to extract systematically better image representations for the target domain, when used in tandem with both simple, CycleGAN-based baselines, as well as more powerful approaches that integrate discriminative segmentation losses and/or residual adapters. When compared to its deterministic counterparts, our approach yields substantial improvements across a broad range of dataset sizes, increasingly strong baselines, and evaluation measures.
Formalizing Trust in Artificial Intelligence: Prerequisites, Causes and Goals of Human Trust in AI
Venue: arXiv cs.AI 2021
Year: 2021

Trust is a central component of the interaction between people and AI, in that 'incorrect' levels of trust may cause misuse, abuse or disuse of the technology. But what, precisely, is the nature of trust in AI? What are the prerequisites and goals of the cognitive mechanism of trust, and how can we promote them, or assess whether they are being satisfied in a given interaction? This work aims to answer these questions. We discuss a model of trust inspired by, but not identical to, sociology's interpersonal trust (i.e., trust between people). This model rests on two key properties of the vulnerability of the user and the ability to anticipate the impact of the AI model's decisions. We incorporate a formalization of 'contractual trust', such that trust between a user and an AI is trust that some implicit or explicit contract will hold, and a formalization of 'trustworthiness' (which detaches from the notion of trustworthiness in sociology), and with it concepts of 'warranted' and 'unwarranted' trust. We then present the possible causes of warranted trust as intrinsic reasoning and extrinsic behavior, and discuss how to design trustworthy AI, how to evaluate whether trust has manifested, and whether it is warranted. Finally, we elucidate the connection between trust and XAI using our formalization.
Asynchronous ε-Greedy Bayesian Optimisation
Venue: arXiv cs.LG 2021
Year: 2021

Batch Bayesian optimisation (BO) is a successful technique for the optimisation of expensive black-box functions. Asynchronous BO can reduce wallclock time by starting a new evaluation as soon as another finishes, thus maximising resource utilisation. To maximise resource allocation, we develop a novel asynchronous BO method, AEGiS (Asynchronous $ε$-Greedy Global Search) that combines greedy search, exploiting the surrogate's mean prediction, with Thompson sampling and random selection from the approximate Pareto set describing the trade-off between exploitation (surrogate mean prediction) and exploration (surrogate posterior variance). We demonstrate empirically the efficacy of AEGiS on synthetic benchmark problems, meta-surrogate hyperparameter tuning problems and real-world problems, showing that AEGiS generally outperforms existing methods for asynchronous BO. When a single worker is available performance is no worse than BO using expected improvement.