OpenRaiser/Intern-Atlas huggingface dataset:
Investigations on the inference optimization techniques and their impact on multiple hardware platforms for Semantic Segmentation
Venue: arXiv cs.CV 2019
Year: 2019

In this work, the task of pixel-wise semantic segmentation in the context of self-driving with a goal to reduce the inference time is explored. Fully Convolutional Network (FCN-8s, FCN-16s, and FCN-32s) with a VGG16 encoder architecture and skip connections is trained and validated on the Cityscapes dataset. Numerical investigations are carried out for several inference optimization techniques built into TensorFlow and TensorRT to quantify their impact on the inference time and network size. Finally, the trained network is ported on to an embedded platform (Nvidia Jetson TX1) and the inference time, as well as the total energy consumed for inference across hardware platforms, are compared.
SAT-Encodings for Treecut Width and Treedepth
Venue: arXiv cs.DS 2019
Year: 2019

In this paper we propose, implement, and test the first practical decomposition algorithms for the width parameters treecut width and treedepth. These two parameters have recently gained a lot of attention in the theoretical research community as they offer the algorithmic advantage over treewidth by supporting so-called fixed-parameter algorithms for certain problems that are not fixed-parameter tractable with respect to treewidth. However, the existing research has mostly been theoretical. A main obstacle for any practical or experimental use of these two width parameters is the lack of any practical or implemented algorithm for actually computing the associated decompositions. We address this obstacle by providing the first practical decomposition algorithms. Our approach for computing treecut width and treedepth decompositions is based on efficient encodings of these decomposition methods to the propositional satisfiability problem (SAT). Once an encoding is generated, any satisfiability solver can be used to find the decomposition. Moreover, we propose new characterisations for treecut width and treedepth that are based on sequences of partitions of the vertex set, a method that was pioneered for clique-width. We implemented and systematically tested our encodings on various benchmark instances, including famous named graphs and random graphs of various density. It turned out that for the considered width parameters, our partition-based SAT encoding even outperforms the best existing SAT encoding for treewidth.
Deep Networks with Adaptive Nyström Approximation
Venue: arXiv cs.LG 2019
Year: 2019

Recent work has focused on combining kernel methods and deep learning to exploit the best of the two approaches. Here, we introduce a new architecture of neural networks in which we replace the top dense layers of standard convolutional architectures with an approximation of a kernel function by relying on the Nystr{ö}m approximation. Our approach is easy and highly flexible. It is compatible with any kernel function and it allows exploiting multiple kernels. We show that our architecture has the same performance than standard architecture on datasets like SVHN and CIFAR100. One benefit of the method lies in its limited number of learnable parameters which makes it particularly suited for small training set sizes, e.g. from 5 to 20 samples per class.
Orthogonal Wasserstein GANs
Venue: arXiv cs.LG 2019
Year: 2019

Wasserstein-GANs have been introduced to address the deficiencies of generative adversarial networks (GANs) regarding the problems of vanishing gradients and mode collapse during the training, leading to improved convergence behaviour and improved image quality. However, Wasserstein-GANs require the discriminator to be Lipschitz continuous. In current state-of-the-art Wasserstein-GANs this constraint is enforced via gradient norm regularization. In this paper, we demonstrate that this regularization does not encourage a broad distribution of spectral-values in the discriminator weights, hence resulting in less fidelity in the learned distribution. We therefore investigate the possibility of substituting this Lipschitz constraint with an orthogonality constraint on the weight matrices. We compare three different weight orthogonalization techniques with regards to their convergence properties, their ability to ensure the Lipschitz condition and the achieved quality of the learned distribution. In addition, we provide a comparison to Wasserstein-GANs trained with current state-of-the-art methods, where we demonstrate the potential of solely using orthogonality-based regularization. In this context, we propose an improved training procedure for Wasserstein-GANs which utilizes orthogonalization to further increase its generalization capability. Finally, we provide a novel metric to evaluate the generalization capabilities of the discriminators of different Wasserstein-GANs.
Sparsely Grouped Input Variables for Neural Networks
Venue: arXiv cs.LG 2019
Year: 2019

In genomic analysis, biomarker discovery, image recognition, and other systems involving machine learning, input variables can often be organized into different groups by their source or semantic category. Eliminating some groups of variables can expedite the process of data acquisition and avoid over-fitting. Researchers have used the group lasso to ensure group sparsity in linear models and have extended it to create compact neural networks in meta-learning. Different from previous studies, we use multi-layer non-linear neural networks to find sparse groups for input variables. We propose a new loss function to regularize parameters for grouped input variables, design a new optimization algorithm for this loss function, and test these methods in three real-world settings. We achieve group sparsity for three datasets, maintaining satisfying results while excluding one nucleotide position from an RNA splicing experiment, excluding 89.9% of stimuli from an eye-tracking experiment, and excluding 60% of image rows from an experiment on the MNIST dataset.
Method and Dataset Mining in Scientific Papers
Venue: arXiv cs.LG 2019
Year: 2019

Literature analysis facilitates researchers better understanding the development of science and technology. The conventional literature analysis focuses on the topics, authors, abstracts, keywords, references, etc., and rarely pays attention to the content of papers. In the field of machine learning, the involved methods (M) and datasets (D) are key information in papers. The extraction and mining of M and D are useful for discipline analysis and algorithm recommendation. In this paper, we propose a novel entity recognition model, called MDER, and constructe datasets from the papers of the PAKDD conferences (2009-2019). Some preliminary experiments are conducted to assess the extraction performance and the mining results are visualized.
Proper Hierarchies in Polylogarithmic Time and Absence of Complete Problems
Venue: arXiv cs.CC 2019
Year: 2019

The polylogarithmic time hierarchy structures sub-linear time complexity. In recent work it was shown that all classes $\tildeΣ_{m}^{\mathit{plog}}$ or $\tildeΠ_{m}^{\mathit{plog}}$ ($m \in \mathbb{N}$) in this hierarchy can be captured by semantically restricted fragments of second-order logic. In this paper the descriptive complexity theory of polylogarithmic time is taken further showing that there are strict hierarchies inside each of the classes of the hierarchy. A straightforward consequence of this result is that there are no complete problems for these complexity classes, not even under polynomial time reductions. As another consequence we show that the polylogarithmic time hierarchy itself is strict.
A Proposed Practical Problem-Solving Framework for Multi-Stakeholder Initiatives in Socio-Ecological Systems Based on a Model of the Human Cognitive Problem-Solving Process
Venue: arXiv cs.CY 2019
Year: 2019

A practical problem-solving framework is proposed for multi-stakeholder initiative (MSI) problem-solving processes involving socio-ecological systems (SES), so-called wicked problems, based on insights borrowed from a model of the individual human, cognitive problem-solving process. The disciplined facilitation of the multi-stakeholder process, adhering to the steps recognized in the individual process, is meant to reduce confusion and conflict. Obtaining a one- to three-sentence human-language description of the desired system state, as a first step, is proposed in multi-stakeholder initiatives for reasons of goal congruence and trust building. The systematic, stakeholder-driven subdivision of obstacles into larger numbers of simpler obstacles is proposed in order to obtain a list of "what needs to be done," inviting a more rational and goal-driven conversation with resource providers. Finally, obtaining and maintaining stakeholder buy-in over the course of the problem-solving effort is reinforced by reflecting back to all stakeholders, as a communication device, a dynamic, visual problem-solving model, taking into account the diversity of cognitive and individual capacities within the stakeholder group in its presentation. Mathematical parameters for gauging applicability of the proposed framework are discussed.
Lossless Size Reduction for Integer Least Squares with Application to Sphere Decoding
Venue: arXiv cs.IT 2019
Year: 2019

Minimum achievable complexity (MAC) for a maximum likelihood (ML) performance-achieving detection algorithm is derived. Using the derived MAC, we prove that the conventional sphere decoding (SD) algorithms suffer from an inherent weakness at low SNRs. To find a solution for the low SNR deficiency, we analyze the effect of zero-forcing (ZF) and minimum mean square error (MMSE) detected symbols on the MAC and demonstrate that although they both improve the SD algorithm in terms of the computational complexity, the MMSE point has a vital difference at low SNRs. By exploiting the information provided by the MMSE method, we prove the existence of a lossless size reduction which can be interpreted as the feasibility of a detection method which is capable of detecting the ML symbol without visiting any nodes at low and high SNRs. We also propose a lossless size reduction-aided detection method which achieves the promised complexity bounds marginally and reduces the overall computational complexity significantly, while obtaining the ML performance. The theoretical analysis is corroborated with numerical simulations.
ModelHub.AI: Dissemination Platform for Deep Learning Models
Venue: arXiv cs.LG 2019
Year: 2019

Recent advances in artificial intelligence research have led to a profusion of studies that apply deep learning to problems in image analysis and natural language processing among others. Additionally, the availability of open-source computational frameworks has lowered the barriers to implementing state-of-the-art methods across multiple domains. Albeit leading to major performance breakthroughs in some tasks, effective dissemination of deep learning algorithms remains challenging, inhibiting reproducibility and benchmarking studies, impeding further validation, and ultimately hindering their effectiveness in the cumulative scientific progress. In developing a platform for sharing research outputs, we present ModelHub.AI (www.modelhub.ai), a community-driven container-based software engine and platform for the structured dissemination of deep learning models. For contributors, the engine controls data flow throughout the inference cycle, while the contributor-facing standard template exposes model-specific functions including inference, as well as pre- and post-processing. Python and RESTful Application programming interfaces (APIs) enable users to interact with models hosted on ModelHub.AI and allows both researchers and developers to utilize models out-of-the-box. ModelHub.AI is domain-, data-, and framework-agnostic, catering to different workflows and contributors' preferences.
A Benchmarking of DCM Based Architectures for Position, Velocity and Torque Controlled Humanoid Robots
Venue: arXiv cs.RO 2019
Year: 2019

This paper contributes towards the benchmarking of control architectures for bipedal robot locomotion. It considers architectures that are based on the Divergent Component of Motion (DCM) and composed of three main layers: trajectory optimization, simplified model control, and whole-body QP control layer. While the first two layers use simplified robot models, the whole-body QP control layer uses a complete robot model to produce either desired positions, velocities, or torques inputs at the joint-level. This paper then compares two implementations of the simplified model control layer, which are tested with position, velocity, and torque control modes for the whole-body QP control layer. In particular, both an instantaneous and a Receding Horizon controller are presented for the simplified model control layer. We show also that one of the proposed architectures allows the humanoid robot iCub to achieve a forward walking velocity of 0.3372 meters per second, which is the highest walking velocity achieved by the iCub robot.
An Optimized and Energy-Efficient Parallel Implementation of Non-Iteratively Trained Recurrent Neural Networks
Venue: arXiv cs.LG 2019
Year: 2019

Recurrent neural networks (RNN) have been successfully applied to various sequential decision-making tasks, natural language processing applications, and time-series predictions. Such networks are usually trained through back-propagation through time (BPTT) which is prohibitively expensive, especially when the length of the time dependencies and the number of hidden neurons increase. To reduce the training time, extreme learning machines (ELMs) have been recently applied to RNN training, reaching a 99\% speedup on some applications. Due to its non-iterative nature, ELM training, when parallelized, has the potential to reach higher speedups than BPTT. In this work, we present \opt, an optimized parallel RNN training algorithm based on ELM that takes advantage of the GPU shared memory and of parallel QR factorization algorithms to efficiently reach optimal solutions. The theoretical analysis of the proposed algorithm is presented on six RNN architectures, including LSTM and GRU, and its performance is empirically tested on ten time-series prediction applications. \opt~is shown to reach up to 845 times speedup over its sequential counterpart and to require up to 20x less time to train than parallel BPTT.
Transflow Learning: Repurposing Flow Models Without Retraining
Venue: arXiv cs.LG 2019
Year: 2019

It is well known that deep generative models have a rich latent space, and that it is possible to smoothly manipulate their outputs by traversing this latent space. Recently, architectures have emerged that allow for more complex manipulations, such as making an image look as though it were from a different class, or painted in a certain style. These methods typically require large amounts of training in order to learn a single class of manipulations. We present Transflow Learning, a method for transforming a pre-trained generative model so that its outputs more closely resemble data that we provide afterwards. In contrast to previous methods, Transflow Learning does not require any training at all, and instead warps the probability distribution from which we sample latent vectors using Bayesian inference. Transflow Learning can be used to solve a wide variety of tasks, such as neural style transfer and few-shot classification.
Location histogram privacy by sensitive location hiding and target histogram avoidance/resemblance (extended version)
Venue: arXiv cs.CR 2019
Year: 2019

A location histogram is comprised of the number of times a user has visited locations as they move in an area of interest, and it is often obtained from the user in applications such as recommendation and advertising. However, a location histogram that leaves a user's computer or device may threaten privacy when it contains visits to locations that the user does not want to disclose (sensitive locations), or when it can be used to profile the user in a way that leads to price discrimination and unsolicited advertising. Our work introduces two privacy notions to protect a location histogram from these threats: sensitive location hiding, which aims at concealing all visits to sensitive locations, and target avoidance/resemblance, which aims at concealing the similarity/dissimilarity of the user's histogram to a target histogram that corresponds to an undesired/desired profile. We formulate an optimization problem around each notion: Sensitive Location Hiding (SLH), which seeks to construct a histogram that is as similar as possible to the user's histogram but associates all visits with nonsensitive locations, and Target Avoidance/Resemblance (TA/TR), which seeks to construct a histogram that is as dissimilar/similar as possible to a given target histogram but remains useful for getting a good response from the application that analyzes the histogram. We develop an optimal algorithm for each notion and also develop a greedy heuristic for the TA/TR problem. Our experiments demonstrate that all algorithms are effective at preserving the distribution of locations in a histogram and the quality of location recommendation. They also demonstrate that the heuristic produces near-optimal solutions while being orders of magnitude faster than the optimal algorithm for TA/TR.
Inapproximability of Additive Weak Contraction under SSEH and Strong UGC
Venue: arXiv cs.CC 2019
Year: 2019

Succinct representations of a graph have been objects of central study in computer science for decades. In this paper, we study the operation called \emph{Distance Preserving Graph Contractions}, which was introduced by Bernstein et al. (ITCS, 2018). This operation gives a minor as a succinct representation of a graph that preserves all the distances of the original (up to some factor). The graph minor given from contractions can be seen as a dual of spanners as the distances can only shrink (while distances are stretched in the case of spanners). Bernstein et al. proved inapproximability results for the problems of finding maximum subset of edges that yields distance preserving graph contractions for almost major classes of graphs except for that of Additive Weak Contraction. The main result in this paper is filling the gap in the paper of Bernstein et al. We show that the Maximum Additive Weak Contraction problem on a graph with $n$ vertices is inapproximable up to a factor of $n^{1-ε}$ for every constant $ε>0$. Our hardness results follow from that of the Maximum Edge Biclique (\textsc{MEB}) problem whose inapproximability of $n^{1-ε}$ has been recently shown by Manurangsi (ICALP, 2017) under the \textsc{Small Set Expansion Hypothesis (SSEH)} and by Bhangale et al. (APPROX, 2016) under the \textsc{Strong Unique Games Conjecture (SUGC)} (both results also assume $\mathrm{NP}\not\subseteq\mathrm{BPP}$).
Fuzzy approach on modelling cyber attacks patterns on data transfer in industrial control systems
Venue: arXiv cs.CR 2019
Year: 2019

Cybersecurity of industrial control system is a very complex and challenging research topic, due to the integration of these systems in national critical infrastructures. The control systems are now interconnected in industrial networks and frequently to the Internet. In this context they are becoming targets of various cyber attacks conducted by malicious people such as hackers, script kiddies, industrial spies and even foreign armies and intelligence agencies. In this paper the authors propose a way to model the most frequent attacker profiles and to estimate the success rate of an attack conducted in given conditions. The authors use a fuzzy approach for generating attacker profiles based on attacker attributes such as knowledge, technical resources and motivation. The attack success rate is obtained by using another fuzzy inference system that analyzes the attacker profile and system intrinsic characteristics.
Exploiting Motion Information from Unlabeled Videos for Static Image Action Recognition
Venue: arXiv cs.CV 2019
Year: 2019

Static image action recognition, which aims to recognize action based on a single image, usually relies on expensive human labeling effort such as adequate labeled action images and large-scale labeled image dataset. In contrast, abundant unlabeled videos can be economically obtained. Therefore, several works have explored using unlabeled videos to facilitate image action recognition, which can be categorized into the following two groups: (a) enhance visual representations of action images with a designed proxy task on unlabeled videos, which falls into the scope of self-supervised learning; (b) generate auxiliary representations for action images with the generator learned from unlabeled videos. In this paper, we integrate the above two strategies in a unified framework, which consists of Visual Representation Enhancement (VRE) module and Motion Representation Augmentation (MRA) module. Specifically, the VRE module includes a proxy task which imposes pseudo motion label constraint and temporal coherence constraint on unlabeled videos, while the MRA module could predict the motion information of a static action image by exploiting unlabeled videos. We demonstrate the superiority of our framework based on four benchmark human action datasets with limited labeled data.
Topic-aware chatbot using Recurrent Neural Networks and Nonnegative Matrix Factorization
Venue: arXiv cs.CL 2019
Year: 2019

We propose a novel model for a topic-aware chatbot by combining the traditional Recurrent Neural Network (RNN) encoder-decoder model with a topic attention layer based on Nonnegative Matrix Factorization (NMF). After learning topic vectors from an auxiliary text corpus via NMF, the decoder is trained so that it is more likely to sample response words from the most correlated topic vectors. One of the main advantages in our architecture is that the user can easily switch the NMF-learned topic vectors so that the chatbot obtains desired topic-awareness. We demonstrate our model by training on a single conversational data set which is then augmented with topic matrices learned from different auxiliary data sets. We show that our topic-aware chatbot not only outperforms the non-topic counterpart, but also that each topic-aware model qualitatively and contextually gives the most relevant answer depending on the topic of question.
A SVBRDF Modeling Pipeline using Pixel Clustering
Venue: arXiv cs.GR 2019
Year: 2019

We present a pipeline for modeling spatially varying BRDFs (svBRDFs) of planar materials which only requires a mobile phone for data acquisition. With a minimum of two photos under the ambient and point light source, our pipeline produces svBRDF parameters, a normal map and a tangent map for the material sample. The BRDF fitting is achieved via a pixel clustering strategy and an optimization based scheme. Our method is light-weight, easy-to-use and capable of producing high-quality BRDF textures.
Preserving Patient Privacy while Training a Predictive Model of In-hospital Mortality
Venue: arXiv cs.LG 2019
Year: 2019

Machine learning models can be used for pattern recognition in medical data in order to improve patient outcomes, such as the prediction of in-hospital mortality. Deep learning models, in particular, require large amounts of data for model training. However, the data is often collected at different hospitals and sharing is restricted due to patient privacy concerns. In this paper, we aimed to demonstrate the potential of distributed training in achieving state-of-the-art performance while maintaining data privacy. Our results show that training the model in the federated learning framework leads to comparable performance to the traditional centralised setting. We also suggest several considerations for the success of such frameworks in future work.