OpenRaiser/Intern-Atlas huggingface dataset:
Stacked Wasserstein Autoencoder
Venue: arXiv cs.CV 2019
Year: 2019

Approximating distributions over complicated manifolds, such as natural images, are conceptually attractive. The deep latent variable model, trained using variational autoencoders and generative adversarial networks, is now a key technique for representation learning. However, it is difficult to unify these two models for exact latent-variable inference and parallelize both reconstruction and sampling, partly due to the regularization under the latent variables, to match a simple explicit prior distribution. These approaches are prone to be oversimplified, and can only characterize a few modes of the true distribution. Based on the recently proposed Wasserstein autoencoder (WAE) with a new regularization as an optimal transport. The paper proposes a stacked Wasserstein autoencoder (SWAE) to learn a deep latent variable model. SWAE is a hierarchical model, which relaxes the optimal transport constraints at two stages. At the first stage, the SWAE flexibly learns a representation distribution, i.e., the encoded prior; and at the second stage, the encoded representation distribution is approximated with a latent variable model under the regularization encouraging the latent distribution to match the explicit prior. This model allows us to generate natural textual outputs as well as perform manipulations in the latent space to induce changes in the output space. Both quantitative and qualitative results demonstrate the superior performance of SWAE compared with the state-of-the-art approaches in terms of faithful reconstruction and generation quality.
Continual Learning in Neural Networks
Venue: arXiv cs.LG 2019
Year: 2019

Artificial neural networks have exceeded human-level performance in accomplishing several individual tasks (e.g. voice recognition, object recognition, and video games). However, such success remains modest compared to human intelligence that can learn and perform an unlimited number of tasks. Humans' ability of learning and accumulating knowledge over their lifetime is an essential aspect of their intelligence. Continual machine learning aims at a higher level of machine intelligence through providing the artificial agents with the ability to learn online from a non-stationary and never-ending stream of data. A key component of such a never-ending learning process is to overcome the catastrophic forgetting of previously seen data, a problem that neural networks are well known to suffer from. The work described in this thesis has been dedicated to the investigation of continual learning and solutions to mitigate the forgetting phenomena in neural networks. To approach the continual learning problem, we first assume a task incremental setting where tasks are received one at a time and data from previous tasks are not stored. Since the task incremental setting can't be assumed in all continual learning scenarios, we also study the more general online continual setting. We consider an infinite stream of data drawn from a non-stationary distribution with a supervisory or self-supervisory training signal. The proposed methods in this thesis have tackled important aspects of continual learning. They were evaluated on different benchmarks and over various learning sequences. Advances in the state of the art of continual learning have been shown and challenges for bringing continual learning into application were critically identified.
Algorithmic Probability-guided Supervised Machine Learning on Non-differentiable Spaces
Venue: arXiv cs.LG 2019
Year: 2019

We show how complexity theory can be introduced in machine learning to help bring together apparently disparate areas of current research. We show that this new approach requires less training data and is more generalizable as it shows greater resilience to random attacks. We investigate the shape of the discrete algorithmic space when performing regression or classification using a loss function parametrized by algorithmic complexity, demonstrating that the property of differentiation is not necessary to achieve results similar to those obtained using differentiable programming approaches such as deep learning. In doing so we use examples which enable the two approaches to be compared (small, given the computational power required for estimations of algorithmic complexity). We find and report that (i) machine learning can successfully be performed on a non-smooth surface using algorithmic complexity; (ii) that parameter solutions can be found using an algorithmic-probability classifier, establishing a bridge between a fundamentally discrete theory of computability and a fundamentally continuous mathematical theory of optimization methods; (iii) a formulation of an algorithmically directed search technique in non-smooth manifolds can be defined and conducted; (iv) exploitation techniques and numerical methods for algorithmic search to navigate these discrete non-differentiable spaces can be performed; in application of the (a) identification of generative rules from data observations; (b) solutions to image classification problems more resilient against pixel attacks compared to neural networks; (c) identification of equation parameters from a small data-set in the presence of noise in continuous ODE system problem, (d) classification of Boolean NK networks by (1) network topology, (2) underlying Boolean function, and (3) number of incoming edges.
ViP: Video Platform for PyTorch
Venue: arXiv cs.CV 2019
Year: 2019

This work presents the Video Platform for PyTorch (ViP), a deep learning-based framework designed to handle and extend to any problem domain based on videos. ViP supports (1) a single unified interface applicable to all video problem domains, (2) quick prototyping of video models, (3) executing large-batch operations with reduced memory consumption, and (4) easy and reproducible experimental setups. ViP's core functionality is built with flexibility and modularity in mind to allow for smooth data flow between different parts of the platform and benchmarking against existing methods. In providing a software platform that supports multiple video-based problem domains, we allow for more cross-pollination of models, ideas and stronger generalization in the video understanding research community.
Work Stealing Simulator
Venue: arXiv cs.DC 2019
Year: 2019

We present in this paper a Work Stealing lightweight PYTHON simulator. Our simulator is used to execute an application (list of tasks with or without dependencies), on a multiple processors platform linked by specific topology. We first give an overview of the different variants of the work stealing algorithm, then we present the architecture of our light Work Stealing simulator. Its architecture facilitates the development of other types of applications and other topologies for interconnecting the processors. We present the use cases of the simulator and the different types of results.
Farkas layers: don't shift the data, fix the geometry
Venue: arXiv cs.LG 2019
Year: 2019

Successfully training deep neural networks often requires either batch normalization, appropriate weight initialization, both of which come with their own challenges. We propose an alternative, geometrically motivated method for training. Using elementary results from linear programming, we introduce Farkas layers: a method that ensures at least one neuron is active at a given layer. Focusing on residual networks with ReLU activation, we empirically demonstrate a significant improvement in training capacity in the absence of batch normalization or methods of initialization across a broad range of network sizes on benchmark datasets.
Trading off Complexity for Expressiveness in Programming Languages: Visions and Preliminary Experiences
Venue: arXiv cs.PL 2019
Year: 2019

When programming resource-scarce embedded smart devices, the designer often requires both the low-level system programming features of a language such as C and higher level capability typical of a language like Java. The choice of a particular language typically implies trade offs between conflicting design goals such as performance, costs, and overheads. The large variety of languages, virtual machines, and translators provides the designer with a dense trade off space, ranging from minimalistic to rich full-fledged approaches, but once a choice is made it is often difficult for the designer to revise it. In this work we propose a system of light-weighted and modular extensions as a method to flexibly reshape the target programming language as needed, adding only those application layer features that match the current design goals. In so doing complexity is made transparent, but not hidden: While the programmer can benefit of higher level constructs, the designer can deal with modular building blocks each characterized by a certain algorithmic complexity and therefore each accountable for a given share of the overhead. As a result the designer is given a finer control on the amount of resources that are consumed by the run-time executive of the chosen programming language.
Synthesizing Credit Card Transactions
Venue: arXiv cs.DB 2019
Year: 2019

Two elements have been essential to AI's recent boom: (1) deep neural nets and the theory and practice behind them; and (2) cloud computing with its abundant labeled data and large computing resources. Abundant labeled data is available for key domains such as images, speech, natural language processing, and recommendation engines. However, there are many other domains where such data is not available, or access to it is highly restricted for privacy reasons, as with health and financial data. Even when abundant data is available, it is often not labeled. Doing such labeling is labor-intensive and non-scalable. As a result, to the best of our knowledge, key domains still lack labeled data or have at most toy data; or the synthetic data must have access to real data from which it can mimic new data. This paper outlines work to generate realistic synthetic data for an important domain: credit card transactions. Some challenges: there are many patterns and correlations in real purchases. There are millions of merchants and innumerable locations. Those merchants offer a wide variety of goods. Who shops where and when? How much do people pay? What is a realistic fraudulent transaction? We use a mixture of technical approaches and domain knowledge including mechanics of credit card processing, a broad set of consumer domains: electronics, clothing, hair styling, etc. Connecting everything is a virtual world. This paper outlines some of our key techniques and provides evidence that the data generated is indeed realistic. Beyond the scope of this paper: (1) use of our data to develop and train models to predict fraud; (2) coupling models and the synthetic dataset to assess performance in designing accelerators such as GPUs and TPUs.
Kernel-based Approach to Handle Mixed Data for Inferring Causal Graphs
Venue: arXiv cs.LG 2019
Year: 2019

Causal learning is a beneficial approach to analyze the cause and effect relationships among variables in a dataset. A causal graph can be generated from a dataset using a particular causal algorithm, for instance, the PC algorithm or Fast Causal Inference (FCI). Generating a causal graph from a dataset that contains different data types (mixed data) is not trivial. This research offers an easy way to handle the mixed data so that it can be used to learn causal graphs using the existing application of the PC algorithm and FCI. This research proposes using kernel functions and Kernel Alignment to handle mixed data. Two main steps of this approach are computing a kernel matrix for each variable and calculating a pseudo-correlation matrix using Kernel Alignment. Kernel Alignment is used as a substitute for the correlation matrix for the conditional independence test for Gaussian data in the PC Algorithm and FCI. The advantage of this idea is that is possible to handle any data type by using a suitable kernel function to compute a kernel matrix for an observed variable. The proposed method is successfully applied to learn a causal graph from mixed data containing categorical, binary, ordinal, and continuous variables.
The Query Translation Landscape: a Survey
Venue: arXiv cs.DB 2019
Year: 2019

Whereas the availability of data has seen a manyfold increase in past years, its value can be only shown if the data variety is effectively tackled ---one of the prominent Big Data challenges. The lack of data interoperability limits the potential of its collective use for novel applications. Achieving interoperability through the full transformation and integration of diverse data structures remains an ideal that is hard, if not impossible, to achieve. Instead, methods that can simultaneously interpret different types of data available in different data structures and formats have been explored. On the other hand, many query languages have been designed to enable users to interact with the data, from relational, to object-oriented, to hierarchical, to the multitude emerging NoSQL languages. Therefore, the interoperability issue could be solved not by enforcing physical data transformation, but by looking at techniques that are able to query heterogeneous sources using one uniform language. Both industry and research communities have been keen to develop such techniques, which require the translation of a chosen 'universal' query language to the various data model specific query languages that make the underlying data accessible. In this article, we survey more than forty query translation methods and tools for popular query languages, and classify them according to eight criteria. In particular, we study which query language is a most suitable candidate for that 'universal' query language. Further, the results enable us to discover the weakly addressed and unexplored translation paths, to discover gaps and to learn lessons that can benefit future research in the area.
Deep Multiphase Level Set for Scene Parsing
Venue: arXiv cs.CV 2019
Year: 2019

Recently, Fully Convolutional Network (FCN) seems to be the go-to architecture for image segmentation, including semantic scene parsing. However, it is difficult for a generic FCN to discriminate pixels around the object boundaries, thus FCN based methods may output parsing results with inaccurate boundaries. Meanwhile, level set based active contours are superior to the boundary estimation due to the sub-pixel accuracy that they achieve. However, they are quite sensitive to initial settings. To address these limitations, in this paper we propose a novel Deep Multiphase Level Set (DMLS) method for semantic scene parsing, which efficiently incorporates multiphase level sets into deep neural networks. The proposed method consists of three modules, i.e., recurrent FCNs, adaptive multiphase level set, and deeply supervised learning. More specifically, recurrent FCNs learn multi-level representations of input images with different contexts. Adaptive multiphase level set drives the discriminative contour for each semantic class, which makes use of the advantages of both global and local information. In each time-step of the recurrent FCNs, deeply supervised learning is incorporated for model training. Extensive experiments on three public benchmarks have shown that our proposed method achieves new state-of-the-art performances.
Random forest model identifies serve strength as a key predictor of tennis match outcome
Venue: arXiv cs.LG 2019
Year: 2019

Tennis is a popular sport worldwide, boasting millions of fans and numerous national and international tournaments. Like many sports, tennis has benefitted from the popularity of rigorous record-keeping of game and player information, as well as the growth of machine learning methods for use in sports analytics. Of particular interest to bettors and betting companies alike is potential use of sports records to predict tennis match outcomes prior to match start. We compiled, cleaned, and used the largest database of tennis match information to date to predict match outcome using fairly simple machine learning methods. Using such methods allows for rapid fit and prediction times to readily incorporate new data and make real-time predictions. We were able to predict match outcomes with upwards of 80% accuracy, much greater than predictions using betting odds alone, and identify serve strength as a key predictor of match outcome. By combining prediction accuracies from three models, we were able to nearly recreate a probability distribution based on average betting odds from betting companies, which indicates that betting companies are using similar information to assign odds to matches. These results demonstrate the capability of relatively simple machine learning models to quite accurately predict tennis match outcomes.
Understanding the Expressive Power of Unhygienic Substitution in Metaprogramming via Combinatory Logic
Venue: arXiv cs.LO 2019
Year: 2019

Recent work on combinatory logic demonstrates a compositional translation from lambda calculus that gives meaning to open terms. As the meaning of open terms is a key difficulty in the study of metaprogramming, we investigate whether this idea can be extended to metaprogramming systems with hygienic and unhygienic substitution, specifically $λ_{e}^{\rightarrow \Box}$ and $λ_{\mathcal{S}}$. We conclude that there is quantitative evidence to support the claim that metaprogramming with unhygienic substitution is more powerful than metaprogramming with hygienic substitution.
Peer Reviewing Revisited: Assessing Research with Interlinked Semantic Comments
Venue: arXiv cs.DL 2019
Year: 2019

Scientific publishing seems to be at a turning point. Its paradigm has stayed basically the same for 300 years but is now challenged by the increasing volume of articles that makes it very hard for scientists to stay up to date in their respective fields. In fact, many have pointed out serious flaws of current scientific publishing practices, including the lack of accuracy and efficiency of the reviewing process. To address some of these problems, we apply here the general principles of the Web and the Semantic Web to scientific publishing, focusing on the reviewing process. We want to determine if a fine-grained model of the scientific publishing workflow can help us make the reviewing processes better organized and more accurate, by ensuring that review comments are created with formal links and semantics from the start. Our contributions include a novel model called Linkflows that allows for such detailed and semantically rich representations of reviews and the reviewing processes. We evaluate our approach on a manually curated dataset from several recent Computer Science journals and conferences that come with open peer reviews. We gathered ground-truth data by contacting the original reviewers and asking them to categorize their own review comments according to our model. Comparing this ground truth to answers provided by model experts, peers, and automated techniques confirms that our approach of formally capturing the reviewers' intentions from the start prevents substantial discrepancies compared to when this information is later extracted from the plain-text comments. In general, our analysis shows that our model is well understood and easy to apply, and it revealed the semantic properties of such review comments.
A Fast Exact Algorithm for Airplane Refueling Problem
Venue: arXiv cs.DS 2019
Year: 2019

We consider the airplane refueling problem, where we have a fleet of airplanes that can refuel each other. Each airplane is characterized by specific fuel tank volume and fuel consumption rate, and the goal is to find a drop out order of airplanes that last airplane in the air can reach as far as possible. This problem is equivalent to the scheduling problem $1||\sum w_j (- \frac{1}{C_j})$. Based on the dominance properties among jobs, we reveal some structural properties of the problem and propose a recursive algorithm to solve the problem exactly. The running time of our algorithm is directly related to the number of schedules that do not violate the dominance properties. An experimental study shows our algorithm outperforms state of the art exact algorithms and is efficient on larger instances.
FlowDyn: Towards a Dynamic Flowlet Gap Detection using Programmable Data Planes
Venue: arXiv cs.NI 2019
Year: 2019

Data center networks offer multiple disjoint paths between Top-of-Rack (ToR) switches to connect server racks providing large bisection bandwidth. An effective load-balancing mechanism is required in order to fully utilize the available capacity of the multiple paths. While packet-based load-balancing can achieve high utilization, it suffers from reordering. Flow-based load-balancing such as equal-cost multipath routing (ECMP) spreads traffic uniformly across multiple paths leading to frequent hash collisions and suboptimal performance. Finally, flowlet based load-balancing such as CONGA or HULA splits flows into smaller units, which are sent on different paths. Most flowlet based load-balancing schemes depend on a proper static setting of the flowlet gap, which decides when new flowlets are detected. While a too small gap may lead to reordering, a too large gap results in missed load-balancing opportunities. In this paper, we propose FlowDyn, which dynamically adapts the flowlet gap to increase the efficiency of the load-balancing schemes while avoiding the reordering problem. Using programmable data planes, FlowDyn uses active probes together with telemetry information to track path latency between different ToR switches. FlowDyn calculates dynamically a suitable flowlet gap that can be used for flowlet based load-balancing mechanism. We evaluate FlowDyn extensively in simulation, showing that it achieves 3.19 times smaller flow completion time at 10% load and 1.16x at 90% load.
Research on the Concept of Liquid State Machine
Venue: arXiv cs.NE 2019
Year: 2019

Liquid State Machine (LSM) is a neural model with real time computations which transforms the time varying inputs stream to a higher dimensional space. The concept of LSM is a novel field of research in biological inspired computation with most research effort on training the model as well as finding the optimum learning method. In this review, the performance of LSM model was investigated using two learning method, online learning and offline (batch) learning methods. The review revealed that optimal performance of LSM was recorded through online method as computational space and other complexities associated with batch learning is eliminated.
A Review of Soft Robots
Venue: arXiv cs.RO 2019
Year: 2019

Traditional robots have rigid links and structures that limit their ability to interact with the dynamics of their immediate environment. For example, conventional robot manipulators with rigid links can only manipulate objects using specific end effectors. These robots often encounter difficulties operating in unstructured and highly congested environments. A variety of biological organisms exhibit complex movement with soft structures devoid of rigid components. Inspired by biology, researchers have been able to design and build soft robots. With a soft structure and redundant degrees of freedom, these robots can be used for delicate tasks in unstructured environments. This review discusses the motivation for soft robots, their design processes as well as their applications and limitations. Soft robots have the ability to operate in unstructured environment due to their inherent potential to exploit morphological computation to adapt to, and interact with, the world in a way that is difficult with rigid systems. Soft robots could be used for operations, ranging from search and rescue operations in a natural disaster relief effort, and of emerging interest is in the field of medical care as seen in personal robots.