OpenRaiser/Intern-Atlas huggingface dataset:
An Inference and Learning Engine for Spiking Neural Networks in Computational RAM (CRAM)
Venue: arXiv cs.ET 2020
Year: 2020

Spiking Neural Networks (SNN) represent a biologically inspired computation model capable of emulating neural computation in human brain and brain-like structures. The main promise is very low energy consumption. Unfortunately, classic Von Neumann architecture based SNN accelerators often fail to address demanding computation and data transfer requirements efficiently at scale. In this work, we propose a promising alternative, an in-memory SNN accelerator based on Spintronic Computational RAM (CRAM) to overcome scalability limitations, which can reduce the energy consumption by up to 164.1$\times$ when compared to a representative ASIC solution.
Digital Currency and Economic Crises: Helping States Respond
Venue: arXiv cs.CY 2020
Year: 2020

The current crisis, at the time of writing, has had a profound impact on the financial world, introducing the need for creative approaches to revitalising the economy at the micro level as well as the macro level. In this informal analysis and design proposal, we describe how infrastructure for digital assets can serve as a useful monetary and fiscal policy tool and an enabler of existing tools in the future, particularly during crises, while aligning the trajectory of financial technology innovation toward a brighter future. We propose an approach to digital currency that would allow people without banking relationships to transact electronically and privately, including both internet purchases and point-of-sale purchases that are required to be cashless. We also propose an approach to digital currency that would allow for more efficient and transparent clearing and settlement, implementation of monetary and fiscal policy, and management of systemic risk. The digital currency could be implemented as central bank digital currency (CBDC), or it could be issued by the government and collateralised by public funds or Treasury assets. Our proposed architecture allows both manifestations and would be operated by banks and other money services businesses, operating within a framework overseen by government regulators. We argue that now is the time for action to undertake development of such a system, not only because of the current crisis but also in anticipation of future crises resulting from geopolitical risks, the continued globalisation of the digital economy, and the changing value and risks that technology brings.
Interactive Music and Synchronous Reactive Programming
Venue: arXiv cs.PL 2020
Year: 2020

This paper presents Skini, a programming methodology and an execution environment for interactive structured music. With this system, the composer programs his scores in the HipHop.js synchronous reactive language. They are then executed, or played, in live concerts, in interaction with the audience. The system aims at helping composers to find a good balance between the determinism of the compositions and the nondeterminism of the interactions with the public. Each execution of a Skini score yields to a different but aesthetically consistent interpretation. This work raises many questions in the musical fields. How to combine composition and interaction? How to control the musical style when the audience influences what is to play next? What are the possible connections with generative music? These are important questions for the Skini system but they are out of the scope of this paper that focuses exclusively on the computer science aspects of the system. From that perspective, the main questions are how to program the scores and in which language? General purpose languages are inappropriate because their elementary constructs (i.e., variables, functions, loops, etc.) do not match the constructions needed to express music and musical constraints. We show that synchronous programming languages are a much better fit because they rely on temporal constructs that can be directly used to represent musical scores and because their malleability enables composers to experiment easily with artistic variations of their initial scores. The paper mostly focuses on scores programming. It exposes the process a composer should follow from his very first musical intuitions up to the generation of a musical artifact. The paper presents some excerpts of the programming of a classical music composition that it then precisely relates to an actual recording. Examples of techno music and jazz are also presented, with audio artifact, to demonstrate the versatility of the system. Finally, brief presentations of past live concerts are presented as an evidence of viability of the system.
Model-Based Generalization Under Parameter Uncertainty Using Path Integral Control
Venue: arXiv cs.RO 2020
Year: 2020

This work addresses the problem of robot interaction in complex environments where online control and adaptation is necessary. By expanding the sample space in the free energy formulation of path integral control, we derive a natural extension to the path integral control that embeds uncertainty into action and provides robustness for model-based robot planning. Our algorithm is applied to a diverse set of tasks using different robots and validate our results in simulation and real-world experiments. We further show that our method is capable of running in real-time without loss of performance. Videos of the experiments as well as additional implementation details can be found at https://sites.google.com/view/emppi.
A Linear Algebraic Approach to Model Parallelism in Deep Learning
Venue: arXiv cs.LG 2020
Year: 2020

Training deep neural networks (DNNs) in large-cluster computing environments is increasingly necessary, as networks grow in size and complexity. Local memory and processing limitations require robust data and model parallelism for crossing compute node boundaries. We propose a linear-algebraic approach to model parallelism in deep learning, which allows parallel distribution of any tensor in the DNN. Rather than rely on automatic differentiation tools, which do not universally support distributed memory parallelism models, we show that parallel data movement operations, e.g., broadcast, sum-reduce, and halo exchange, are linear operators, and by defining the relevant spaces and inner products, we manually develop the adjoint, or backward, operators required for gradient-based training of DNNs. We build distributed DNN layers using these parallel primitives, composed with sequential layer implementations, and demonstrate their application by building and training a distributed DNN using DistDL, a PyTorch and MPI-based distributed deep learning toolkit.
MLE-guided parameter search for task loss minimization in neural sequence modeling
Venue: arXiv cs.LG 2020
Year: 2020

Neural autoregressive sequence models are used to generate sequences in a variety of natural language processing (NLP) tasks, where they are evaluated according to sequence-level task losses. These models are typically trained with maximum likelihood estimation, which ignores the task loss, yet empirically performs well as a surrogate objective. Typical approaches to directly optimizing the task loss such as policy gradient and minimum risk training are based around sampling in the sequence space to obtain candidate update directions that are scored based on the loss of a single sequence. In this paper, we develop an alternative method based on random search in the parameter space that leverages access to the maximum likelihood gradient. We propose maximum likelihood guided parameter search (MGS), which samples from a distribution over update directions that is a mixture of random search around the current parameters and around the maximum likelihood gradient, with each direction weighted by its improvement in the task loss. MGS shifts sampling to the parameter space, and scores candidates using losses that are pooled from multiple sequences. Our experiments show that MGS is capable of optimizing sequence-level losses, with substantial reductions in repetition and non-termination in sequence completion, and similar improvements to those of minimum risk training in machine translation.
MSDU-net: A Multi-Scale Dilated U-net for Blur Detection
Venue: arXiv cs.CV 2020
Year: 2020

Blur detection is the separation of blurred and clear regions of an image, which is an important and challenging task in computer vision. In this work, we regard blur detection as an image segmentation problem. Inspired by the success of the U-net architecture for image segmentation, we design a Multi-Scale Dilated convolutional neural network based on U-net, which we call MSDU-net. The MSDU-net uses a group of multi-scale feature extractors with dilated convolutions to extract texture information at different scales. The U-shape architecture of the MSDU-net fuses the different-scale texture features and generates a semantic feature which allows us to achieve better results on the blur detection task. We show that using the MSDU-net we are able to outperform other state of the art blur detection methods on two publicly available benchmarks.
Egocentric Object Manipulation Graphs
Venue: arXiv cs.CV 2020
Year: 2020

We introduce Egocentric Object Manipulation Graphs (Ego-OMG) - a novel representation for activity modeling and anticipation of near future actions integrating three components: 1) semantic temporal structure of activities, 2) short-term dynamics, and 3) representations for appearance. Semantic temporal structure is modeled through a graph, embedded through a Graph Convolutional Network, whose states model characteristics of and relations between hands and objects. These state representations derive from all three levels of abstraction, and span segments delimited by the making and breaking of hand-object contact. Short-term dynamics are modeled in two ways: A) through 3D convolutions, and B) through anticipating the spatiotemporal end points of hand trajectories, where hands come into contact with objects. Appearance is modeled through deep spatiotemporal features produced through existing methods. We note that in Ego-OMG it is simple to swap these appearance features, and thus Ego-OMG is complementary to most existing action anticipation methods. We evaluate Ego-OMG on the EPIC Kitchens Action Anticipation Challenge. The consistency of the egocentric perspective of EPIC Kitchens allows for the utilization of the hand-centric cues upon which Ego-OMG relies. We demonstrate state-of-the-art performance, outranking all other previous published methods by large margins and ranking first on the unseen test set and second on the seen test set of the EPIC Kitchens Action Anticipation Challenge. We attribute the success of Ego-OMG to the modeling of semantic structure captured over long timespans. We evaluate the design choices made through several ablation studies. Code will be released upon acceptance
FP-Stereo: Hardware-Efficient Stereo Vision for Embedded Applications
Venue: arXiv cs.CV 2020
Year: 2020

Fast and accurate depth estimation, or stereo matching, is essential in embedded stereo vision systems, requiring substantial design effort to achieve an appropriate balance among accuracy, speed and hardware cost. To reduce the design effort and achieve the right balance, we propose FP-Stereo for building high-performance stereo matching pipelines on FPGAs automatically. FP-Stereo consists of an open-source hardware-efficient library, allowing designers to obtain the desired implementation instantly. Diverse methods are supported in our library for each stage of the stereo matching pipeline and a series of techniques are developed to exploit the parallelism and reduce the resource overhead. To improve the usability, FP-Stereo can generate synthesizable C code of the FPGA accelerator with our optimized HLS templates automatically. To guide users for the right design choice meeting specific application requirements, detailed comparisons are performed on various configurations of our library to investigate the accuracy/speed/cost trade-off. Experimental results also show that FP-Stereo outperforms the state-of-the-art FPGA design from all aspects, including 6.08% lower error, 2x faster speed, 30% less resource usage and 40% less energy consumption. Compared to GPU designs, FP-Stereo achieves the same accuracy at a competitive speed while consuming much less energy.
GMAT: Global Memory Augmentation for Transformers
Venue: arXiv cs.LG 2020
Year: 2020

Transformer-based models have become ubiquitous in natural language processing thanks to their large capacity, innate parallelism and high performance. The contextualizing component of a Transformer block is the $\textit{pairwise dot-product}$ attention that has a large $Ω(L^2)$ memory requirement for length $L$ sequences, limiting its ability to process long documents. This has been the subject of substantial interest recently, where multiple approximations were proposed to reduce the quadratic memory requirement using sparse attention matrices. In this work, we propose to augment sparse Transformer blocks with a dense attention-based $\textit{global memory}$ of length $M$ ($\ll L$) which provides an aggregate global view of the entire input sequence to each position. Our augmentation has a manageable $O(M\cdot(L+M))$ memory overhead, and can be seamlessly integrated with prior sparse solutions. Moreover, global memory can also be used for sequence compression, by representing a long input sequence with the memory representations only. We empirically show that our method leads to substantial improvement on a range of tasks, including (a) synthetic tasks that require global reasoning, (b) masked language modeling, and (c) reading comprehension.
Conflict-Based Search for Connected Multi-Agent Path Finding
Venue: arXiv cs.AI 2020
Year: 2020

We study a variant of the multi-agent path finding problem (MAPF) in which agents are required to remain connected to each other and to a designated base. This problem has applications in search and rescue missions where the entire execution must be monitored by a human operator. We re-visit the conflict-based search algorithm known for MAPF, and define a variant where conflicts arise from disconnections rather than collisions. We study optimizations, and give experimental results in which we compare our algorithms with the literature.
SEAL: Scientific Keyphrase Extraction and Classification
Venue: arXiv cs.IR 2020
Year: 2020

Automatic scientific keyphrase extraction is a challenging problem facilitating several downstream scholarly tasks like search, recommendation, and ranking. In this paper, we introduce SEAL, a scholarly tool for automatic keyphrase extraction and classification. The keyphrase extraction module comprises two-stage neural architecture composed of Bidirectional Long Short-Term Memory cells augmented with Conditional Random Fields. The classification module comprises of a Random Forest classifier. We extensively experiment to showcase the robustness of the system. We evaluate multiple state-of-the-art baselines and show a significant improvement. The current system is hosted at http://lingo.iitgn.ac.in:5000/.
Utilizing machine learning to prevent water main breaks by understanding pipeline failure drivers
Venue: arXiv cs.CY 2020
Year: 2020

Data61 and Western Water worked collaboratively to apply engineering expertise and Machine Learning tools to find a cost-effective solution to the pipe failure problem in the region west of Melbourne, where on average 400 water main failures occur per year. To achieve this objective, we constructed a detailed picture and understanding of the behaviour of the water pipe network by 1) discovering the underlying drivers of water main breaks, and 2) developing a Machine Learning system to assess and predict the failure likelihood of water main breaking using historical failure records, descriptors of pipes, and other environmental factors. The ensuing results open up an avenue for Western Water to identify the priority of pipe renewals
Application of Optimization and Simulation to Musical Composition that Emerges Dynamically during Ensemble Singing Performance
Venue: arXiv cs.SD 2020
Year: 2020

This paper presents and tests a new approach to composing for ensemble singing performance: reality opera. In the performance of such a composition, emotions of the singers are real and emerge as a consequence of their interactions and reaction and to a dynamic narrative. This paper gives background and motivation for the form, based on three key concepts, incorporating the use of technology. Then proposed techniques for creating reality opera are instantiated in an example, which is performed and a behavioral analysis done of performer reactions, leading to support for the feasibility of the reality opera concept.
Online Motion Planning based on Nonlinear Model Predictive Control with Non-Euclidean Rotation Groups
Venue: arXiv cs.RO 2020
Year: 2020

This paper proposes a novel online motion planning approach to robot navigation based on nonlinear model predictive control. Common approaches rely on pure Euclidean optimization parameters. In robot navigation, however, state spaces often include rotational components which span over non-Euclidean rotation groups. The proposed approach applies nonlinear increment and difference operators in the entire optimization scheme to explicitly consider these groups. Realizations include but are not limited to quadratic form and time-optimal objectives. A complex parking scenario for the kinematic bicycle model demonstrates the effectiveness and practical relevance of the approach. In case of simpler robots (e.g. differential drive), a comparative analysis in a hierarchical planning setting reveals comparable computation times and performance. The approach is available in a modular and highly configurable open-source C++ software framework.
Masked Language Modeling for Proteins via Linearly Scalable Long-Context Transformers
Venue: arXiv cs.LG 2020
Year: 2020

Transformer models have achieved state-of-the-art results across a diverse range of domains. However, concern over the cost of training the attention mechanism to learn complex dependencies between distant inputs continues to grow. In response, solutions that exploit the structure and sparsity of the learned attention matrix have blossomed. However, real-world applications that involve long sequences, such as biological sequence analysis, may fall short of meeting these assumptions, precluding exploration of these models. To address this challenge, we present a new Transformer architecture, Performer, based on Fast Attention Via Orthogonal Random features (FAVOR). Our mechanism scales linearly rather than quadratically in the number of tokens in the sequence, is characterized by sub-quadratic space complexity and does not incorporate any sparsity pattern priors. Furthermore, it provides strong theoretical guarantees: unbiased estimation of the attention matrix and uniform convergence. It is also backwards-compatible with pre-trained regular Transformers. We demonstrate its effectiveness on the challenging task of protein sequence modeling and provide detailed theoretical analysis.
Hybrid Control for Learning Motor Skills
Venue: arXiv cs.RO 2020
Year: 2020

We develop a hybrid control approach for robot learning based on combining learned predictive models with experience-based state-action policy mappings to improve the learning capabilities of robotic systems. Predictive models provide an understanding of the task and the physics (which improves sample-efficiency), while experience-based policy mappings are treated as "muscle memory" that encode favorable actions as experiences that override planned actions. Hybrid control tools are used to create an algorithmic approach for combining learned predictive models with experience-based learning. Hybrid learning is presented as a method for efficiently learning motor skills by systematically combining and improving the performance of predictive models and experience-based policies. A deterministic variation of hybrid learning is derived and extended into a stochastic implementation that relaxes some of the key assumptions in the original derivation. Each variation is tested on experience-based learning methods (where the robot interacts with the environment to gain experience) as well as imitation learning methods (where experience is provided through demonstrations and tested in the environment). The results show that our method is capable of improving the performance and sample-efficiency of learning motor skills in a variety of experimental domains.
Joint learning of variational representations and solvers for inverse problems with partially-observed data
Venue: arXiv cs.LG 2020
Year: 2020

Designing appropriate variational regularization schemes is a crucial part of solving inverse problems, making them better-posed and guaranteeing that the solution of the associated optimization problem satisfies desirable properties. Recently, learning-based strategies have appeared to be very efficient for solving inverse problems, by learning direct inversion schemes or plug-and-play regularizers from available pairs of true states and observations. In this paper, we go a step further and design an end-to-end framework allowing to learn actual variational frameworks for inverse problems in such a supervised setting. The variational cost and the gradient-based solver are both stated as neural networks using automatic differentiation for the latter. We can jointly learn both components to minimize the data reconstruction error on the true states. This leads to a data-driven discovery of variational models. We consider an application to inverse problems with incomplete datasets (image inpainting and multivariate time series interpolation). We experimentally illustrate that this framework can lead to a significant gain in terms of reconstruction performance, including w.r.t. the direct minimization of the variational formulation derived from the known generative model.
Sparse Gaussian Processes via Parametric Families of Compactly-supported Kernels
Venue: arXiv cs.LG 2020
Year: 2020

Gaussian processes are powerful models for probabilistic machine learning, but are limited in application by their $O(N^3)$ inference complexity. We propose a method for deriving parametric families of kernel functions with compact spatial support, which yield naturally sparse kernel matrices and enable fast Gaussian process inference via sparse linear algebra. These families generalize known compactly-supported kernel functions, such as the Wendland polynomials. The parameters of this family of kernels can be learned from data using maximum likelihood estimation. Alternatively, we can quickly compute compact approximations of a target kernel using convex optimization. We demonstrate that these approximations incur minimal error over the exact models when modeling data drawn directly from a target GP, and can out-perform the traditional GP kernels on real-world signal reconstruction tasks, while exhibiting sub-quadratic inference complexity.
BERT-XML: Large Scale Automated ICD Coding Using BERT Pretraining
Venue: arXiv cs.IR 2020
Year: 2020

Clinical interactions are initially recorded and documented in free text medical notes. ICD coding is the task of classifying and coding all diagnoses, symptoms and procedures associated with a patient's visit. The process is often manual and extremely time-consuming and expensive for hospitals. In this paper, we propose a machine learning model, BERT-XML, for large scale automated ICD coding from EHR notes, utilizing recently developed unsupervised pretraining that have achieved state of the art performance on a variety of NLP tasks. We train a BERT model from scratch on EHR notes, learning with vocabulary better suited for EHR tasks and thus outperform off-the-shelf models. We adapt the BERT architecture for ICD coding with multi-label attention. While other works focus on small public medical datasets, we have produced the first large scale ICD-10 classification model using millions of EHR notes to predict thousands of unique ICD codes.