OpenRaiser/Intern-Atlas huggingface dataset:
HGC: Hierarchical Group Convolution for Highly Efficient Neural Network
Venue: arXiv cs.CV 2019
Year: 2019

Group convolution works well with many deep convolutional neural networks (CNNs) that can effectively compress the model by reducing the number of parameters and computational cost. Using this operation, feature maps of different group cannot communicate, which restricts their representation capability. To address this issue, in this work, we propose a novel operation named Hierarchical Group Convolution (HGC) for creating computationally efficient neural networks. Different from standard group convolution which blocks the inter-group information exchange and induces the severe performance degradation, HGC can hierarchically fuse the feature maps from each group and leverage the inter-group information effectively. Taking advantage of the proposed method, we introduce a family of compact networks called HGCNets. Compared to networks using standard group convolution, HGCNets have a huge improvement in accuracy at the same model size and complexity level. Extensive experimental results on the CIFAR dataset demonstrate that HGCNets obtain significant reduction of parameters and computational cost to achieve comparable performance over the prior CNN architectures designed for mobile devices such as MobileNet and ShuffleNet.
Question Answering as Global Reasoning over Semantic Abstractions
Venue: arXiv cs.CL 2019
Year: 2019

We propose a novel method for exploiting the semantic structure of text to answer multiple-choice questions. The approach is especially suitable for domains that require reasoning over a diverse set of linguistic constructs but have limited training data. To address these challenges, we present the first system, to the best of our knowledge, that reasons over a wide range of semantic abstractions of the text, which are derived using off-the-shelf, general-purpose, pre-trained natural language modules such as semantic role labelers, coreference resolvers, and dependency parsers. Representing multiple abstractions as a family of graphs, we translate question answering (QA) into a search for an optimal subgraph that satisfies certain global and local properties. This formulation generalizes several prior structured QA systems. Our system, SEMANTICILP, demonstrates strong performance on two domains simultaneously. In particular, on a collection of challenging science QA datasets, it outperforms various state-of-the-art approaches, including neural models, broad coverage information retrieval, and specialized techniques using structured knowledge bases, by 2%-6%.
Curiosity-Driven Multi-Criteria Hindsight Experience Replay
Venue: arXiv cs.LG 2019
Year: 2019

Dealing with sparse rewards is a longstanding challenge in reinforcement learning. The recent use of hindsight methods have achieved success on a variety of sparse-reward tasks, but they fail on complex tasks such as stacking multiple blocks with a robot arm in simulation. Curiosity-driven exploration using the prediction error of a learned dynamics model as an intrinsic reward has been shown to be effective for exploring a number of sparse-reward environments. We present a method that combines hindsight with curiosity-driven exploration and curriculum learning in order to solve the challenging sparse-reward block stacking task. We are the first to stack more than two blocks using only sparse reward without human demonstrations.
Degrees of Freedom Analysis of Unrolled Neural Networks
Venue: arXiv cs.LG 2019
Year: 2019

Unrolled neural networks emerged recently as an effective model for learning inverse maps appearing in image restoration tasks. However, their generalization risk (i.e., test mean-squared-error) and its link to network design and train sample size remains mysterious. Leveraging the Stein's Unbiased Risk Estimator (SURE), this paper analyzes the generalization risk with its bias and variance components for recurrent unrolled networks. We particularly investigate the degrees-of-freedom (DOF) component of SURE, trace of the end-to-end network Jacobian, to quantify the prediction variance. We prove that DOF is well-approximated by the weighted \textit{path sparsity} of the network under incoherence conditions on the trained weights. Empirically, we examine the SURE components as a function of train sample size for both recurrent and non-recurrent (with many more parameters) unrolled networks. Our key observations indicate that: 1) DOF increases with train sample size and converges to the generalization risk for both recurrent and non-recurrent schemes; 2) recurrent network converges significantly faster (with less train samples) compared with non-recurrent scheme, hence recurrence serves as a regularization for low sample size regimes.
BAGS: An automatic homework grading system using the pictures taken by smart phones
Venue: arXiv cs.CV 2019
Year: 2019

Homework grading is critical to evaluate teaching quality and effect. However, it is usually time-consuming to grade the homework manually. In automatic homework grading scenario, many optical mark reader (OMR)-based solutions which require specific equipments have been proposed. Although many of them can achieve relatively high accuracy, they are less convenient for users. In contrast, with the popularity of smart phones, the automatic grading system which depends on the image photographed by phones becomes more available. In practice, due to different photographing angles or uneven papers, images may be distorted. Moreover, most of images are photographed under complex backgrounds, making answer areas detection more difficult. To solve these problems, we propose BAGS, an automatic homework grading system which can effectively locate and recognize handwritten answers. In BAGS, all the answers would be written above the answer area underlines (AAU), and we use two segmentation networks based on DeepLabv3+ to locate the answer areas. Then, we use the characters recognition part to recognize students' answers. Finally, the grading part is designed for the comparison between the recognized answers and the standard ones. In our test, BAGS correctly locates and recognizes the handwritten answers in 91% of total answer areas.
Intriguing properties of adversarial training at scale
Venue: arXiv cs.CV 2019
Year: 2019

Adversarial training is one of the main defenses against adversarial attacks. In this paper, we provide the first rigorous study on diagnosing elements of adversarial training, which reveals two intriguing properties. First, we study the role of normalization. Batch normalization (BN) is a crucial element for achieving state-of-the-art performance on many vision tasks, but we show it may prevent networks from obtaining strong robustness in adversarial training. One unexpected observation is that, for models trained with BN, simply removing clean images from training data largely boosts adversarial robustness, i.e., 18.3%. We relate this phenomenon to the hypothesis that clean images and adversarial images are drawn from two different domains. This two-domain hypothesis may explain the issue of BN when training with a mixture of clean and adversarial images, as estimating normalization statistics of this mixture distribution is challenging. Guided by this two-domain hypothesis, we show disentangling the mixture distribution for normalization, i.e., applying separate BNs to clean and adversarial images for statistics estimation, achieves much stronger robustness. Additionally, we find that enforcing BNs to behave consistently at training and testing can further enhance robustness. Second, we study the role of network capacity. We find our so-called "deep" networks are still shallow for the task of adversarial learning. Unlike traditional classification tasks where accuracy is only marginally improved by adding more layers to "deep" networks (e.g., ResNet-152), adversarial training exhibits a much stronger demand on deeper networks to achieve higher adversarial robustness. This robustness improvement can be observed substantially and consistently even by pushing the network capacity to an unprecedented scale, i.e., ResNet-638.
FairLedger: A Fair Blockchain Protocol for Financial Institutions
Venue: arXiv cs.DC 2019
Year: 2019

Financial institutions are currently looking into technologies for permissioned blockchains. A major effort in this direction is Hyperledger, an open source project hosted by the Linux Foundation and backed by a consortium of over a hundred companies. A key component in permissioned blockchain protocols is a byzantine fault tolerant (BFT) consensus engine that orders transactions. However, currently available BFT solutions in Hyperledger (as well as in the literature at large) are inadequate for financial settings; they are not designed to ensure fairness or to tolerate selfish behavior that arises when financial institutions strive to maximize their own profit. We present FairLedger, a permissioned blockchain BFT protocol, which is fair, designed to deal with rational behavior, and, no less important, easy to understand and implement. The secret sauce of our protocol is a new communication abstraction, called detectable all-to-all (DA2A), which allows us to detect participants (byzantine or rational) that deviate from the protocol, and punish them. We implement FairLedger in the Hyperledger open source project, using Iroha framework, one of the biggest projects therein. To evaluate FairLegder's performance, we also implement it in the PBFT framework and compare the two protocols. Our results show that in failure-free scenarios FairLedger achieves better throughput than both Iroha's implementation and PBFT in wide-area settings.
Stochastic Mirror Descent on Overparameterized Nonlinear Models: Convergence, Implicit Regularization, and Generalization
Venue: arXiv cs.LG 2019
Year: 2019

Most modern learning problems are highly overparameterized, meaning that there are many more parameters than the number of training data points, and as a result, the training loss may have infinitely many global minima (parameter vectors that perfectly interpolate the training data). Therefore, it is important to understand which interpolating solutions we converge to, how they depend on the initialization point and the learning algorithm, and whether they lead to different generalization performances. In this paper, we study these questions for the family of stochastic mirror descent (SMD) algorithms, of which the popular stochastic gradient descent (SGD) is a special case. Our contributions are both theoretical and experimental. On the theory side, we show that in the overparameterized nonlinear setting, if the initialization is close enough to the manifold of global minima (something that comes for free in the highly overparameterized case), SMD with sufficiently small step size converges to a global minimum that is approximately the closest one in Bregman divergence. On the experimental side, our extensive experiments on standard datasets and models, using various initializations, various mirror descents, and various Bregman divergences, consistently confirms that this phenomenon happens in deep learning. Our experiments further indicate that there is a clear difference in the generalization performance of the solutions obtained by different SMD algorithms. Experimenting on a standard image dataset and network architecture with SMD with different kinds of implicit regularization, $\ell_1$ to encourage sparsity, $\ell_2$ yielding SGD, and $\ell_{10}$ to discourage large components in the parameter vector, consistently and definitively shows that $\ell_{10}$-SMD has better generalization performance than SGD, which in turn has better generalization performance than $\ell_1$-SMD.
Indicators of Open Access for universities
Venue: arXiv cs.DL 2019
Year: 2019

This paper presents a first attempt to analyse Open Access integration at the institutional level. For this, we combine information from Unpaywall and the Leiden Ranking to offer basic OA indicators for universities. We calculate the overall number of Open Access publications for 930 universities worldwide. OA indicators are also disaggregated by green, gold and hybrid Open Access. We then explore differences between and within countries and offer a general ranking of universities based on the proportion of their output which is openly accessible.
Laplacian Spectral Basis Functions
Venue: arXiv cs.GR 2019
Year: 2019

Representing a signal as a linear combination of a set of basis functions is central in a wide range of applications, such as approximation, de-noising, compression, shape correspondence and comparison. In this context, our paper addresses the main aspects of signal approximation, such as the definition, computation, and comparison of basis functions on arbitrary 3D shapes. Focusing on the class of basis functions induced by the Laplace-Beltrami operator and its spectrum, we introduce the diffusion and Laplacian spectral basis functions, which are then compared with the harmonic and Laplacian eigenfunctions. As main properties of these basis functions, which are commonly used for numerical geometry processing and shape analysis, we discuss the partition of the unity and non-negativity; the intrinsic definition and invariance with respect to shape transformations (e.g., translation, rotation, uniform scaling); the locality, smoothness, and orthogonality; the numerical stability with respect to the domain discretisation; the computational cost and storage overhead. Finally, we consider geometric metrics, such as the area, conformal, and kernel-based norms, for the comparison and characterisation of the main properties of the Laplacian basis functions.
Scale Steerable Filters for Locally Scale-Invariant Convolutional Neural Networks
Venue: arXiv cs.CV 2019
Year: 2019

Augmenting transformation knowledge onto a convolutional neural network's weights has often yielded significant improvements in performance. For rotational transformation augmentation, an important element to recent approaches has been the use of a steerable basis i.e. the circular harmonics. Here, we propose a scale-steerable filter basis for the locally scale-invariant CNN, denoted as log-radial harmonics. By replacing the kernels in the locally scale-invariant CNN \cite{lsi_cnn} with scale-steered kernels, significant improvements in performance can be observed on the MNIST-Scale and FMNIST-Scale datasets. Training with a scale-steerable basis results in filters which show meaningful structure, and feature maps demonstrate which demonstrate visibly higher spatial-structure preservation of input. Furthermore, the proposed scale-steerable CNN shows on-par generalization to global affine transformation estimation methods such as Spatial Transformers, in response to test-time data distortions.
LASSi: Metric based I/O analytics for HPC
Venue: arXiv cs.DC 2019
Year: 2019

LASSi is a tool aimed at analyzing application usage and contention caused by use of shared resources (filesystem or network) in a HPC system. LASSi was initially developed to support the ARCHER system where there are large variations in application requirements and occasional user complaints regarding filesystem performance manifested by variation in job runtimes or poor interactive response. LASSi takes an approach of defining derivative risk and ops metrics that relate to unusually high application I/O behaviour. The metrics are shown to correlate to applications that can experience variable performance or that may impact the performance of other applications. LASSi uses I/O statistics over time to provide application I/O profiles and has been automated to generate daily reports for ARCHER. We demonstrate how LASSi provides holistic I/O analysis by monitoring filesystem I/O, generating coarse profiles of filesystems and application runs and automating analysis of application slowdown using metrics.
Formalization of the Axiom of Choice and its Equivalent Theorems
Venue: arXiv cs.LO 2019
Year: 2019

In this paper, we describe the formalization of the axiom of choice and several of its famous equivalent theorems in Morse-Kelley set theory. These theorems include Tukey's lemma, the Hausdorff maximal principle, the maximal principle, Zermelo's postulate, Zorn's lemma and the well-ordering theorem. We prove the above theorems by the axiom of choice in turn, and finally prove the axiom of choice by Zermelo's postulate and the well-ordering theorem, thus completing the cyclic proof of equivalence between them. The proofs are checked formally using the Coq proof assistant in which Morse-Kelley set theory is formalized. The whole process of formal proof demonstrates that the Coq-based machine proving of mathematics theorem is highly reliable and rigorous. The formal work of this paper is enough for most applications, especially in set theory, topology and algebra.
Java Generics: An Order-Theoretic Approach (Detailed Outline)
Venue: arXiv cs.PL 2019
Year: 2019

Generics have been added to Java so as to increase the expressiveness of its type system. Generics in Java, however, include some features---such as Java wildcards, $F$-bounded generics, and Java erasure---that have been hard to analyze and reason about so far, reflecting the fact that the mathematical modeling of generics in Java and other similar nominally-typed object-oriented programming (OOP) languages is a challenge. As a result, the type systems of mainstream nominally-typed OOP languages, which are built based on current models of generics, are overly complex, which hinders the progress of these type systems. In this paper we present a detailed outline of a new approach to modeling Java generics that uses concepts and tools from order theory, and we report on our progress in developing this approach. Fundamentally, we use the nominal subclassing relation (as a partial order) together with some standard and novel order-theoretic tools to construct the generic nominal subtyping relation (as a partial order) and the containment relation between generic type arguments (a third partial order). We further analyze the relation between these three ordering relations---which lie at the heart of mainstream generic OO type systems---using order theoretic tools, and accordingly we explore extensions of OO type systems suggested by such analysis. In our approach we also make use of some concepts and tools from category theory. We believe a combined order-theoretic and category-theoretic approach to modeling generics holds the keys to overcoming much of the adversity found when analyzing features of generic OO type systems.
Exploration and Exploitation in Symbolic Regression using Quality-Diversity and Evolutionary Strategies Algorithms
Venue: arXiv cs.NE 2019
Year: 2019

By combining Genetic Programming, MAP-Elites and Covariance Matrix Adaptation Evolution Strategy, we demonstrate very high success rates in Symbolic Regression problems. MAP-Elites is used to improve exploration while preserving diversity and avoiding premature convergence and bloat. Then, a Covariance Matrix Adaptation-Evolution Strategy is used to evaluate free scalars through a non-gradient-based black-box optimizer. Although this evaluation approach is not computationally scalable to high dimensional problems, our algorithm is able to find exactly most of the $31$ targets extracted from the literature on which we evaluate it.
STAMINA: STochastic Approximate Model-checker for INfinite-state Analysis
Venue: arXiv cs.LO 2019
Year: 2019

Stochastic model checking is a technique for analyzing systems that possess probabilistic characteristics. However, its scalability is limited as probabilistic models of real-world applications typically have very large or infinite state space. This paper presents a new infinite state CTMC model checker, STAMINA, with improved scalability. It uses a novel state space approximation method to reduce large and possibly infinite state CTMC models to finite state representations that are amenable to existing stochastic model checkers. It is integrated with a new property-guided state expansion approach that improves the analysis accuracy. Demonstration of the tool on several benchmark examples shows promising results in terms of analysis efficiency and accuracy compared with a state-of-the-art CTMC model checker that deploys a similar approximation method.
Unit Impulse Response as an Explainer of Redundancy in a Deep Convolutional Neural Network
Venue: arXiv cs.LG 2019
Year: 2019

Convolutional neural networks (CNN) are generally designed with a heuristic initialization of network architecture and trained for a certain task. This often leads to overparametrization after learning and induces redundancy in the information flow paths within the network. This robustness and reliability is at the increased cost of redundant computations. Several methods have been proposed which leverage metrics that quantify the redundancy in each layer. However, layer-wise evaluation in these methods disregards the long-range redundancy which exists across depth on account of the distributed nature of the features learned by the model. In this paper, we propose (i) a mechanism to empirically demonstrate the robustness in performance of a CNN on account of redundancy across its depth, (ii) a method to identify the systemic redundancy in response of a CNN across depth using the understanding of unit impulse response, we subsequently demonstrate use of these methods to interpret redundancy in few networks as example. These techniques provide better insights into the internal dynamics of a CNN
Errors-in-variables Modeling of Personalized Treatment-Response Trajectories
Venue: arXiv cs.LG 2019
Year: 2019

Estimating the effect of a treatment on a given outcome, conditioned on a vector of covariates, is central in many applications. However, learning the impact of a treatment on a continuous temporal response, when the covariates suffer extensively from measurement error and even the timing of the treatments is uncertain, has not been addressed. We introduce a novel data-driven method that can estimate treatment-response trajectories in this challenging scenario. We model personalized treatment-response curves as a combination of parametric response functions, hierarchically sharing information across individuals, and a sparse Gaussian process for the baseline trend. Importantly, our model considers measurement error not only in treatment covariates, but also in treatment times, a problem which arises in practice for example when treatment information is based on self-reporting. In a challenging and timely problem of estimating the impact of diet on continuous blood glucose measurements, our model leads to significant improvements in estimation accuracy and prediction.