OpenRaiser/Intern-Atlas huggingface dataset:
Efficient Distributed SGD with Variance Reduction
Venue: arXiv cs.LG 2017
Year: 2017

Stochastic Gradient Descent (SGD) has become one of the most popular optimization methods for training machine learning models on massive datasets. However, SGD suffers from two main drawbacks: (i) The noisy gradient updates have high variance, which slows down convergence as the iterates approach the optimum, and (ii) SGD scales poorly in distributed settings, typically experiencing rapidly decreasing marginal benefits as the number of workers increases. In this paper, we propose a highly parallel method, CentralVR, that uses error corrections to reduce the variance of SGD gradient updates, and scales linearly with the number of worker nodes. CentralVR enjoys low iteration complexity, provably linear convergence rates, and exhibits linear performance gains up to hundreds of cores for massive datasets. We compare CentralVR to state-of-the-art parallel stochastic optimization methods on a variety of models and datasets, and find that our proposed methods exhibit stronger scaling than other SGD variants.
Rectangular Kronecker coefficients and plethysms in geometric complexity theory
Venue: arXiv cs.CC 2017
Year: 2017

We prove that in the geometric complexity theory program the vanishing of rectangular Kronecker coefficients cannot be used to prove superpolynomial determinantal complexity lower bounds for the permanent polynomial. Moreover, we prove the positivity of rectangular Kronecker coefficients for a large class of partitions where the side lengths of the rectangle are at least quadratic in the length of the partition. We also compare rectangular Kronecker coefficients with their corresponding plethysm coefficients, which leads to a new lower bound for rectangular Kronecker coefficients. Moreover, we prove that the saturation of the rectangular Kronecker semigroup is trivial, we show that the rectangular Kronecker positivity stretching factor is 2 for a long first row, and we completely classify the positivity of rectangular limit Kronecker coefficients that were introduced by Manivel in 2011.
Small-footprint Deep Neural Networks with Highway Connections for Speech Recognition
Venue: arXiv cs.CL 2017
Year: 2017

For speech recognition, deep neural networks (DNNs) have significantly improved the recognition accuracy in most of benchmark datasets and application domains. However, compared to the conventional Gaussian mixture models, DNN-based acoustic models usually have much larger number of model parameters, making it challenging for their applications in resource constrained platforms, e.g., mobile devices. In this paper, we study the application of the recently proposed highway network to train small-footprint DNNs, which are {\it thinner} and {\it deeper}, and have significantly smaller number of model parameters compared to conventional DNNs. We investigated this approach on the AMI meeting speech transcription corpus which has around 70 hours of audio data. The highway neural networks constantly outperformed their plain DNN counterparts, and the number of model parameters can be reduced significantly without sacrificing the recognition accuracy.
Visibly Linear Dynamic Logic
Venue: arXiv cs.LO 2017
Year: 2017

We introduce Visibly Linear Dynamic Logic (VLDL), which extends Linear Temporal Logic (LTL) by temporal operators that are guarded by visibly pushdown languages over finite words. In VLDL one can, e.g., express that a function resets a variable to its original value after its execution, even in the presence of an unbounded number of intermediate recursive calls. We prove that VLDL describes exactly the $ω$-visibly pushdown languages. Thus it is strictly more expressive than LTL and able to express recursive properties of programs with unbounded call stacks. The main technical contribution of this work is a translation of VLDL into $ω$-visibly pushdown automata of exponential size via one-way alternating jumping automata. This translation yields exponential-time algorithms for satisfiability, validity, and model checking. We also show that visibly pushdown games with VLDL winning conditions are solvable in triply-exponential time. We prove all these problems to be complete for their respective complexity classes.
Multiregion Bilinear Convolutional Neural Networks for Person Re-Identification
Venue: arXiv cs.CV 2017
Year: 2017

In this work we propose a new architecture for person re-identification. As the task of re-identification is inherently associated with embedding learning and non-rigid appearance description, our architecture is based on the deep bilinear convolutional network (Bilinear-CNN) that has been proposed recently for fine-grained classification of highly non-rigid objects. While the last stages of the original Bilinear-CNN architecture completely removes the geometric information from consideration by performing orderless pooling, we observe that a better embedding can be learned by performing bilinear pooling in a more local way, where each pooling is confined to a predefined region. Our architecture thus represents a compromise between traditional convolutional networks and bilinear CNNs and strikes a balance between rigid matching and completely ignoring spatial information. We perform the experimental validation of the new architecture on the three popular benchmark datasets (Market-1501, CUHK01, CUHK03), comparing it to baselines that include Bilinear-CNN as well as prior art. The new architecture outperforms the baseline on all three datasets, while performing better than state-of-the-art on two out of three. The code and the pretrained models of the approach can be found at https://github.com/madkn/MultiregionBilinearCNN-ReId.
A Lower Bound on the Entropy Rate for a Large Class of Stationary Processes and its Relation to the Hyperplane Conjecture
Venue: arXiv cs.IT 2017
Year: 2017

We present a new lower bound on the differential entropy rate of stationary processes whose sequences of probability density functions fulfill certain regularity conditions. This bound is obtained by showing that the gap between the differential entropy rate of such a process and the differential entropy rate of a Gaussian process with the same autocovariance function is bounded. This result is based on a recent result on bounding the Kullback-Leibler divergence by the Wasserstein distance given by Polyanskiy and Wu. Moreover, it is related to the famous hyperplane conjecture, also known as slicing problem, in convex geometry originally stated by J. Bourgain. Based on an entropic formulation of the hyperplane conjecture given by Bobkov and Madiman we discuss the relation of our result to the hyperplane conjecture.
Optimization of Tree Modes for Parallel Hash Functions: A Case Study
Venue: arXiv cs.DC 2017
Year: 2017

This paper focuses on parallel hash functions based on tree modes of operation for an inner Variable-Input-Length function. This inner function can be either a single-block-length (SBL) and prefix-free MD hash function, or a sponge-based hash function. We discuss the various forms of optimality that can be obtained when designing parallel hash functions based on trees where all leaves have the same depth. The first result is a scheme which optimizes the tree topology in order to decrease the running time. Then, without affecting the optimal running time we show that we can slightly change the corresponding tree topology so as to minimize the number of required processors as well. Consequently, the resulting scheme decreases in the first place the running time and in the second place the number of required processors.
Inside the clustering window for random linear equations
Venue: arXiv cs.CC 2017
Year: 2017

We study a random system of cn linear equations over n variables in GF(2), where each equation contains exactly r variables; this is equivalent to r-XORSAT. Previous work has established a clustering threshold, c^*_r for this model: if c=c_r^*-εfor any constant ε>0 then with high probability all solutions form a well-connected cluster; whereas if c=c^*_r+ε, then with high probability the solutions partition into well-connected, well-separated clusters (with probability tending to 1 as n goes to infinity). This is part of a general clustering phenomenon which is hypothesized to arise in most of the commonly studied models of random constraint satisfaction problems, via sophisticated but mostly non-rigorous techniques from statistical physics. We extend that study to the range c=c^*_r+o(1), and prove that the connectivity parameters of the r-XORSAT clusters undergo a smooth transition around the clustering threshold.
Polynomially Solvable Instances of the Shortest and Closest Vector Problems with Applications to Compute-and-Forward
Venue: arXiv cs.IT 2017
Year: 2017

A particular instance of the Shortest Vector Problem (SVP) appears in the context of Compute-and-Forward. Despite the NP-hardness of the SVP, we will show that this certain instance can be solved in complexity order $O(nψ\log(nψ))$ where $ψ= \sqrt{P\|{\bf h}\|^2+1}$ depends on the transmission power and the norm of the channel vector. We will then extend our results to Integer-Forcing and finally, introduce a more general class of lattices for which the SVP and the and the Closest Vector Problem (CVP) can be approximated within a constant factor.
Blind Deconvolution Meets Blind Demixing: Algorithms and Performance Bounds
Venue: arXiv cs.IT 2017
Year: 2017

Suppose that we have $r$ sensors and each one intends to send a function $\boldsymbol{g}_i$ (e.g.\ a signal or an image) to a receiver common to all $r$ sensors. During transmission, each $\boldsymbol{g}_i$ gets convolved with a function $\boldsymbol{f}_i$. The receiver records the function $\boldsymbol{y}$, given by the sum of all these convolved signals. When and under which conditions is it possible to recover the individual signals $\boldsymbol{g}_i$ and the blurring functions $\boldsymbol{f}_i$ from just one received signal $\boldsymbol{y}$? This challenging problem, which intertwines blind deconvolution with blind demixing, appears in a variety of applications, such as audio processing, image processing, neuroscience, spectroscopy, and astronomy. It is also expected to play a central role in connection with the future Internet-of-Things. We will prove that under reasonable and practical assumptions, it is possible to solve this otherwise highly ill-posed problem and recover the $r$ transmitted functions $\boldsymbol{g}_i$ and the impulse responses $\boldsymbol{f}_i$ in a robust, reliable, and efficient manner from just one single received function $\boldsymbol{y}$ by solving a semidefinite program. We derive explicit bounds on the number of measurements needed for successful recovery and prove that our method is robust in the presence of noise. Our theory is actually sub-optimal, since numerical experiments demonstrate that, quite remarkably, recovery is still possible if the number of measurements is close to the number of degrees of freedom.
Classification of graphs based on homotopy equivalence. Homotopy equivalent graphs. Basic graphs and complexity of homotopy equivalence classes of graphs
Venue: arXiv cs.DM 2017
Year: 2017

Graph classification plays an important role is data mining, and various methods have been developed recently for classifying graphs. In this paper, we propose a novel method for graph classification that is based on homotopy equivalence of graphs. Graphs are called homotopy equivalent if one of them can be converted to the other one by contractible transformations. A basic graph and the complexity of a homotopy equivalence class are defined and investigated. It is shown all graphs belonging to a given homotopy equivalence class have similar topological properties and are represented by a basic graph with the minimal number of points and edges. Diagrams are given of basic graphs with the complexity N<7. The advantage of this classification is that it relies on computer experiments demonstrating a close connection between homotopy equivalent topological spaces and homotopy equivalent graphs.
A Lambda-Calculus Foundation for Universal Probabilistic Programming
Venue: arXiv cs.PL 2017
Year: 2017

We develop the operational semantics of an untyped probabilistic lambda-calculus with continuous distributions, as a foundation for universal probabilistic programming languages such as Church, Anglican, and Venture. Our first contribution is to adapt the classic operational semantics of lambda-calculus to a continuous setting via creating a measure space on terms and defining step-indexed approximations. We prove equivalence of big-step and small-step formulations of this distribution-based semantics. To move closer to inference techniques, we also define the sampling-based semantics of a term as a function from a trace of random samples to a value. We show that the distribution induced by integrating over all traces equals the distribution-based semantics. Our second contribution is to formalize the implementation technique of trace Markov chain Monte Carlo (MCMC) for our calculus and to show its correctness. A key step is defining sufficient conditions for the distribution induced by trace MCMC to converge to the distribution-based semantics. To the best of our knowledge, this is the first rigorous correctness proof for trace MCMC for a higher-order functional language.
Ideal Databases
Venue: arXiv cs.DB 2017
Year: 2017

From algebraic geometry perspective database relations are succinctly defined as Finite Varieties. After establishing basic framework, we give analytic proof of Heath theorem from Database Dependency theory. Next, we leverage Algebra/Geometry dictionary and focus on algebraic counterparts of finite varieties, polynomial ideals. It is well known that intersection and sum of ideals are lattice operations. We generalize this fact to ideals from different rings, therefore establishing that algebra of ideals is Relational Lattice. The final stop is casting the framework into Linear Algebra, and traversing to Quantum Theory.
Compressed sensing with local structure: uniform recovery guarantees for the sparsity in levels class
Venue: arXiv cs.IT 2017
Year: 2017

In compressed sensing, it is often desirable to consider signals possessing additional structure beyond sparsity. One such structured signal model - which forms the focus of this paper - is the local sparsity in levels class. This class has recently found applications in problems such as compressive imaging, multi-sensor acquisition systems and sparse regularization in inverse problems. In this paper we present uniform recovery guarantees for this class when the measurement matrix corresponds to a subsampled isometry. We do this by establishing a variant of the standard restricted isometry property for sparse in levels vectors, known as the restricted isometry property in levels. Interestingly, besides the usual log factors, our uniform recovery guarantees are simpler and less stringent than existing nonuniform recovery guarantees. For the particular case of discrete Fourier sampling with Haar wavelet sparsity, a corollary of our main theorem yields a new recovery guarantee which improves over the current state-of-the-art.
Automatic Description Generation from Images: A Survey of Models, Datasets, and Evaluation Measures
Venue: arXiv cs.CL 2017
Year: 2017

Automatic description generation from natural images is a challenging problem that has recently received a large amount of interest from the computer vision and natural language processing communities. In this survey, we classify the existing approaches based on how they conceptualize this problem, viz., models that cast description as either generation problem or as a retrieval problem over a visual or multimodal representational space. We provide a detailed review of existing models, highlighting their advantages and disadvantages. Moreover, we give an overview of the benchmark image datasets and the evaluation measures that have been developed to assess the quality of machine-generated image descriptions. Finally we extrapolate future directions in the area of automatic image description generation.
Probabilistic Inference of Twitter Users' Age based on What They Follow
Venue: arXiv cs.SI 2017
Year: 2017

Twitter provides an open and rich source of data for studying human behaviour at scale and is widely used in social and network sciences. However, a major criticism of Twitter data is that demographic information is largely absent. Enhancing Twitter data with user ages would advance our ability to study social network structures, information flows and the spread of contagions. Approaches toward age detection of Twitter users typically focus on specific properties of tweets, e.g., linguistic features, which are language dependent. In this paper, we devise a language-independent methodology for determining the age of Twitter users from data that is native to the Twitter ecosystem. The key idea is to use a Bayesian framework to generalise ground-truth age information from a few Twitter users to the entire network based on what/whom they follow. Our approach scales to inferring the age of 700 million Twitter accounts with high accuracy.
Construction of a Channel Code from an Arbitrary Source Code with Decoder Side Information
Venue: arXiv cs.IT 2017
Year: 2017

The construction of a channel code by using a source code with decoder side information is introduced. For the construction, any pair of encoder and decoder is available for a source code with decoder side information. A constrained-random-number generator, which generates random numbers satisfying a condition specified by a function and its value, is used to construct a stochastic channel encoder. The result suggests that we can divide the channel coding problem into the problems of channel encoding and source decoding with side information.
Polarization Sensitive Array Based Physical-Layer Security
Venue: arXiv cs.IT 2017
Year: 2017

We propose a comprehensive framework exploiting the polarization sensitive array (PLA) to improve the physical layer security of wireless communications. More specifically, the polarization difference among signals is utilized to improve the secrecy rate of wireless communications, especially when these signals are spatially indistinguishable. We firstly investigate the PLA based secure communications for point-to-point wireless systems from the perspectives of both total power minimization and secrecy rate maximization. We then take a step further to apply the PLA based secure beamforming designs to relaying networks. The secrecy rate maximization for relaying networks is discussed in detail under both the perfect channel state information and the polarization sensitive array pointing error. In the later case, a robust scheme to achieve secure communications for relaying networks is proposed. Simulation results show that the proposed PLA based algorithms achieve lower total power consumption and better security performance compared with the conventional scalar array designs, especially under challenging environments where all received signals at destination are difficult to be distinguished from the spatial domain.
Edge coloring in unstructured CFD codes
Venue: arXiv cs.DC 2017
Year: 2017

We propose a way of preventing race conditions in the evaluation of the surface integral contribution in discontinuous Galerkin and finite volume flow solvers by coloring the edges (or faces) of the computational mesh. In this work we use a partitioning algorithm that separates the edges of triangular elements into three groups and the faces of quadrangular and tetrahedral elements into four groups; we then extend this partitioning to adaptively refined, nonconforming meshes. We use the ascribed coloring to reduce code memory requirements and optimize accessing the elemental data in memory. This process reduces memory access latencies and speeds up computations on graphics processing units.
Simple Online and Realtime Tracking
Venue: arXiv cs.CV 2017
Year: 2017

This paper explores a pragmatic approach to multiple object tracking where the main focus is to associate objects efficiently for online and realtime applications. To this end, detection quality is identified as a key factor influencing tracking performance, where changing the detector can improve tracking by up to 18.9%. Despite only using a rudimentary combination of familiar techniques such as the Kalman Filter and Hungarian algorithm for the tracking components, this approach achieves an accuracy comparable to state-of-the-art online trackers. Furthermore, due to the simplicity of our tracking method, the tracker updates at a rate of 260 Hz which is over 20x faster than other state-of-the-art trackers.
On Properties of the Support of Capacity-Achieving Distributions for Additive Noise Channel Models with Input Cost Constraints
Venue: arXiv cs.IT 2017
Year: 2017

We study the classical problem of characterizing the channel capacity and its achieving distribution in a generic fashion. We derive a simple relation between three parameters: the input-output function, the input cost function and the noise probability density function, one which dictates the type of the optimal input. In Layman terms we prove that the support of the optimal input is bounded whenever the cost grows faster than a cut-off rate equal to the logarithm of the noise PDF evaluated at the input-output function. Furthermore, we prove a converse statement that says whenever the cost grows slower than the cut-off rate, the optimal input has necessarily an unbounded support. In addition, we show how the discreteness of the optimal input is guaranteed whenever the triplet satisfy some analyticity properties. We argue that a suitable cost function to be imposed on the channel input is one that grows similarly to the cut-off rate. Our results are valid for any cost function that is super-logarithmic. They summarize a large number of previous channel capacity results and give new ones for a wide range of communication channel models, such as Gaussian mixtures, generalized-Gaussians and heavy-tailed noise models, that we state along with numerical computations.
Random Forest DBSCAN for USPTO Inventor Name Disambiguation
Venue: arXiv cs.IR 2017
Year: 2017

Name disambiguation and the subsequent name conflation are essential for the correct processing of person name queries in a digital library or other database. It distinguishes each unique person from all other records in the database. We study inventor name disambiguation for a patent database using methods and features from earlier work on author name disambiguation and propose a feature set appropriate for a patent database. A random forest was selected for the pairwise linking classifier since they outperform Naive Bayes, Logistic Regression, Support Vector Machines (SVM), Conditional Inference Tree, and Decision Trees. Blocking size, very important for scaling, was selected based on experiments that determined feature importance and accuracy. The DBSCAN algorithm is used for clustering records, using a distance function derived from random forest classifier. For additional scalability clustering was parallelized. Tests on the USPTO patent database show that our method successfully disambiguated 12 million inventor mentions within 6.5 hours. Evaluation on datasets from USPTO PatentsView inventor name disambiguation competition shows our algorithm outperforms all algorithms in the competition.