OpenRaiser/Intern-Atlas huggingface dataset:
On a Sparse Shortcut Topology of Artificial Neural Networks
Venue: arXiv cs.LG 2021
Year: 2021

In established network architectures, shortcut connections are often used to take the outputs of earlier layers as additional inputs to later layers. Despite the extraordinary effectiveness of shortcuts, there remain open questions on the mechanism and characteristics. For example, why are shortcuts powerful? Why do shortcuts generalize well? In this paper, we investigate the expressivity and generalizability of a novel sparse shortcut topology. First, we demonstrate that this topology can empower a one-neuron-wide deep network to approximate any univariate continuous function. Then, we present a novel width-bounded universal approximator in contrast to depth-bounded universal approximators and extend the approximation result to a family of equally competent networks. Furthermore, with generalization bound theory, we show that the proposed shortcut topology enjoys excellent generalizability. Finally, we corroborate our theoretical analyses by comparing the proposed topology with popular architectures, including ResNet and DenseNet, on well-known benchmarks and perform a saliency map analysis to interpret the proposed topology. Our work helps enhance the understanding of the role of shortcuts and suggests further opportunities to innovate neural architectures.
SNOW Revisited: Understanding When Ideal READ Transactions Are Possible
Venue: arXiv cs.DC 2021
Year: 2021

READ transactions that read data distributed across servers dominate the workloads of real-world distributed storage systems. The SNOW Theorem stated that ideal READ transactions that have optimal latency and the strongest guarantees, i.e. "SNOW" READ transactions, are impossible in one specific setting that requires three or more clients: at least two readers and one writer. However, it left many open questions. We close all of these open questions with new impossibility results and new algorithms. First, we prove rigorously the result from the The SNOW Theorem paper saying that it is impossible to have a READ transactions system that satisfies SNOW properties with three or more clients. The insight we gained from this proof led to teasing out the implicit assumptions that are required to state the results and also, resolving the open question regarding the possibility of SNOW with two clients. We show that it is possible to design an algorithm, where SNOW is possible in a multi-writer, single-reader (MWSR) setting when a client can send messages to other clients; on the other hand, we prove it is impossible to implement SNOW in a multi-writer, single-reader (MWSR) setting, which is more general than the two-client setting, when client-to-client communication is disallowed. We also correct the previous claim in The SNOW Theorem paper that incorrectly identified one existing system, Eiger, as supporting the strongest guarantees (SW) and whose read-only transactions had bounded latency. Thus, there were no previous algorithms that provided the strongest guarantees and had bounded latency. Finally, we introduce the first two algorithms to provide the strongest guarantees with bounded latency.
Class of scalable parallel and vectorizable pseudorandom number generators based on non-cryptographic RSA exponentiation ciphers
Venue: arXiv cs.CR 2021
Year: 2021

Parallel supercomputer-based Monte Carlo and stochastic simulations require pseudorandom number generators that can produce distinct pseudorandom streams across many independent processes. We propose a scalable class of parallel and vectorizable pseudorandom number generators based on a non-cryptographic version of the RSA public-key exponentiation cipher. Our method generates uniformly distributed IEEE double-precision floating point pseudorandom sequences on $[0,1)$ by encrypting pseudorandom sequences of 64-bit integer messages by modular exponentiation. The advantages of the method are: the method is parallelizable by parameterization with each pseudorandom number generator instance derived from an independent 64-bit composite modulus, the method is fully scalable on massively parallel computing clusters because of the millions of available 32-bit prime numbers, the seeding and initialization of the independent streams is simple, the periods of the independent instances are all different and greater than $8.5\times 10^{37}$, and the method passes a battery of intrastream and interstream correlation tests. The calculations in each instance can be vectorized using steam splitting and can produce more than $10^8$ pseudorandom numbers per second on each multicore CPU.
A Comparison of Automatic Differentiation and Continuous Sensitivity Analysis for Derivatives of Differential Equation Solutions
Venue: arXiv cs.NA 2021
Year: 2021

Derivatives of differential equation solutions are commonly for parameter estimation, fitting neural differential equations, and as model diagnostics. However, with a litany of choices and a Cartesian product of potential methods, it can be difficult for practitioners to understand which method is likely to be the most effective on their particular application. In this manuscript we investigate the performance characteristics of Discrete Local Sensitivity Analysis implemented via Automatic Differentiation (DSAAD) against continuous adjoint sensitivity analysis. Non-stiff and stiff biological and pharmacometric models, including a PDE discretization, are used to quantify the performance of sensitivity analysis methods. Our benchmarks show that on small systems of ODEs (approximately $<100$ parameters+ODEs), forward-mode DSAAD is more efficient than both reverse-mode and continuous forward/adjoint sensitivity analysis. The scalability of continuous adjoint methods is shown to be more efficient than discrete adjoints and forward methods after crossing this size range. These comparative studies demonstrate a trade-off between memory usage and performance in the continuous adjoint methods that should be considered when choosing the technique, while numerically unstable backsolve techniques from the machine learning literature are demonstrated as unsuitable for most scientific models. The performance of adjoint methods is shown to be heavily tied to the reverse-mode AD method, with tape-based AD methods shown to be 2 orders of magnitude slower on nonlinear partial differential equations than static AD techniques. These results also demonstrate the applicability of DSAAD to differential-algebraic equations, delay differential equations, and hybrid differential equation systems, showcasing an ease of implementation advantage for DSAAD approaches.
Euler Transformation of Polyhedral Complexes
Venue: arXiv cs.CG 2021
Year: 2021

We propose an Euler transformation that transforms a given $d$-dimensional cell complex $K$ for $d=2,3$ into a new $d$-complex $\hat{K}$ in which every vertex is part of a uniform even number of edges. Hence every vertex in the graph $\hat{G}$ that is the $1$-skeleton of $\hat{K}$ has an even degree, which makes $\hat{G}$ Eulerian, i.e., it is guaranteed to contain an Eulerian tour. Meshes whose edges admit Eulerian tours are crucial in coverage problems arising in several applications including 3D printing and robotics. For $2$-complexes in $\mathbb{R}^2$ ($d=2$) under mild assumptions (that no two adjacent edges of a $2$-cell in $K$ are boundary edges), we show that the Euler transformed $2$-complex $\hat{K}$ has a geometric realization in $\mathbb{R}^2$, and that each vertex in its $1$-skeleton has degree $4$. We bound the numbers of vertices, edges, and $2$-cells in $\hat{K}$ as small scalar multiples of the corresponding numbers in $K$. We prove corresponding results for $3$-complexes in $\mathbb{R}^3$ under an additional assumption that the degree of a vertex in each $3$-cell containing it is $3$. In this setting, every vertex in $\hat{G}$ is shown to have a degree of $6$. We also present bounds on parameters measuring geometric quality (aspect ratios, minimum edge length, and maximum angle) of $\hat{K}$ in terms of the corresponding parameters of $K$ (for $d=2, 3$). Finally, we illustrate a direct application of the proposed Euler transformation in additive manufacturing.
Compression with wildcards: Abstract simplicial complexes
Venue: arXiv cs.DS 2021
Year: 2021

Despite the more handy terminology of abstract simplicial complexes SC, in its core this article is about antitone Boolean functions. Given the maximal faces (=facets) of SC, our main algorithm, called Facets-To-Faces, outputs SC in a compressed format. The degree of compression of Facets-To-Faces, which is programmed in high-level Mathematica code, compares favorably to both the Mathematica command BooleanConvert, and to the BDD's provided by Python. A novel way to calculate the face-numbers from the facets is also presented. Both algorithms can be parallelized and are applicable (e.g.) to reliability analysis, combinatorial topology, and frequent set mining.
Physics-Based Learning for Robotic Environmental Sensing
Venue: arXiv cs.RO 2021
Year: 2021

We propose a physics-based method to learn environmental fields (EFs) using a mobile robot. Common purely data-driven methods require prohibitively many measurements to accurately learn such complex EFs. Alternatively, physics-based models provide global knowledge of EFs but require experimental validation, depend on uncertain parameters, and are intractable for mobile robots. To address these challenges, we propose a Bayesian framework to select the most likely physics-based models of EFs in real-time, from a pool of numerical solutions generated offline as a function of the uncertain parameters. Specifically, we focus on turbulent flow fields and utilize Gaussian processes (GPs) to construct statistical models for them, using the pool of numerical solutions to inform their prior mean. To incorporate flow measurements into these GPs, we control a custom-built mobile robot through a sequence of waypoints that maximize the information content of the measurements. We experimentally demonstrate that our proposed framework constructs a posterior distribution of the flow field that better approximates the real flow compared to the prior numerical solutions and purely data-driven methods.
Completing the Picture: Complexity of Graded Modal Logics with Converse
Venue: arXiv cs.LO 2021
Year: 2021

A complete classification of the complexity of the local and global satisfiability problems for graded modal language over traditional classes of frames have already been established. By "traditional" classes of frames, we mean those characterized by any positive combination of reflexivity, seriality, symmetry, transitivity, and the Euclidean property. In this paper, we fill the gaps remaining in an analogous classification of the graded modal language with graded converse modalities. In particular, we show its NExpTime-completeness over the class of Euclidean frames, demonstrating this way that over this class the considered language is harder than the language without graded modalities or without converse modalities. We also consider its variation disallowing graded converse modalities, but still admitting basic converse modalities. Our most important result for this variation is confirming an earlier conjecture that it is decidable over transitive frames. This contrasts with the undecidability of the language with graded converse modalities.
A First Look at Deep Learning Apps on Smartphones
Venue: arXiv cs.LG 2021
Year: 2021

We are in the dawn of deep learning explosion for smartphones. To bridge the gap between research and practice, we present the first empirical study on 16,500 the most popular Android apps, demystifying how smartphone apps exploit deep learning in the wild. To this end, we build a new static tool that dissects apps and analyzes their deep learning functions. Our study answers threefold questions: what are the early adopter apps of deep learning, what do they use deep learning for, and how do their deep learning models look like. Our study has strong implications for app developers, smartphone vendors, and deep learning R\&D. On one hand, our findings paint a promising picture of deep learning for smartphones, showing the prosperity of mobile deep learning frameworks as well as the prosperity of apps building their cores atop deep learning. On the other hand, our findings urge optimizations on deep learning models deployed on smartphones, the protection of these models, and validation of research ideas on these models.
Construction of Differentially Private Empirical Distributions from a low-order Marginals Set through Solving Linear Equations with l2 Regularization
Venue: arXiv cs.LG 2021
Year: 2021

We introduce a new algorithm, Construction of dIfferentially Private Empirical Distributions from a low-order marginals set tHrough solving linear Equations with l2 Regularization (CIPHER), that produces differentially private empirical joint distributions from a set of low-order marginals. CIPHER is conceptually simple and requires no more than decomposing joint probabilities via basic probability rules to construct a linear equation set and subsequently solving the equations. Compared to the full-dimensional histogram (FDH) sanitization, CIPHER has drastic\-ally lower requirements on computational storage and memory, which is practically attractive especially considering that the high-order signals preserved by the FDH sanitization are likely just sample randomness and rarely of interest. Our experiments demonstrate that CIPHER outperforms the multiplicative weighting exponential mechanism in preserving original information and has similar or superior cost-normalized utility to FDH sanitization at the same privacy budget.
Quantum query complexity of symmetric oracle problems
Venue: arXiv cs.CC 2021
Year: 2021

We study the query complexity of quantum learning problems in which the oracles form a group $G$ of unitary matrices. In the simplest case, one wishes to identify the oracle, and we find a description of the optimal success probability of a $t$-query quantum algorithm in terms of group characters. As an application, we show that $Ω(n)$ queries are required to identify a random permutation in $S_n$. More generally, suppose $H$ is a fixed subgroup of the group $G$ of oracles, and given access to an oracle sampled uniformly from $G$, we want to learn which coset of $H$ the oracle belongs to. We call this problem coset identification and it generalizes a number of well-known quantum algorithms including the Bernstein-Vazirani problem, the van Dam problem and finite field polynomial interpolation. We provide character-theoretic formulas for the optimal success probability achieved by a $t$-query algorithm for this problem. One application involves the Heisenberg group and provides a family of problems depending on $n$ which require $n+1$ queries classically and only $1$ query quantumly.
An Efficient Algorithm for Generalized Polynomial Partitioning and Its Applications
Venue: arXiv cs.CG 2021
Year: 2021

In 2015, Guth proved that if $S$ is a collection of $n$ $g$-dimensional semi-algebraic sets in $\mathbb{R}^d$ and if $D\geq 1$ is an integer, then there is a $d$-variate polynomial $P$ of degree at most $D$ so that each connected component of $\mathbb{R}^d\setminus Z(P)$ intersects $O(n/D^{d-g})$ sets from $S$. Such a polynomial is called a generalized partitioning polynomial. We present a randomized algorithm that computes such polynomials efficiently -- the expected running time of our algorithm is linear in $|S|$. Our approach exploits the technique of quantifier elimination combined with that of $ε$-samples. We also present an extension of our construction to multi-level polynomial partitioning for semi-algebraic sets in $\mathbb{R}^d$. We present five applications of our result. The first is a data structure for answering point-enclosure queries among a family of semi-algebraic sets in $\mathbb{R}^d$ in $O(\log n)$ time, with storage complexity and expected preprocessing time of $O(n^{d+ε})$. The second is a data structure for answering range-searching queries with semi-algebraic ranges in $\mathbb{R}^d$ in $O(\log n)$ time, with $O(n^{t+ε})$ storage and expected preprocessing time, where $t > 0$ is an integer that depends on $d$ and the description complexity of the ranges. The third is a data structure for answering vertical ray-shooting queries among semi-algebraic sets in $\mathbb{R}^{d}$ in $O(\log^2 n)$ time, with $O(n^{d+ε})$ storage and expected preprocessing time. The fourth is an efficient algorithm for cutting algebraic curves in $\mathbb{R}^2$ into pseudo-segments. The fifth application is for eliminating depth cycles among triangles in $\mathbb{R}^3$, where we show a nearly-optimal algorithm to cut $n$ pairwise disjoint non-vertical triangles in $\mathbb{R}^3$ into pieces that form a depth order.
Fair Coresets and Streaming Algorithms for Fair k-Means Clustering
Venue: arXiv cs.DS 2021
Year: 2021

We study fair clustering problems as proposed by Chierichetti et al. (NIPS 2017). Here, points have a sensitive attribute and all clusters in the solution are required to be balanced with respect to it (to counteract any form of data-inherent bias). Previous algorithms for fair clustering do not scale well. We show how to model and compute so-called coresets for fair clustering problems, which can be used to significantly reduce the input data size. We prove that the coresets are composable and show how to compute them in a streaming setting. Furthermore, we propose a variant of Lloyd's algorithm that computes fair clusterings and extend it to a fair k-means++ clustering algorithm. We implement these algorithms and provide empirical evidence that the combination of our approximation algorithms and the coreset construction yields a scalable algorithm for fair k-means clustering.
Soft Autoencoder and Its Wavelet Adaptation Interpretation
Venue: arXiv cs.LG 2021
Year: 2021

Recently, deep learning becomes the main focus of machine learning research and has greatly impacted many important fields. However, deep learning is criticized for lack of interpretability. As a successful unsupervised model in deep learning, the autoencoder embraces a wide spectrum of applications, yet it suffers from the model opaqueness as well. In this paper, we propose a new type of convolutional autoencoders, termed as Soft Autoencoder (Soft-AE), in which the activation functions of encoding layers are implemented with adaptable soft-thresholding units while decoding layers are realized with linear units. Consequently, Soft-AE can be naturally interpreted as a learned cascaded wavelet shrinkage system. Our denoising experiments demonstrate that Soft-AE not only is interpretable but also offers a competitive performance relative to its counterparts. Furthermore, we propose a generalized linear unit (GenLU) to make an autoencoder more adaptive in nonlinearly filtering images and data, such as denoising and deblurring.
On Reproducing Kernel Banach Spaces: Generic Definitions and Unified Framework of Constructions
Venue: arXiv cs.LG 2021
Year: 2021

Recently, there has been emerging interest in constructing reproducing kernel Banach spaces (RKBS) for applied and theoretical purposes such as machine learning, sampling reconstruction, sparse approximation and functional analysis. Existing constructions include the reflexive RKBS via a bilinear form, the semi-inner-product RKBS, the RKBS with $\ell^1$ norm, the $p$-norm RKBS via generalized Mercer kernels, etc. The definitions of RKBS and the associated reproducing kernel in those references are dependent on the construction. Moreover, relations among those constructions are unclear. We explore a generic definition of RKBS and the reproducing kernel for RKBS that is independent of construction. Furthermore, we propose a framework of constructing RKBSs that unifies existing constructions mentioned above via a continuous bilinear form and a pair of feature maps. A new class of Orlicz RKBSs is proposed. Finally, we develop representer theorems for machine learning in RKBSs constructed in our framework, which also unifies representer theorems in existing RKBSs.
Optimal Age over Erasure Channels
Venue: arXiv cs.IT 2021
Year: 2021

Previous works on age of information and erasure channels have dealt with specific models and computed the average age or average peak age for certain settings. In this paper, given a source that produces a letter every $T_s$ seconds and an erasure channel that can be used every $T_c$ seconds, we ask what is the coding strategy that minimizes the time-average age of information that an observer of the channel output incurs. We first analyze the case where the source alphabet and the channel-input alphabet have the same size. We show that a trivial coding strategy is optimal and a closed form expression for the age can be derived. We then analyze the case where the alphabets have different sizes. We use a random coding argument to bound the average age and show that the average age achieved using random codes converges to the optimal average age of linear block codes as the source alphabet becomes large.
Dynamic Programming for Sequential Deterministic Quantization of Discrete Memoryless Channels
Venue: arXiv cs.IT 2021
Year: 2021

In this paper, under a general cost function $C$, we present a dynamic programming (DP) method to obtain an optimal sequential deterministic quantizer (SDQ) for $q$-ary input discrete memoryless channel (DMC). The DP method has complexity $O(q (N-M)^2 M)$, where $N$ and $M$ are the alphabet sizes of the DMC output and quantizer output, respectively. Then, starting from the quadrangle inequality, two techniques are applied to reduce the DP method's complexity. One technique makes use of the Shor-Moran-Aggarwal-Wilber-Klawe (SMAWK) algorithm and achieves complexity $O(q (N-M) M)$. The other technique is much easier to be implemented and achieves complexity $O(q (N^2 - M^2))$. We further derive a sufficient condition under which the optimal SDQ is optimal among all quantizers and the two techniques are applicable. This generalizes the results in the literature for binary-input DMC. Next, we show that the cost function of $α$-mutual information ($α$-MI)-maximizing quantizer belongs to the category of $C$. We further prove that under a weaker condition than the sufficient condition we derived, the aforementioned two techniques are applicable to the design of $α$-MI-maximizing quantizer. Finally, we illustrate the particular application of our design method to practical pulse-amplitude modulation systems.
On the Robustness of Distributed Computing Networks
Venue: arXiv cs.NI 2021
Year: 2021

Traffic flows in a distributed computing network require both transmission and processing, and can be interdicted by removing either communication or computation resources. We study the robustness of a distributed computing network under the failures of communication links and computation nodes. We define cut metrics that measure the connectivity, and show a non-zero gap between the maximum flow and the minimum cut. Moreover, we study a network flow interdiction problem that minimizes the maximum flow by removing communication and computation resources within a given budget. We develop mathematical programs to compute the optimal interdiction, and polynomial-time approximation algorithms that achieve near-optimal interdiction in simulation.