OpenRaiser/Intern-Atlas huggingface dataset:
The committee machine: Computational to statistical gaps in learning a two-layers neural network
Venue: arXiv cs.LG 2024
Year: 2024

Heuristic tools from statistical physics have been used in the past to locate the phase transitions and compute the optimal learning and generalization errors in the teacher-student scenario in multi-layer neural networks. In this contribution, we provide a rigorous justification of these approaches for a two-layers neural network model called the committee machine. We also introduce a version of the approximate message passing (AMP) algorithm for the committee machine that allows to perform optimal learning in polynomial time for a large set of parameters. We find that there are regimes in which a low generalization error is information-theoretically achievable while the AMP algorithm fails to deliver it, strongly suggesting that no efficient algorithm exists for those cases, and unveiling a large computational gap.
Approximating the shortest path problem with scenarios
Venue: arXiv cs.DS 2024
Year: 2024

This paper discusses the shortest path problem in a general directed graph with $n$ nodes and $K$ cost scenarios (objectives). In order to choose a solution, the min-max criterion is applied. The min-max version of the problem is hard to approximate within $Ω(\log^{1-ε} K)$ for any $ε>0$ unless NP$\subseteq \text{DTIME}(n^{\text{polylog} \,n})$ even for arc series-parallel graphs and within $Ω(\log n/\log\log n)$ unless NP$\subseteq \text{ZPTIME}(n^{\log\log n})$ for acyclic graphs. The best approximation algorithm for the min-max shortest path problem in general graphs, known to date, has an approximation ratio of~$K$. In this paper, an $\widetilde{O}(\sqrt{n})$ flow LP-based approximation algorithm for min-max shortest path in general graphs is constructed. It is also shown that the approximation ratio obtained is close to an integrality gap of the corresponding flow LP relaxation.
Classically Time-Controlled Quantum Automata: Definition and Properties
Venue: arXiv cs.FL 2024
Year: 2024

In this paper, we introduce classically time-controlled quantum automata or CTQA, which is a reasonable modification of Moore-Crutchfield quantum finite automata that uses time-dependent evolution and a "scheduler" defining how long each Hamiltonian will run. Surprisingly enough, time-dependent evolution provides a significant change in the computational power of quantum automata with respect to a discrete quantum model. Indeed, we show that if a scheduler is not computationally restricted, then a CTQA could even decide the Halting problem. In order to unearth the computational capabilities of CTQAs we study the case of a computationally restricted scheduler. In particular, we showed that depending on the type of restriction imposed on the scheduler, a CTQA can (i) recognize non-regular languages with cut-point, even in the presence of Karp-Lipton advice, and (ii) recognize non-regular promise languages with bounded-error. Furthermore, we study the cutpoint-union of cutpoint languages by introducing a new model of Moore-Crutchfield quantum finite automata with a rotating tape head. CTQA presents itself as a new model of computation that provides a different approach to a formal study of "classical control, quantum data" schemes in quantum computing.
Exact Algorithms and Lower Bounds for Stable Instances of Euclidean k-Means
Venue: arXiv cs.DS 2024
Year: 2024

We investigate the complexity of solving stable or perturbation-resilient instances of $k$-Means and $k$-Median clustering in fixed dimension Euclidean metrics (more generally doubling metrics). The notion of stable (perturbation resilient) instances was introduced by Bilu and Linial [2010] and Awasthi et al. [2012]. In our context we say a $k$-Means instance is $α$-stable if there is a unique OPT which remains optimum if distances are (non-uniformly) stretched by a factor of at most $α$. Stable clustering instances have been studied to explain why heuristics such as Lloyd's algorithm perform well in practice. In this work we show that for any fixed $ε>0$, $(1+ε)$-stable instances of $k$-Means in doubling metrics can be solved in polynomial time. More precisely we show a natural multiswap local search algorithm finds OPT for $(1+ε)$-stable instances of $k$-Means and $k$-Median in a polynomial number of iterations. We complement this result by showing that under a new PCP theorem, this is essentially tight: that when the dimension d is part of the input, there is a fixed $ε_0>0$ s.t. there is not even a PTAS for $(1+ε_0)$-stable $k$-Means in $R^d$ unless NP=RP. To do this, we consider a robust property of CSPs; call an instance stable if there is a unique optimum solution $x^*$ and for any other solution $x'$, the number of unsatisfied clauses is proportional to the Hamming distance between $x^*$ and $x'$. Dinur et al. have already shown stable QSAT is hard to approximate for some constant Q, our hypothesis is simply that stable QSAT with bounded variable occurrence is also hard. Given this hypothesis we consider "stability-preserving" reductions to prove our hardness for stable k-Means. Such reductions seem to be more fragile than standard L-reductions and may be of further use to demonstrate other stable optimization problems are hard.
Linear Programming Based Near-Optimal Pricing for Laminar Bayesian Online Selection
Venue: arXiv cs.GT 2024
Year: 2024

The Bayesian online selection problem aims to design a pricing scheme for a sequence of arriving buyers that maximizes the expected social welfare (or revenue) subject to different structural constraints. Inspired by applications with a hierarchy of service, this paper focuses on the cases where a laminar matroid characterizes the set of served buyers. We give the first Polynomial-Time Approximation Scheme (PTAS) for the problem when the laminar matroid has constant depth. Our approach is based on rounding the solution of a hierarchy of linear programming relaxations that approximate the optimum online solution with any degree of accuracy, plus a concentration argument showing that rounding incurs a small loss. We also study another variation, which we call the production-constrained problem. The allowable set of served buyers is characterized by a collection of production and shipping constraints that form a particular example of a laminar matroid. Using a similar LP-based approach, we design a PTAS for this problem, although in this special case the depth of the underlying laminar matroid is not necessarily a constant. The analysis exploits the negative dependency of the optimum selection rule in the lower levels of the laminar family. Finally, to demonstrate the generality of our technique, we employ the linear programming-based approach employed in the paper to re-derive some of the classic prophet inequalities known in the literature -- as a side result.
Evolution of Preferences in Multiple Populations
Venue: arXiv cs.GT 2024
Year: 2024

We study the evolution of preferences in multi-population settings that allow matches across distinct populations. Each individual has subjective preferences over potential outcomes, and chooses a best response based on his preferences and the information about the opponents' preferences. Individuals' realized fitnesses are given by material payoff functions. Following Dekel et al. (2007), we assume that individuals observe their opponents' preferences with probability $p$. We first derive necessary and sufficient conditions for stability for $p=1$ and $p=0$, and then check the robustness of our results against small perturbations on observability for the case of pure-strategy outcomes.
On the Convergence of Adaptive Gradient Methods for Nonconvex Optimization
Venue: arXiv cs.LG 2024
Year: 2024

Adaptive gradient methods are workhorses in deep learning. However, the convergence guarantees of adaptive gradient methods for nonconvex optimization have not been thoroughly studied. In this paper, we provide a fine-grained convergence analysis for a general class of adaptive gradient methods including AMSGrad, RMSProp and AdaGrad. For smooth nonconvex functions, we prove that adaptive gradient methods in expectation converge to a first-order stationary point. Our convergence rate is better than existing results for adaptive gradient methods in terms of dimension. In addition, we also prove high probability bounds on the convergence rates of AMSGrad, RMSProp as well as AdaGrad, which have not been established before. Our analyses shed light on better understanding the mechanism behind adaptive gradient methods in optimizing nonconvex objectives.
Automated Machine Learning: From Principles to Practices
Venue: arXiv cs.AI 2024
Year: 2024

Machine learning (ML) methods have been developing rapidly, but configuring and selecting proper methods to achieve a desired performance is increasingly difficult and tedious. To address this challenge, automated machine learning (AutoML) has emerged, which aims to generate satisfactory ML configurations for given tasks in a data-driven way. In this paper, we provide a comprehensive survey on this topic. We begin with the formal definition of AutoML and then introduce its principles, including the bi-level learning objective, the learning strategy, and the theoretical interpretation. Then, we summarize the AutoML practices by setting up the taxonomy of existing works based on three main factors: the search space, the search algorithm, and the evaluation strategy. Each category is also explained with the representative methods. Then, we illustrate the principles and practices with exemplary applications from configuring ML pipeline, one-shot neural architecture search, and integration with foundation models. Finally, we highlight the emerging directions of AutoML and conclude the survey.
Elastic CRFs for Open-ontology Slot Filling
Venue: arXiv cs.CL 2024
Year: 2024

Slot filling is a crucial component in task-oriented dialog systems that is used to parse (user) utterances into semantic concepts called slots. An ontology is defined by the collection of slots and the values that each slot can take. The most widely used practice of treating slot filling as a sequence labeling task suffers from two main drawbacks. First, the ontology is usually pre-defined and fixed and therefore is not able to detect new labels for unseen slots. Second, the one-hot encoding of slot labels ignores the correlations between slots with similar semantics, which makes it difficult to share knowledge learned across different domains. To address these problems, we propose a new model called elastic conditional random field (eCRF), where each slot is represented by the embedding of its natural language description and modeled by a CRF layer. New slot values can be detected by eCRF whenever a language description is available for the slot. In our experiment, we show that eCRFs outperform existing models in both in-domain and cross-domain tasks, especially in predicting unseen slots and values.
Universal Polarization for Processes with Memory
Venue: arXiv cs.IT 2024
Year: 2024

A transform that is universally polarizing over a set of channels with memory is presented. Memory may be present in both the input to the channel and the channel itself. Both the encoder and the decoder are aware of the input distribution, which is fixed. However, only the decoder is aware of the actual channel being used. The transform can be used to design a universal code for this scenario. The code is to have vanishing error probability when used over any channel in the set, and achieve the infimal information rate over the set. The setting considered is, in fact, more general: we consider a set of processes with memory. Universal polarization is established for the case where each process in the set: (a) has memory in the form of an underlying hidden Markov state sequence that is aperiodic and irreducible, and (b) satisfies a `forgetfulness' property. Forgetfulness, which we believe to be of independent interest, occurs when two hidden Markov states become approximately independent of each other given a sufficiently long sequence of observations between them. We show that aperiodicity and irreducibility of the underlying Markov chain is not sufficient for forgetfulness, and develop a sufficient condition for a hidden Markov process to be forgetful.
Understanding the Topology and the Geometry of the Space of Persistence Diagrams via Optimal Partial Transport
Venue: arXiv cs.CG 2024
Year: 2024

Despite the obvious similarities between the metrics used in topological data analysis and those of optimal transport, an optimal-transport based formalism to study persistence diagrams and similar topological descriptors has yet to come. In this article, by considering the space of persistence diagrams as a space of discrete measures, and by observing that its metrics can be expressed as optimal partial transport problems, we introduce a generalization of persistence diagrams, namely Radon measures supported on the upper half plane. Such measures naturally appear in topological data analysis when considering continuous representations of persistence diagrams (e.g.\ persistence surfaces) but also as limits for laws of large numbers on persistence diagrams or as expectations of probability distributions on the persistence diagrams space. We explore topological properties of this new space, which will also hold for the closed subspace of persistence diagrams. New results include a characterization of convergence with respect to Wasserstein metrics, a geometric description of barycenters (Fréchet means) for any distribution of diagrams, and an exhaustive description of continuous linear representations of persistence diagrams. We also showcase the strength of this framework to study random persistence diagrams by providing several statistical results made meaningful thanks to this new formalism.
A study on general visual categorization of objects into animal and plant groups using global shape descriptors with a focus on category-specific deficits
Venue: arXiv cs.CV 2024
Year: 2024

How do humans distinguish between general categories of objects? In a number of semantic category deficits, patients are good at making broad categorization but are unable to remember fine and specific details. It has been well accepted that general information about concepts is more robust to damages related to semantic memory. Results from patients with semantic memory disorders demonstrate the loss of ability in subcategory recognition. In this paper, we review the behavioral evidence for category specific disorder and show that general categories of animal and plant are visually distinguishable without processing textural information. To this aim, we utilize shape descriptors with an additional phase of feature learning. The results are evaluated with both supervised and unsupervised learning mechanisms and confirm that the proposed method can effectively discriminates between animal and plant object categories in visual domain.
A face cover perspective to $\ell_1$ embeddings of planar graphs
Venue: arXiv cs.DS 2024
Year: 2024

It was conjectured by Gupta et al. [Combinatorica04] that every planar graph can be embedded into $\ell_1$ with constant distortion. However, given an $n$-vertex weighted planar graph, the best upper bound on the distortion is only $O(\sqrt{\log n})$, by Rao [SoCG99]. In this paper we study the case where there is a set $K$ of terminals, and the goal is to embed only the terminals into $\ell_1$ with low distortion. In a seminal paper, Okamura and Seymour [J.Comb.Theory81] showed that if all the terminals lie on a single face, they can be embedded isometrically into $\ell_1$. The more general case, where the set of terminals can be covered by $γ$ faces, was studied by Lee and Sidiropoulos [STOC09] and Chekuri et al. [J.Comb.Theory13]. The state of the art is an upper bound of $O(\log γ)$ by Krauthgamer, Lee and Rika [SODA19]. Our contribution is a further improvement on the upper bound to $O(\sqrt{\logγ})$. Since every planar graph has at most $O(n)$ faces, any further improvement on this result, will be a major breakthrough, directly improving upon Rao's long standing upper bound. Moreover, it is well known that the flow-cut gap equals to the distortion of the best embedding into $\ell_1$. Therefore, our result provides a polynomial time $O(\sqrt{\log γ})$-approximation to the sparsest cut problem on planar graphs, for the case where all the demand pairs can be covered by $γ$ faces.
Parametric Timed Pattern Matching
Venue: arXiv cs.FL 2024
Year: 2024

Given a log and a specification, timed pattern matching aims at exhibiting for which start and end dates a specification holds on that log. For example, "a given action is always followed by another action before a given deadline". This problem has strong connections with monitoring real-time systems. We address here timed pattern matching in the presence of an uncertain specification, i.e., that may contain timing parameters (e.g., the deadline can be uncertain or unknown). We want to know for which start and end dates, and for what values of the timing parameters, a property holds. For instance, we look for the minimum or maximum deadline (together with the corresponding start and end dates) for which the property holds. We propose two frameworks for parametric timed pattern matching. The first one is based on parametric timed model checking. In contrast to most parametric timed problems, the solution is effectively computable. The second one is a dedicated method; not only we largely improve the efficiency compared to the first method, but we further propose optimizations with skipping. Our experiment results suggest that our algorithms, especially the second one, are efficient and practically relevant.
From Abstractions to Grounded Languages for Robust Coordination of Task Planning Robots
Venue: arXiv cs.AI 2024
Year: 2024

In this paper, we consider a first step to bridge a gap in coordinating task planning robots. Specifically, we study the automatic construction of languages that are maximally flexible while being sufficiently explicative for coordination. To this end, we view language as a machinery for specifying temporal-state constraints of plans. Such a view enables us to reverse-engineer a language from the ground up by mapping these composable constraints to words. Our language expresses a plan for any given task as a "plan sketch" to convey just-enough details while maximizing the flexibility to realize it, leading to robust coordination with optimality guarantees among other benefits. We formulate and analyze the problem, provide an approximate solution, and validate the advantages of our approach under various scenarios to shed light on its applications.
Zero-Knowledge Proof-of-Identity: Sybil-Resistant, Anonymous Authentication on Permissionless Blockchains and Incentive Compatible, Strictly Dominant Cryptocurrencies
Venue: arXiv cs.CR 2024
Year: 2024

Zero-Knowledge Proof-of-Identity from trusted public certificates (e.g., national identity cards and/or ePassports; eSIM) is introduced here to permissionless blockchains in order to remove the inefficiencies of Sybil-resistant mechanisms such as Proof-of-Work (i.e., high energy and environmental costs) and Proof-of-Stake (i.e., capital hoarding and lower transaction volume). The proposed solution effectively limits the number of mining nodes a single individual would be able to run while keeping membership open to everyone, circumventing the impossibility of full decentralization and the blockchain scalability trilemma when instantiated on a blockchain with a consensus protocol based on the cryptographic random selection of nodes. Resistance to collusion is also considered. Solving one of the most pressing problems in blockchains, a zk-PoI cryptocurrency is proved to have the following advantageous properties: - an incentive-compatible protocol for the issuing of cryptocurrency rewards based on a unique Nash equilibrium - strict domination of mining over all other PoW/PoS cryptocurrencies, thus the zk-PoI cryptocurrency becoming the preferred choice by miners is proved to be a Nash equilibrium and the Evolutionarily Stable Strategy - PoW/PoS cryptocurrencies are condemned to pay the Price of Crypto-Anarchy, redeemed by the optimal efficiency of zk-PoI as it implements the social optimum - the circulation of a zk-PoI cryptocurrency Pareto dominates other PoW/PoS cryptocurrencies - the network effects arising from the social networks inherent to national identity cards and ePassports dominate PoW/PoS cryptocurrencies - the lower costs of its infrastructure imply the existence of a unique equilibrium where it dominates other forms of payment
Learning Interpretable Models Using Uncertainty Oracles
Venue: arXiv cs.LG 2024
Year: 2024

A desirable property of interpretable models is small size, so that they are easily understandable by humans. This leads to the following challenges: (a) small sizes typically imply diminished accuracy, and (b) bespoke levers provided by model families to restrict size, e.g., L1 regularization, might be insufficient to reach the desired size-accuracy trade-off. We address these challenges here. Earlier work has shown that learning the training distribution creates accurate small models. Our contribution is a new technique that exploits this idea. The training distribution is encoded as a Dirichlet Process to allow for a flexible number of modes that is learnable from the data. Its parameters are learned using Bayesian Optimization; a design choice that makes the technique applicable to non-differentiable loss functions. To avoid the challenges with high dimensionality, the data is first projected down to one-dimension using uncertainty scores of a separate probabilistic model, that we refer to as the uncertainty oracle. We show that this technique addresses the above challenges: (a) it arrests the reduction in accuracy that comes from shrinking a model (in some cases we observe $\sim 100\%$ improvement over baselines), and also, (b) that this maybe applied with no change across model families with different notions of size; results are shown for Decision Trees, Linear Probability models and Gradient Boosted Models. Additionally, we show that (1) it is more accurate than its predecessor, (2) requires only one hyperparameter to be set in practice, (3) accommodates a multi-variate notion of model size, e.g., both maximum depth of a tree and number of trees in Gradient Boosted Models, and (4) works across different feature spaces between the uncertainty oracle and the interpretable model, e.g., a GRU might act as an oracle for a decision tree that ingests n-grams.
ENCORE: Ensemble Learning using Convolution Neural Machine Translation for Automatic Program Repair
Venue: arXiv cs.SE 2024
Year: 2024

Automated generate-and-validate (G&V) program repair techniques typically rely on hard-coded rules, only fix bugs following specific patterns, and are hard to adapt to different programming languages. We propose ENCORE, a new G&V technique, which uses ensemble learning on convolutional neural machine translation (NMT) models to automatically fix bugs in multiple programming languages. We take advantage of the randomness in hyper-parameter tuning to build multiple models that fix different bugs and combine them using ensemble learning. This new convolutional NMT approach outperforms the standard long short-term memory (LSTM) approach used in previous work, as it better captures both local and long-distance connections between tokens. Our evaluation on two popular benchmarks, Defects4J and QuixBugs, shows that ENCORE fixed 42 bugs, including 16 that have not been fixed by existing techniques. In addition, ENCORE is the first G&V repair technique to be applied to four popular programming languages (Java, C++, Python, and JavaScript), fixing a total of 67 bugs across five benchmarks.