OpenRaiser/Intern-Atlas huggingface dataset:
Trains, Games, and Complexity: 0/1/2-Player Motion Planning through Input/Output Gadgets
Venue: arXiv cs.CC 2023
Year: 2023

We analyze the computational complexity of motion planning through local "input/output" gadgets with separate entrances and exits, and a subset of allowed traversals from entrances to exits, each of which changes the state of the gadget and thereby the allowed traversals. We study such gadgets in the zero-, one-, and two-player settings, in particular extending past motion-planning-through-gadgets work [DGLR18, DHL20] to zero-player games for the first time, by considering "branchless" connections between gadgets that route every gadget's exit to a unique gadget's entrance. Our complexity results include containment in L, NL, P, NP, and PSPACE; as well as hardness for NL, P, NP, and PSPACE. We apply these results to show PSPACE-completeness for certain mechanics in the video games Factorio, [the Sequence], and a restricted version of Trainyard, improving the result of [ALP18a]. This work strengthens prior results on switching graphs, ARRIVAL [DGK+17], and reachability switching games [FGMS21].
Fair Algorithms for Hierarchical Agglomerative Clustering
Venue: arXiv cs.LG 2023
Year: 2023

Hierarchical Agglomerative Clustering (HAC) algorithms are extensively utilized in modern data science, and seek to partition the dataset into clusters while generating a hierarchical relationship between the data samples. HAC algorithms are employed in many applications, such as biology, natural language processing, and recommender systems. Thus, it is imperative to ensure that these algorithms are fair -- even if the dataset contains biases against certain protected groups, the cluster outputs generated should not discriminate against samples from any of these groups. However, recent work in clustering fairness has mostly focused on center-based clustering algorithms, such as k-median and k-means clustering. In this paper, we propose fair algorithms for performing HAC that enforce fairness constraints 1) irrespective of the distance linkage criteria used, 2) generalize to any natural measures of clustering fairness for HAC, 3) work for multiple protected groups, and 4) have competitive running times to vanilla HAC. Through extensive experiments on multiple real-world UCI datasets, we show that our proposed algorithm finds fairer clusterings compared to vanilla HAC as well as other state-of-the-art fair clustering approaches.
A Logical Characterization of Constant-Depth Circuits over the Reals
Venue: arXiv cs.CC 2023
Year: 2023

In this paper we give an Immerman's Theorem for real-valued computation. We define circuits operating over real numbers and show that families of such circuits of polynomial size and constant depth decide exactly those sets of vectors of reals that can be defined in first-order logic on R-structures in the sense of Cucker and Meer. Our characterization holds both non-uniformily as well as for many natural uniformity conditions.
Positional Games and QBF: A Polished Encoding
Venue: arXiv cs.LO 2023
Year: 2023

Positional games are a mathematical class of two-player games comprising Tic-tac-toe and its generalizations. We propose a novel encoding of these games into Quantified Boolean Formulas (QBFs) such that a game instance admits a winning strategy for the first player if and only if the corresponding formula is true. Our approach improves over previous QBF encodings of games in multiple ways. First, it is generic and lets us encode other positional games, such as Hex. Second, the structural properties of positional games, together with careful treatment of illegal moves, let us generate more compact instances that can be solved faster by state-of-the-art QBF solvers. We establish the latter fact through extensive experiments. Finally, the compactness of our new encoding makes it feasible to translate realistic game problems. We identify a few such problems of historical significance and put them forward to the QBF community as milestones of increasing difficulty.
Generalization Bounds via Information Density and Conditional Information Density
Venue: arXiv cs.LG 2023
Year: 2023

We present a general approach, based on exponential inequalities, to derive bounds on the generalization error of randomized learning algorithms. Using this approach, we provide bounds on the average generalization error as well as bounds on its tail probability, for both the PAC-Bayesian and single-draw scenarios. Specifically, for the case of sub-Gaussian loss functions, we obtain novel bounds that depend on the information density between the training data and the output hypothesis. When suitably weakened, these bounds recover many of the information-theoretic bounds available in the literature. We also extend the proposed exponential-inequality approach to the setting recently introduced by Steinke and Zakynthinou (2020), where the learning algorithm depends on a randomly selected subset of the available training data. For this setup, we present bounds for bounded loss functions in terms of the conditional information density between the output hypothesis and the random variable determining the subset choice, given all training data. Through our approach, we recover the average generalization bound presented by Steinke and Zakynthinou (2020) and extend it to the PAC-Bayesian and single-draw scenarios. For the single-draw scenario, we also obtain novel bounds in terms of the conditional $α$-mutual information and the conditional maximal leakage.
Semi-Supervised Learning: the Case When Unlabeled Data is Equally Useful
Venue: arXiv cs.LG 2023
Year: 2023

Semi-supervised learning algorithms attempt to take advantage of relatively inexpensive unlabeled data to improve learning performance. In this work, we consider statistical models where the data distributions can be characterized by continuous parameters. We show that under certain conditions on the distribution, unlabeled data is equally useful as labeled date in terms of learning rate. Specifically, let $n, m$ be the number of labeled and unlabeled data, respectively. It is shown that the learning rate of semi-supervised learning scales as $O(1/n)$ if $m\sim n$, and scales as $O(1/n^{1+γ})$ if $m\sim n^{1+γ}$ for some $γ>0$, whereas the learning rate of supervised learning scales as $O(1/n)$.
Solution Path Algorithm for Twin Multi-class Support Vector Machine
Venue: arXiv cs.LG 2023
Year: 2023

The twin support vector machine and its extensions have made great achievements in dealing with binary classification problems. However, it suffers from difficulties in effective solution of multi-classification and fast model selection. This work devotes to the fast regularization parameter tuning algorithm for the twin multi-class support vector machine. Specifically, a novel sample data set partition strategy is first adopted, which is the basis for the model construction. Then, combining the linear equations and block matrix theory, the Lagrangian multipliers are proved to be piecewise linear w.r.t. the regularization parameters, so that the regularization parameters are continuously updated by only solving the break points. Next, Lagrangian multipliers are proved to be 1 as the regularization parameter approaches infinity, thus, a simple yet effective initialization algorithm is devised. Finally, eight kinds of events are defined to seek for the starting event for the next iteration. Extensive experimental results on nine UCI data sets show that the proposed method can achieve comparable classification performance without solving any quadratic programming problem.
PatchUp: A Feature-Space Block-Level Regularization Technique for Convolutional Neural Networks
Venue: arXiv cs.LG 2023
Year: 2023

Large capacity deep learning models are often prone to a high generalization gap when trained with a limited amount of labeled training data. A recent class of methods to address this problem uses various ways to construct a new training sample by mixing a pair (or more) of training samples. We propose PatchUp, a hidden state block-level regularization technique for Convolutional Neural Networks (CNNs), that is applied on selected contiguous blocks of feature maps from a random pair of samples. Our approach improves the robustness of CNN models against the manifold intrusion problem that may occur in other state-of-the-art mixing approaches. Moreover, since we are mixing the contiguous block of features in the hidden space, which has more dimensions than the input space, we obtain more diverse samples for training towards different dimensions. Our experiments on CIFAR10/100, SVHN, Tiny-ImageNet, and ImageNet using ResNet architectures including PreActResnet18/34, WRN-28-10, ResNet101/152 models show that PatchUp improves upon, or equals, the performance of current state-of-the-art regularizers for CNNs. We also show that PatchUp can provide a better generalization to deformed samples and is more robust against adversarial attacks.
Estimates on Learning Rates for Multi-Penalty Distribution Regression
Venue: arXiv cs.LG 2023
Year: 2023

This paper is concerned with functional learning by utilizing two-stage sampled distribution regression. We study a multi-penalty regularization algorithm for distribution regression under the framework of learning theory. The algorithm aims at regressing to real valued outputs from probability measures. The theoretical analysis on distribution regression is far from maturity and quite challenging, since only second stage samples are observable in practical setting. In the algorithm, to transform information from samples, we embed the distributions to a reproducing kernel Hilbert space $\mathcal{H}_K$ associated with Mercer kernel $K$ via mean embedding technique. The main contribution of the paper is to present a novel multi-penalty regularization algorithm to capture more features of distribution regression and derive optimal learning rates for the algorithm. The work also derives learning rates for distribution regression in the nonstandard setting $f_ρ\notin\mathcal{H}_K$, which is not explored in existing literature. Moreover, we propose a distribution regression-based distributed learning algorithm to face large-scale data or information challenge. The optimal learning rates are derived for the distributed learning algorithm. By providing new algorithms and showing their learning rates, we improve the existing work in different aspects in the literature.
First Steps Towards a Runtime Analysis When Starting With a Good Solution
Venue: arXiv cs.NE 2023
Year: 2023

The mathematical runtime analysis of evolutionary algorithms traditionally regards the time an algorithm needs to find a solution of a certain quality when initialized with a random population. In practical applications it may be possible to guess solutions that are better than random ones. We start a mathematical runtime analysis for such situations. We observe that different algorithms profit to a very different degree from a better initialization. We also show that the optimal parameterization of the algorithm can depend strongly on the quality of the initial solutions. To overcome this difficulty, self-adjusting and randomized heavy-tailed parameter choices can be profitable. Finally, we observe a larger gap between the performance of the best evolutionary algorithm we found and the corresponding black-box complexity. This could suggest that evolutionary algorithms better exploiting good initial solutions are still to be found. These first findings stem from analyzing the performance of the $(1+1)$ evolutionary algorithm and the static, self-adjusting, and heavy-tailed $(1 + (λ,λ))$ GA on the OneMax benchmark. We are optimistic that the question how to profit from good initial solutions is interesting beyond these first examples.
Fast Deep Mixtures of Gaussian Process Experts
Venue: arXiv cs.LG 2023
Year: 2023

Mixtures of experts have become an indispensable tool for flexible modelling in a supervised learning context, allowing not only the mean function but the entire density of the output to change with the inputs. Sparse Gaussian processes (GP) have shown promise as a leading candidate for the experts in such models, and in this article, we propose to design the gating network for selecting the experts from such mixtures of sparse GPs using a deep neural network (DNN). Furthermore, a fast one pass algorithm called Cluster-Classify-Regress (CCR) is leveraged to approximate the maximum a posteriori (MAP) estimator extremely quickly. This powerful combination of model and algorithm together delivers a novel method which is flexible, robust, and extremely efficient. In particular, the method is able to outperform competing methods in terms of accuracy and uncertainty quantification. The cost is competitive on low-dimensional and small data sets, but is significantly lower for higher-dimensional and big data sets. Iteratively maximizing the distribution of experts given allocations and allocations given experts does not provide significant improvement, which indicates that the algorithm achieves a good approximation to the local MAP estimator very fast. This insight can be useful also in the context of other mixture of experts models.
Dissecting power of intersection of two context-free languages
Venue: arXiv cs.FL 2023
Year: 2023

We say that a language $L$ is \emph{constantly growing} if there is a constant $c$ such that for every word $u\in L$ there is a word $v\in L$ with $\vert u\vert<\vert v\vert\leq c+\vert u\vert$. We say that a language $L$ is \emph{geometrically growing} if there is a constant $c$ such that for every word $u\in L$ there is a word $v\in L$ with $\vert u\vert<\vert v\vert\leq c\vert u\vert$. Given two infinite languages $L_1,L_2$, we say that $L_1$ \emph{dissects} $L_2$ if $\vert L_2\setminus L_1\vert=\infty$ and $\vert L_1\cap L_2\vert=\infty$. In 2013, it was shown that for every constantly growing language $L$ there is a regular language $R$ such that $R$ dissects $L$. In the current article we show how to dissect a geometrically growing language by a homomorphic image of intersection of two context-free languages. Consider three alphabets $Γ$, $Σ$, and $Θ$ such that $\vert Σ\vert=1$ and $\vert Θ\vert=4$. We prove that there are context-free languages $M_1,M_2\subseteq Θ^*$, an erasing alphabetical homomorphism $π:Θ^*\rightarrow Σ^*$, and a nonerasing alphabetical homomorphism $φ: Γ^*\rightarrow Σ^*$ such that: If $L\subseteq Γ^*$ is a geometrically growing language then there is a regular language $R\subseteq Θ^*$ such that $φ^{-1}\left(π\left(R\cap M_1\cap M_2\right)\right)$ dissects the language $L$.
Interdisciplinary research and technological impact: Evidence from biomedicine
Venue: arXiv cs.DL 2023
Year: 2023

Interdisciplinary research (IDR) has been considered as an important source for scientific breakthroughs and as a solution to today's complex societal challenges. While ample empirical evidence has suggested its benefits within the academia such as better creativity and higher scientific impact and visibility, its societal benefits -- a key argument originally used for promoting IDR -- remain relatively unexplored. Here, we study one aspect of societal benefits, that is contributing to the development of patented technologies, and examine how IDR papers are referenced as "prior art" by patents over time. We draw on a large sample of biomedical papers published in 23 years and measure the degree of interdisciplinarity of a paper using three popular indicators, namely variety, balance, and disparity. We find that papers that cites more fields (variety) and whose distributions over those cited fields are more even (balance) are more likely to receive patent citations, but both effects can be offset if papers draw upon more distant fields (disparity). These associations are consistent across different citation-window lengths. We further find that conditional on receiving patent citations, the intensity of their technological impact, as measured as both raw and quality-adjusted number of citing patents, increases with balance and disparity. Our work may have policy implications for interdisciplinary research and scientific and technological impact.
Age of Information in Ultra-Dense IoT Systems: Performance and Mean-Field Game Analysis
Venue: arXiv cs.IT 2023
Year: 2023

In this paper, a dense Internet of Things (IoT) monitoring system is considered in which a large number of IoT devices contend for channel access so as to transmit timely status updates to the corresponding receivers using a carrier sense multiple access (CSMA) scheme. Under two packet management schemes with and without preemption in service, the closed-form expressions of the average age of information (AoI) and the average peak AoI of each device is characterized. It is shown that the scheme with preemption in service always leads to a smaller average AoI and a smaller average peak AoI, compared to the scheme without preemption in service. Then, a distributed noncooperative medium access control game is formulated in which each device optimizes its waiting rate so as to minimize its average AoI or average peak AoI under an average energy cost constraint on channel sensing and packet transmitting. To overcome the challenges of solving this game for an ultra-dense IoT, a mean-field game (MFG) approach is proposed to study the asymptotic performance of each device for the system in the large population regime. The accuracy of the MFG is analyzed, and the existence, uniqueness, and convergence of the mean-field equilibrium (MFE) are investigated. Simulation results show that the proposed MFG is accurate even for a small number of devices; and the proposed CSMA-type scheme under the MFG analysis outperforms three baseline schemes with fixed and dynamic waiting rates. Moreover, it is observed that the average AoI and the average peak AoI under the MFE do not necessarily decrease with the arrival rate.
Quality Inference in Federated Learning with Secure Aggregation
Venue: arXiv cs.LG 2023
Year: 2023

Federated learning algorithms are developed both for efficiency reasons and to ensure the privacy and confidentiality of personal and business data, respectively. Despite no data being shared explicitly, recent studies showed that the mechanism could still leak sensitive information. Hence, secure aggregation is utilized in many real-world scenarios to prevent attribution to specific participants. In this paper, we focus on the quality of individual training datasets and show that such quality information could be inferred and attributed to specific participants even when secure aggregation is applied. Specifically, through a series of image recognition experiments, we infer the relative quality ordering of participants. Moreover, we apply the inferred quality information to detect misbehaviours, to stabilize training performance, and to measure the individual contributions of participants.
A Survey of Privacy Attacks in Machine Learning
Venue: arXiv cs.CR 2023
Year: 2023

As machine learning becomes more widely used, the need to study its implications in security and privacy becomes more urgent. Although the body of work in privacy has been steadily growing over the past few years, research on the privacy aspects of machine learning has received less focus than the security aspects. Our contribution in this research is an analysis of more than 40 papers related to privacy attacks against machine learning that have been published during the past seven years. We propose an attack taxonomy, together with a threat model that allows the categorization of different attacks based on the adversarial knowledge, and the assets under attack. An initial exploration of the causes of privacy leaks is presented, as well as a detailed analysis of the different attacks. Finally, we present an overview of the most commonly proposed defenses and a discussion of the open problems and future directions identified during our analysis.
Denial-of-Service Vulnerability of Hash-based Transaction Sharding: Attack and Countermeasure
Venue: arXiv cs.CR 2023
Year: 2023

Since 2016, sharding has become an auspicious solution to tackle the scalability issue in legacy blockchain systems. Despite its potential to strongly boost the blockchain throughput, sharding comes with its own security issues. To ease the process of deciding which shard to place transactions, existing sharding protocols use a hash-based transaction sharding in which the hash value of a transaction determines its output shard. Unfortunately, we show that this mechanism opens up a loophole that could be exploited to conduct a single-shard flooding attack, a type of Denial-of-Service (DoS) attack, to overwhelm a single shard that ends up reducing the performance of the system as a whole. To counter the single-shard flooding attack, we propose a countermeasure that essentially eliminates the loophole by rejecting the use of hash-based transaction sharding. The countermeasure leverages the Trusted Execution Environment (TEE) to let blockchain's validators securely execute a transaction sharding algorithm with a negligible overhead. We provide a formal specification for the countermeasure and analyze its security properties in the Universal Composability (UC) framework. Finally, a proof-of-concept is developed to demonstrate the feasibility and practicality of our solution.
Exploiting No-Regret Algorithms in System Design
Venue: arXiv cs.GT 2023
Year: 2023

We investigate a repeated two-player zero-sum game setting where the column player is also a designer of the system, and has full control on the design of the payoff matrix. In addition, the row player uses a no-regret algorithm to efficiently learn how to adapt their strategy to the column player's behaviour over time in order to achieve good total payoff. The goal of the column player is to guide her opponent to pick a mixed strategy which is favourable for the system designer. Therefore, she needs to: (i) design an appropriate payoff matrix $A$ whose unique minimax solution contains the desired mixed strategy of the row player; and (ii) strategically interact with the row player during a sequence of plays in order to guide her opponent to converge to that desired behaviour. To design such a payoff matrix, we propose a novel solution that provably has a unique minimax solution with the desired behaviour. We also investigate a relaxation of this problem where uniqueness is not required, but all the minimax solutions have the same mixed strategy for the row player. Finally, we propose a new game playing algorithm for the system designer and prove that it can guide the row player, who may play a \emph{stable} no-regret algorithm, to converge to a minimax solution.
Segmentation-Driven Feature-Preserving Mesh Denoising
Venue: arXiv cs.GR 2023
Year: 2023

Feature-preserving mesh denoising has received noticeable attention in visual media, with the aim of recovering high-fidelity, clean mesh shapes from the ones that are contaminated by noise. Existing denoising methods often design smaller weights for anisotropic surfaces and larger weights for isotropic surfaces in order to preserve sharp features, such as edges or corners, on the mesh shapes. However, they often disregard the fact that such small weights on anisotropic surfaces still pose negative impacts on the denoising outcomes and detail preservation results on the shapes. In this paper, we propose a novel segmentation-driven mesh denoising method which performs region-wise denoising, and thus avoids the disturbance of anisotropic neighbour faces for better feature preservation results. Also, our backbone can be easily embedded into commonly-used mesh denoising frameworks. Extensive experiments have demonstrated that our method can enhance the denoising results on a wide range of synthetic and real mesh models, both quantitatively and visually.