Reviewerly/exHarmony huggingface dataset:
Hyperparameter Learning of Bayesian Context Tree Models

In recent years, Bayesian counterparts of the context tree weighting method are studied for many tasks. All these tasks require a hyperparameter setting of the prior distribution for context tree models. Therefore, we provide a framework for statistically learning these hyperparameters from data. Specifically, we consider a hierarchical Bayesian model that assumes hyperprior distributions behind the hyperparameters and learn them using an empirical variational Bayesian (EVB) method. This is the first study to propose an EVB method on the Bayesian context trees. The derived algorithm has a suggestive form that consists of subroutines partially optimal to each local probabilistic model.
Private Aggregation in Wireless Federated Learning with Heterogeneous Clusters

Federated learning collaboratively trains a neural network on privately owned data held by several participating clients. The gradient descent algorithm, a well-known and popular iterative optimization procedure, is run to train the neural network. Every client uses its local data to compute partial gradients and sends it to the federator which aggregates the results. Privacy of the clients’ data is a major concern. In fact, observing the partial gradients can be enough to reveal the clients’ data. Private aggregation schemes have been investigated to tackle the privacy problem in federated learning where all the users are connected to each other and to the federator. In this paper, we consider a wireless system architecture where clients are only connected to the federator via base stations. We derive fundamental limits on the communication cost when information-theoretic privacy is required, and introduce and analyze a private aggregation scheme tailored for this setting.
Representation Transfer Learning via Multiple Pre-trained models for Linear Regression

In this paper, we consider the problem of learning a linear regression model on a data domain of interest (target) given few samples. To aid learning, we are provided with a set of pre-trained regression models that are trained on potentially different data domains (sources). Assuming a representation structure for the data generating linear models at the sources and the target domains, we propose a representation transfer based learning method for constructing the target model. The proposed scheme is comprised of two phases: (i) utilizing the different source representations to construct a representation that is adapted to the target data, and (ii) using the obtained model as an initialization to a fine-tuning procedure that re-trains the entire (over-parameterized) regression model on the target data. For each phase of the training method, we provide excess risk bounds for the learned model compared to the true data generating target model. The derived bounds show a gain in sample complexity for our proposed method compared to the baseline method of not leveraging source representations when achieving the same excess risk, therefore, theoretically demonstrating the effectiveness of transfer learning for linear regression.
The Asymptotics of Difference Systems of Sets for Synchronization and Phase Detection

We settle the problem of determining the asymptotic behavior of the parameters of optimal difference systems of sets, or DSSes for short, which were originally introduced for computationally efficient frame synchronization under the presence of additive noise. We prove that the lowest achievable redundancy of a DSS asymptotically attains Levenshtein's lower bound for any alphabet size and relative index, answering the question of Levenshtein posed in 1971. Our proof is probabilistic and gives a linear-time randomized algorithm for constructing asymptotically optimal DSSes with high probability for any alphabet size and information rate. This provides efficient self-synchronizing codes with strong noise resilience. We also point out an application of DSSes to phase detection.
Unconditionally Secure Access Control Encryption

Access control encryption (ACE) enforces, through a sanitizer as the mediator, that only legitimate sender-receiver pairs can communicate, without the sanitizer knowing the communication metadata, including its sender and recipient identity, the policy over them, and the underlying plaintext. Any illegitimate transmission is indistinguishable from pure noise. Existing works focused on computational security and require trapdoor functions and possibly other heavyweight primitives. We present the first ACE scheme with information-theoretic security (unconditionally against unbounded adversaries). Our novel randomization techniques over matrices realize sanitization (traditionally via homomorphism over a fixed randomness space) such that the secret message in the hidden message subspace remains intact if and only if there is no illegitimate transmission.
A Stochastic Rate-Distortion Approach to Supervised Learning Systems

Machine learning applications have exploded in recent years due to the availability of huge data sets as well as advances in computational and storage capabilities. Although successful methods have been proposed to reduce learning system complexity while maintaining required accuracy levels, theoretical understanding of the underlying trade-offs remains elusive. In this paper, the classical supervised learning problem is reformulated within a rate-distortion framework. It provides insights into crucial accuracy-complexity trade-offs, by considering the overall learning system as consisting of two components. The first is tasked with extracting (learning) from the source the minimal number of information bits necessary to ultimately achieve the prescribed output accuracy. The learned bits are then used to retrieve the desired output from the second component, an appropriately designed codebook. The premise here is that an optimal system is characterized by having to learn the minimum amount of information from the source, just sufficient to yield the system output at the desired precision, which implies efficiency in terms of system complexity, generalization and training data requirements. The design and training of such a reformulated system is detailed in this paper, and asymptotically optimal performance achieving the rate-distortion bound is established.
A Worst-Case Analysis of a Renormalisation Decoder for Kitaev’s Toric Code

Kitaev's toric code is arguably the most studied quantum code and is expected to be implemented in future generations of quantum computers. The renormalisation decoders introduced by Duclos-Cianci and Poulin exhibit one of the best trade-offs between efficiency and speed, but one question that was left open is how they handle worst-case or adversarial errors, i.e. what is the order of magnitude of the smallest weight of an error pattern that will be wrongly decoded. We initiate such a study involving a simple hard-decision and deterministic version of a renormalisation decoder. We exhibit an uncorrectable error pattern whose weight scales like d <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1/2</sup> and prove that the decoder corrects all error patterns of weight less than $\frac{5}{6}{d^{{{\log }_2}(6/5)}}$, where d is the minimum distance of the toric code.
Exploiting Feature Heterogeneity for Improved Generalization in Federated Multi-task Learning

In this work, we investigate a general federated multitask learning (FMTL) problem where each task may be performed at multiple clients, and each client may perform multiple tasks. Although the tasks share some common representation (i.e., feature-map) that can help to learn, the distribution of the features in the feature space may vary across different tasks at different clients, which poses a significant challenge to FMTL. While non-independent and identically distributed (non-IID) local datasets at different clients are often considered detrimental to model convergence in federated learning (FL), such statistical heterogeneity in feature space may be beneficial to the generalization performance. In this work, we establish the impact of statistical feature heterogeneity on generalization, through the lens of a multi-task linear regression model. In order to leverage the feature distribution heterogeneity, we propose a novel augmented dataset based approach, and prove that under certain conditions, FMTL on heterogeneous datasets can outperform the homogeneous counterpart in terms of the generalization performance. The theoretical analysis further leads to a simple client weighting method based on optimizing the excess risk upper bound. Experimental results demonstrate that the generalization performance can be improved on a real-world dataset with the proposed method.
Information Gain and Uniform Generalization Bounds for Neural Kernel Models

Neural tangent (NT) kernel models have attracted great attention lately, partly for explaining overparameterized neural networks in their limit. In previous work, generalization bounds for overparameterized neural networks were given in expectation. In this work, we prove uniform generalization bounds for NT kernel models by characterizing the information gain of the NT kernel. Our bounds capture the exact error rates based on the smoothness of the activation functions. Our results on information gain can be applied to other problems where overparameterized neural networks are used; e.g., certain reinforcement learning and bandit problems.
Maximal Length Constrained de Bruijn Sequences

The run length limited de Bruijn (RLLdB) sequences have been studied recently owing to their application in communication systems. In each RLLdB sequence, all length-k substrings are distinct and each substring is a run length limited sequence, where there are at most s consecutive symbol 0’s. In previous work, the maximal length RLLdB sequence was designed only for the case s = 1.In this work, we propose to study a more generalised problem of constrained de Bruijn sequence, called weight bounded run length limited de Bruijn (WRdB) sequence. In each WRdB sequence, all length-k substrings are distinct and each length-k substring satisfies both constraints: the number of consecutive symbol 0’s is at most s, and the number of symbol 1’s is at least l. We aim to design a WRdB sequence with maximal length in all cases.Firstly, we build a constrained de Bruijn graph that represents a WRdB sequence and use some properties of the graph to show an upper bound on the maximal length of the sequence. Next, using Lyndon words, we present a construction of a WRdB sequence. Finally, we compute the length of the constructed WRdB sequence and show that it has the maximal length.
Wireless Coded Distributed Learning with Gaussian-based Local Differential Privacy

Differentially private distributed machine learning protects privacy by injecting artificial noise to the computing results. To further improve energy efficiency, the natural noise in the wireless environment can be used to protect privacy. In this paper, we study the problem of coded distributed machine learning over Gaussian multiple-access wireless channels to achieve differential privacy by exploiting the natural noise. Firstly, we propose an aggregation scheme using differentially private Lagrange encoding in a wireless environment, where the local computing results are uploaded to the master through orthogonal channels. Then, we develop an achievable privacy protection level to illustrate the impact of transmit power and power allocation on privacy. Additionally, we establish a theoretical convergence upper bound of the proposed scheme, providing a clear understanding of the potential limitations and capabilities of the system. Finally, we demonstrate a trade-off between system resource settings, convergence, and privacy protection levels through experiments. Specifically, increasing the signal-to-noise ratio (SNR) and power allocated for gradient computation leads to a decrease in the privacy protection level of the system and an increase in training accuracy. Moreover, reducing the dataset partitions results in better training accuracy.
Time Series Analysis with Bayesian Context Trees: Classical Asymptotics and Finite-n Bounds

The Bayesian Context Trees (BCT) framework is a recently introduced, general collection of statistical and algorithmic tools for modelling, analysis and inference with discrete-valued time series. The foundation of this development is built in part on some well-known information-theoretic ideas and techniques, including Rissanen’s tree sources and Willems et al.’s context-tree weighting (CTW) algorithm. This paper presents a collection of theoretical results that provide mathematical justifications and further insight into the BCT modelling framework and the associated practical tools, including the CTW algorithm. It is shown that the BCT prior predictive likelihood (the probability of a time series of observations averaged over all models and parameters) is both pointwise and minimax optimal, in agreement with the MDL principle and the BIC criterion. The posterior distribution is shown to be asymptotically consistent with probability one (over both models and parameters), and asymptotically Gaussian (over the parameters). And the posterior predictive distribution is also shown to be asymptotically consistent with probability one.
CSS codes and QSCs from Whiteman’s Generalized Cyclotomy of order four

In this paper, Calderbank-Shor-Steane (CSS) quantum codes and Quantum synchronizable codes (QSCs) are obtained from cyclic codes, which are constructed using Whiteman’s generalized cyclotomy of order four. These CSS and QSCs have good error-correcting capability against phase and bit errors because many of the cyclic codes used to design them are optimal or nearly optimal. Moreover, QSCs obtained in this article achieve maximum tolerance against misalignment.
Two new algorithms for error support recovery of low rank parity check codes

Due to their weak algebraic structure, low rank parity check (LRPC) codes have been employed in several post-quantum cryptographic schemes. In this paper we propose new improved decoding algorithms for ${\left[ {n,{\text{ }}k} \right]_{{q^m}}}$ LRPC codes of dual rank weight d. The proposed algorithms can efficiently decode LRPC codes with the parameters satisfying n − k = rd − c, where r is the dimension of the error support and c ≤ d − 2. They outperform the original decoding algorithm of LRPC codes when d > 2 and allow for decoding LRPC codes with a higher code rate and smaller values m.
Transaction Confirmation in Coded Blockchain

As blockchains continue to seek to scale to a larger number of nodes, the communication complexity of protocols has become a significant priority as the network can quickly become overburdened. Several schemes have attempted to address this, one of which uses coded computation to lighten the load. Here we seek to address one issue with all such coded blockchain schemes known to the authors: transaction confirmation. In a coded blockchain, only the leader has access to the uncoded block, while the nodes receive encoded data that makes it effectively impossible for them to identify which transactions were included in the block. As a result, a Byzantine leader might choose not to notify a sender or receiver of a transaction that the transaction went into the block, and even with an honest leader, they would not be able to produce a proof of a transaction’s inclusion. To address this, we have constructed a protocol to send the nodes enough information so that a client sending or receiving a transaction is guaranteed to not only be notified but also to receive a proof of that transaction’s inclusion in the block. Crucially, we do this without substantially increasing the bit complexity of the original coded blockchain protocol.
A Search-to-Decision Reduction for the Permutation Code Equivalence Problem

In this paper, we describe an efficient search-to-decision reduction for the permutation code equivalence problem. Given two linear codes ${\mathcal{C}_1}$, ${\mathcal{C}_2}$ of length n and dimension k over ${\mathbb{F}_q}$, we describe an algorithm that finds $\pi \in {\mathcal{S}_n}$ such that $\pi \left( {{\mathcal{C}_1}} \right) = {\mathcal{C}_2}$ by using a polynomial number of queries to an oracle that decides whether two codes are permutation-equivalent.
The Re-encoding Transform in Algebraic List Decoding of Algebraic Geometric Codes

This paper proposes the re-encoding transformed (ReT) based list decoding using the module basis reduction (BR) interpolation for algebraic geometric (AG) codes on C <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">ab</inf> curves. The two ReT approaches are introduced to facilitate the BR interpolation. One is realized by the bivariate Lagrange polynomial. The other is conducted by the ReT of Reed-Solomon (RS) codes based on the mathematical structure of AG codes. The ReT based BR interpolation (ReT-BR) algorithm for decoding the AG codes is further introduced. Finally, complexity of the proposed algorithm is analyzed and validated by the simulation results, demonstrating its complexity advantage over the non-ReT counterpart.
MacWilliams-Like Identities for Certain Vectorial Bent Functions

It is well-known that MacWilliams identities play a significant role in coding theory. In [5]-[7], MacWilliams-like identities for p-ary bent functions $f:\mathbb{F}_p^n \to {\mathbb{F}_p}$ were given, where p is a prime. The aim of this paper is to investigate MacWilliams-like identities for vectorial bent functions. We give MacWillaims-like identities for certain vectorial bent functions $F:\mathbb{F}_q^t \to {\mathbb{F}_q}$, where q is a power of a prime p. We illustrate that when q = p, the MacWilliams-like identities for weakly regular p-ary bent functions can be obtained by our results. Based on the obtained MacWilliams-like identities, we give some nonexistence results on vectorial bent functions.
Rate-limited Quantum-to-Classical Optimal Transport: A Lossy Source Coding Perspective

We consider the rate-limited quantum-to-classical optimal transport in terms of output-constrained rate-distortion coding for discrete quantum measurement systems with limited classical common randomness. The main coding theorem provides the achievable rate region of a lossy measurement source coding for an exact construction of the destination distribution (or the equivalent quantum state) while maintaining a threshold of distortion from the source state according to a generally defined distortion observable. The constraint on the output space fixes the output distribution to an i.i.d. predefined probability mass function. Therefore, this problem can also be viewed as information-constrained optimal transport which finds the optimal cost of transporting the source quantum state to the destination state via an entanglement-breaking channel with limited communication rate and common randomness. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup>
Hierarchical Coded Gradient Aggregation Based on Layered MDS Codes

The growing privacy concerns and the communication costs associated with transmitting raw data have resulted in techniques like federated learning, where the machine learning models are trained at the edge nodes, and the parameter updates are shared with a central server. Because communications from the edge nodes are often unreliable, a hierarchical setup involving intermediate helper nodes is considered. The communication links between the edges and the helper nodes are error-prone and are modeled as straggling/failing links. To overcome the issue of link failures, coding techniques are proposed. The edge nodes communicate encoded versions of the model updates to the helper nodes, which pass them on to the master after suitable aggregation. The primary work in this area uses repetition codes and Maximum Distance Separable (MDS) codes at the edge nodes to arrive at the Aligned Repetition Coding (ARC) and Aligned MDS Coding (AMC) schemes, respectively. We propose using vector codes, specifically a family of layered MDS codes parameterized by a variable ν, at the edge nodes. For the proposed family of codes, suitable aggregation strategies at the helper nodes are also developed. At the extreme values of ν, our scheme matches the communication costs incurred by the ARC and AMC schemes, resulting in a graceful transition between these schemes.
Improved Field Size Bounds for Higher Order MDS Codes

Higher order MDS codes are an interesting generalization of MDS codes recently introduced by Brakensiek, Gopi and Makam (IEEE Trans. Inf. Theory 2022). In later works, they were shown to be intimately connected to optimally list-decodable codes and maximally recoverable tensor codes. Therefore (explicit) constructions of higher order MDS codes over small fields is an important open problem. Higher order MDS codes are denoted by MDS(ℓ) where ℓ denotes the order of generality, MDS(2) codes are equivalent to the usual MDS codes. The best prior lower bound on the field size of an (n, k)-MDS(ℓ) codes is Ω <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">ℓ</inf> (n <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">ℓ−1</sup> ), whereas the best known (non-explicit) upper bound is O <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">ℓ</inf> (n <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">k(ℓ−1)</sup> ) which is exponential in the dimension.In this work, we nearly close this exponential gap between upper and lower bounds. We show that an (n, k)-MDS(3) codes requires a field of size Ω <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">k</inf> (n <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">k−1</sup> ), which is close to the known upper bound. Using the connection between higher order MDS codes and optimally list-decodable codes, we show that even for a list size of 2, a code which meets the optimal list-decoding Singleton bound requires exponential field size; this resolves an open question from Shangguan and Tamo (STOC 2020).We also give explicit constructions of (n, k)-MDS(ℓ) code over fields of size ${n^{{{(\ell k)}^{O(\ell k)}}}}$. The smallest non-trivial case where we still do not have optimal constructions is (n, 3)-MDS(3). In this case, the known lower bound on the field size is Ω(n <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> ) and the best known upper bounds are O(n <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">5</sup> ) for a non-explicit construction and O(n <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">32</sup> ) for an explicit construction. In this paper, we give an explicit construction over fields of size O(n <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3</sup> ) which comes very close to being optimal.