Reviewerly/exHarmony huggingface dataset:
Information Bottleneck Revisited: Posterior Probability Perspective with Optimal Transport

Information bottleneck (IB) is a paradigm to extract information in one target random variable from another relevant random variable, which has aroused great interest due to its potential to explain deep neural networks in terms of information compression and prediction. Despite its great importance, finding the optimal bottleneck variable involves a difficult nonconvex optimization problem due to the nonconvexity of mutual information constraint. The Blahut-Arimoto algorithm and its variants provide an approach by considering its Lagrangian with fixed Lagrange multiplier. However, only the strictly concave IB curve can be fully obtained by the BA algorithm, which strongly limits its application in machine learning and related fields, as strict concavity cannot be guaranteed in those problems. To overcome the above difficulty, we derive an entropy regularized optimal transport (OT) model for IB problem from a posterior probability perspective. Correspondingly, we use the alternating optimization procedure and generalize the Sinkhorn algorithm to solve the above OT model. The effectiveness and efficiency of our approach are demonstrated via numerical experiments.
Secure Aggregation for Clustered Federated Learning

Clustered federated learning is a popular paradigm to tackle data heterogeneity in federated learning, by training personalized models for groups of users with similar data distributions. A critical challenge is to protect the privacy of individual user updates, as the latter can reveal extensive information about sensitive local datasets. To do so, a recent promising approach is information-theoretic secure aggregation, where parties learn the aggregate (sum) of user updates, but no further information is revealed about the individual updates. In this work, we present the first secure aggregation frameworks in the context of clustered federated learning, to learn the aggregate of user updates for any clustering of users, but without learning any information about the local updates or cluster identities. Our frameworks can achieve linear communication complexity under formal information-theoretic privacy guarantees, while providing key trade-offs between communication and computation complexity, adversary tolerance, and resilience to user dropouts.
Differentially Private Online Federated Learning With Personalization and Fairness

State-of-the-art federated learning algorithms are faced with many critical challenges that limit their applications in many domains. The vanilla federated learning architecture contains a central server that coordinates the training process. However, the central server is prone to congestion and a single point of failure as the number of clients grows exponentially. More so, the state-of-the-art federated learning algorithms are designed for fixed data distributions. Yet in many real-world scenarios, such as in real-time traffic monitoring systems, the data distribution of each client is time-varying. Hence, such algorithms do not perform well in these scenarios. Another challenge is that a single trained global model in federated learning may not perform optimally on the test dataset of any client since each client may have a specific task it aims to achieve despite task relatedness among clients. A more serious challenge caused by high data heterogeneity among the clients’ dataset is the unfairness of the trained model towards a minority group. Minority groups are classified by race, gender, social status, etc. This unfairness occurs when the trained global model discriminates against minority groups even when a local model is bias-free. Therefore, this paper addresses these challenges by proposing the Differentially private Personalized Online Federated Learning (DIPOFEL) and Differentially private Personalized Online Federated Learning with FAIRness (DIPOFLAIR) Algorithms for a peer-to-peer federated learning setting reinforced with local differential privacy. DIPOFEL Algorithm employs the online learning technique to handle time-varying data distribution, the meta-learning technique to personalize the trained model to each client’s specific task, and local differential privacy to prevent reconstruction attacks on the global model. DIPOFLAIR Algorithm is based on a constrained optimization problem that is solved using a Lagrangian approach with DIPOFEL Algorithm as a subroutine. The simulation result shows that the regret bounds of the proposed algorithms are better than the state-of-the-art algorithms.
Compression-Informed Coded Computing

Large-scale computations are ubiquitous and demand exorbitant resources, with matrix multiplication being a prominent example. Multiplying high-dimensional matrices is cumbersome for an individual server but is frequently needed in many applications. To alleviate the computational cost, one can take a low-rank approximation of the matrix product and distribute it over multiple workers. However, the tail latency of such distributed computations is degraded by straggling workers. One solution is to query extra workers with coded inputs to replace the outputs of straggling workers; this technique is called "coded computing." Nearly all existing coded computing schemes apply to multiplying any matrices. Instead, we propose a new framework to design coded computing schemes to take advantage of the structure induced by compression, which we call compression-informed coded computing. We then showcase the benefits of the framework in two steps. First, we illustrate how sketching can lead to linear dependencies in the matrices multiplied by the workers. Second, we apply locality-based coded computing to leverage these linear dependencies to make do with fewer workers compared to coded computing schemes that ignore the structure of the matrices being multiplied.
Secure Gradient Aggregation for Wireless Multi-Server Federated Learning

In this paper, we investigate the secure gradient aggregation problem for K-server wireless Federated learning (FL) systems. We propose a coded aggregation scheme such that any set of up to T colluding servers cannot infer any information about the local updates, including the aggregation value. In our scheme, each user encodes its local update to K confidential messages using Lagrange Coding, and then sends one confidential message to each server using an artificial noise alignment approach. In the downlink, each server delivers the summation of the confidential messages, by which every user can recover the aggregation of local updates. For the proposed scheme, we characterize the uplink and downlink communication latency, and show that the communication latency monotonically decreases with the total number of servers K while increasing with the number of colluding servers T.
Principled OOD Detection via Multiple Testing

We study the problem of Out-of-Distribution (OOD) detection, that is, detecting whether a Machine Learning (ML) model's output can be trusted at inference time. While a number of tests for OOD detection have been proposed in prior work, a formal framework for studying this problem is lacking. We propose a definition for the notion of OOD that includes both the input distribution and the ML model, which provides insights for the construction of powerful tests for OOD detection. We also propose a multiple hypothesis testing inspired procedure to systematically combine any number of different statistics from the ML model using conformal p-values. We further provide strong guarantees on the probability of incorrectly classifying an in-distribution sample as OOD. In our experiments, we find that threshold-based tests proposed in prior work perform well in specific settings, but not uniformly well across different OOD instances. In contrast, our proposed method that combines multiple statistics performs uniformly well across different datasets and neural networks.
Computation of Rate-Distortion-Perception Functions With Wasserstein Barycenter

The nascent field of Rate-Distortion-Perception (RDP) theory is seeing a surge of research interest due to the application of machine learning techniques in the area of lossy compression. The information RDP function characterizes the three-way trade-off between description rate, average distortion, and perceptual quality measured by discrepancy between probability distributions. However, computing RDP functions has been a challenge due to the introduction of the perceptual constraint, and existing research often resorts to data-driven methods. In this paper, we show that the information RDP function can be transformed into a Wasserstein Barycenter problem. The non-strictly convexity brought by the perceptual constraint can be regularized by an entropy regularization term. We prove that the entropy regularized model converges to the original problem. Furthermore, we propose an alternating iteration method based on the Sinkhorn algorithm to numerically solve the regularized optimization problem. Experimental results demonstrate the efficiency and accuracy of the proposed algorithm.
Entropy Rate Estimation for Markov Chains with Continuous Distributions

Entropy rate estimation has a broad range of applications such as bioinformatics, feature clustering etc. Although there are many existing work on the estimation of entropy rate for Markov chains with discrete distributions, the understanding for the entropy rate estimation of Markov chains with continuous distributions is limited. In this paper, efficient methods for estimating the entropy rate for Markov chains with continuous distributions are proposed. Moreover, we derive bounds on the convergence rate of the proposed entropy rate estimators.
On the Relation between Quantum Data Hiding and Quantum Key Distribution

In 2004, Horodecki et al., showed that the amount of secret key that can be extracted from noisy bipartite quantum states may exceed the amount of distillable maximally entangled quantum bits. Their work was based on the intuition of quantum data hiding, but did not offer a quantitative relationship to this phenomenon.In this work, which is set in a one-way classical communication scenario, we define a rate of quantum data hiding and provide entropic bounds on it. We then relate the quantum data hiding rate to the gap between the rates at which secret key and entanglement can be extracted, improving on earlier work on this topic.
Compression with Unlabeled Graph Side Information

With the growth of big data in the past few decades, compression has become inseparable from data generation. The data generated daily across different platforms are correlated: friend networks on Facebook and Instagram, contact networks in subsequent days, and many more. This raises the question of compressing a dataset using another correlated dataset. For instance, can we compress the Facebook graph of friends when we know Instagram’s graph? This can be cast as the classical problem of source coding with side information, and the answer is known to be positive when the graphs are "labeled" and/or aligned, meaning we need to know the node corresponding to Jon Doe in both Facebook and Instagram graphs. The classical idea is to utilize joint typicality to decide whether two graphs are correlated or not. In practice, graphs are often not aligned and/or the labels are concealed to keep the identity of the users private. In these scenarios, classical ideas are no longer applicable as joint typicality highly depends on the ordering of sequences. In this work, we prove for the first time the existence of lossless graph compression schemes that utilize unlabeled side information and improve the compression rate. In order to do that, we design binning along with a novel testing criterion that relies on graph matching, the closely related quadratic assignment problem and its asymptotic properties.
Pointwise Sliced Mutual Information for Neural Network Explainability

When deploying deep learning models such as convolutional neural networks (CNNs) in safety-critical domains, it is important to understand the predictions made by these black-box models. While many different approaches have been taken to improve the interpretability of deep models, most methods lack theoretical properties, making it hard to justify the correctness of the explanations provided. In this paper, we take an information-theoretic approach to understand why CNNs make their predictions. Building upon sliced mutual information, we propose pointwise sliced mutual information (PSI) as a tool for measuring the amount of useful information that a feature has about the label for a single instance. We theoretically justify the use of PSI for explaining predictions made by CNNs through the connection with margin. We show that PSI works as an explainability tool in two ways: (i) Fiber-wise PSI constructs a saliency map that highlights regions in the image which are important in predicting the labels; (ii) Sample-wise PSI provides confidence scores for predictions on the labels.
Assisted Unsupervised Domain Adaptation

Unsupervised domain adaptation (UDA) is a popular machine learning technique that allows one to train models over diverse data collected from different domains. However, this technique requires the learner to collect a large number of properly labeled data samples, which can be costly and unrealistic in many applications. In this work, we propose a decentralized assisted learning framework for UDA. In this framework, a learner has only a limited number of labeled data samples collected from a certain source domain and aims to train a classifier for the target domain. To improve domain adaptation performance, it seeks assistance by interacting with an external service provider, who possesses many labeled data samples collected from a related source domain. We develop an assisted UDA algorithm that avoids data sharing and can significantly improve the learner’s domain adaptation performance within a few rounds of interaction. Experiments using deep neural networks on benchmark datasets demonstrate the effectiveness of this algorithm.
Differentially Private Coded Computing

Distributed computing has attracted significant recent attention for speeding up large-scale computations by disseminating computational jobs from a central master node across several worker nodes/servers. However, worker nodes are often untrusted and can also collude to gain unauthorized access to sensitive data. Hence, sharing sensitive data with them raises data privacy concerns. Coded computing has emerged as a promising framework for speeding up distributed computing and can be also adapted to address security and privacy concerns utilizing tools from secret sharing and multi-party computing. However, ensuring perfect information-theoretic privacy imposes a strict threshold on the maximum number of colluding workers the protocol can tolerate and, also, necessitates quantizing/mapping data to finite fields. Differential privacy is a widely accepted practical measure to capture the privacy leakage of the shared data. The mainstream approach is then to add perturbations to the data via randomized mechanisms. In this paper, we revisit coded computing, and especially when it is adapted to handle real-valued data, and analyze the privacy guarantees through the lens of differential privacy in terms of the (ϵ,δ)-differential privacy metric, for the first time in the literature. All the computations are done over the field of real/complex numbers and data privacy, in terms of differential privacy, is attained by adding noise terms in a certain structured way. In particular, the noise is added through the secret sharing mechanism (which can be, in principle, decoded and cancelled out at the master) as means of ensuring differential privacy. Furthermore, we propose a differentially private distributed matrix multiplication protocol for matrix multiplications that keeps the privacy of data in the worst adversarial case.
Side Encoding for MDS Array Codes

This paper considers the parity-check matrix of a maximum distance separable (MDS) array code as a superposition of two matrices, block diagonal matrix A and side matrix S. By matching their entries, the syndrome calculation of the matrix A can share computations with that of the side matrix S. Then, a low-complexity encoding, referred to as side encoding, is proposed to encode matched MDS array codes efficiently. Moreover, it can combine with the Reed-Muller transform-based (RMTB) Reed-Solomon encoding algorithm to reduce the encoding complexity further. The analysis indicates that for the MDS array code [1] with four parity nodes, the number of multiplications is reduced by 85.4% and 52.6% compared to the traditional encoding and only RMTB encoding, respectively.
Third-Party Private Set Intersection

Private set intersection (PSI) enables two parties, each holding a private set to compute their intersection without revealing other information in the process. We introduce a generalized extension of conventional PSI termed as third-party PSI, whereby the intersection output of the two parties is only known to an inputless third party. In this setting, the two parties who participate in the protocol have no knowledge of the intersection result or any information of the set content of the other party. In general, third-party PSI settings arise where there is a need for an external party to obtain the intersection outcome without leakage of additional information to any other party. This setting is motivated by an increasing importance in several real-world applications. We describe protocols which achieve this functionality with minimal communication overhead. To the best of knowledge, our work is the first of its kind to explore this variant of PSI.
Utility-Privacy Trade-Offs with Limited Leakage for Encoder

The utilization of database such as IoT has progressed. As pioneering work, Yamamoto (1983) assumed the source (database) which consists of public information and private information and found theoretical limits (first-order rate analysis) among the coding rate, privacy for the decoder and utility in two special cases. In this paper, we consider a more general case based on the work by Shinohara and Yagi (2022). Introducing a measure of privacy for the encoder, we investigate two problems as follows: The first problem is the first-order rate analysis among the coding rate, utility, privacy for the decoder, and privacy for the encoder in which utility is measured by the expected distortion or the excess-distortion probability. The second task is establishing the strong converse theorem for utility-privacy trade-offs in which utility is measured by the excess-distortion probability. These results may lead to a more refined analysis such as the second-order rate analysis.
On the Inevitability of the Rashomon Effect

The Rashomon effect in machine learning (ML) occurs when multiple distinct models achieve similar average loss on a given learning task. The set of all models with expected loss smaller than ϵ is called the Rashomon set. The characterization of this set for a given learning task allows searching for models that satisfy additional constraints (e.g., interpretability, fairness) without compromising accuracy. Though folklore treats the Rashomon set as the collection of all indistinguishable "good" models, there are no established theoretical guarantees that models in this set are statistically indistinguishable. We fill this gap by proposing a hypothesis test framework to choose the best-performing model between two elements in the Rashomon set and derive lower and upper bounds for its probability of error. Specifically, we prove that for any ϵ > 0 if the data set has less than $O\left( {{{[\varepsilon {\text{log}}\left( {\varepsilon /\left( {1 - \varepsilon } \right)} \right)]}^{ - 1}}} \right)$ instances, models in the Rashomon set are statistically indistinguishable and the Rashomon effect is inevitable. Additionally, our bounds can guide data scientists to choose an ϵ that generates a Rashomon set so that any two models in it are indistinguishable.
Communication-Efficient and Error-Free Gradecast with Optimal Resilience

Gradecast is a variant of the Byzantine broadcast problem introduced by Feldman and Micali in 1988. In Gradecast, n processors would like to agree on a value sent from a leader, as well as a grade in {0, 1, 2}, such that the following three requirements are satisfied: 1) Every non-faulty processor outputs the leader’s initial value and grade 2 if the leader is non-faulty; 2) For any two non-faulty processors, if their decided grades are greater than zero, then they output the same value; and 3) For any two non-faulty processors, the difference of their decided grades is less than 2. In this work, we present a new Gradecast protocol with a total communication complexity of O(nℓ + n <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> log n) bits, given t < n/3, where ℓ is the message size and t is the maximum number of faulty processors tolerated in n consensus processors. The proposed protocol is an error-free and deterministic Gradecast protocol that does not rely on the authentication techniques such as signatures and secret sharing. The proposed protocol is also information-theoretic secure, i.e., it satisfies the above three requirements even if the computation power of the adversary is unbounded.
Fast Encoding of Hermitian Codes Based on Lin-Chung-Han Fast Fourier Transform

In this paper, we present fast encoding algorithms for Hermitian codes based on the Lin-Chung-Han fast Fourier transform (LCH-FFT). For non-systematic encoding, we extend the LCH basis to the bivariate polynomial space and develop a two-dimensional FFT algorithm. For systematic encoding, we propose a modified partial FFT algorithm and present a procedure for computing the unknown intermediates. For a Hermitian code of length $n$, the computational complexity of the presented non-systematic and systematic encoding algorithms are both $O(n{\text{log}}n)$, improving upon the currently best-known encoding complexity $O\left( {n{\text{lo}}{{\text{g}}^2}n{\text{loglog}}n} \right)$.
On characterizing optimal Wasserstein GAN solutions for non-Gaussian data

The generative adversarial network (GAN) aims to approximate an unknown distribution via a parameterized neural network (NN). While GANs have been widely applied in reinforcement and semi-supervised learning as well as computer vision tasks, selecting their parameters often needs an exhaustive search and only a few selection methods can be proved to be theoretically optimal. One of the most promising GAN variants is the Wasserstein GAN (WGAN). Prior work on optimal parameters for WGAN is limited to the linear-quadratic-Gaussian (LQG) setting, where the NN is linear and the data is Gaussian. In this paper, we focus on the characterization of optimal WGAN parameters beyond the LQG setting. We derive closed-form optimal parameters for one-dimensional WGANs with non-linear sigmoid and ReLU activation functions. Extensions to high-dimensional WGANs are also discussed. Empirical studies show that our closed-form WGAN parameters have good convergence behavior with data under both Gaussian and Laplace distributions.