Miracle-dz/UniFAR-FTUs huggingface dataset:
Can we Generalize and Distribute Private Representation Learning?

We study the problem of learning representations that are private yet informative, i.e., provide information about intended “ally” targets while hiding sensitive “adversary” attributes. We propose Exclusion-Inclusion Generative Adversarial Network (EIGAN), a generalized private representation learning (PRL) architecture that accounts for multiple ally and adversary attributes unlike existing PRL solutions. While centrally-aggregated dataset is a prerequisite for most PRL techniques, data in real-world is often siloed across multiple distributed nodes un-willing to share the raw data because of privacy concerns. We address this practical constraint by developing D-EIGAN, the ﬁrst distributed PRL method that learns representations at each node without transmitting the source data. We theoretically analyze the behavior of adversaries under the optimal EIGAN and D-EIGAN encoders and the impact of dependencies among ally and adversary tasks on the optimization objective. Our experiments on various datasets demonstrate the advantages of EIGAN in terms of performance, robustness, and scalability. In partic-ular, EIGAN outperforms the previous state-of-the-art by a signiﬁcant accuracy margin (47% improvement), and D-EIGAN’s performance is consistently on par with EIGAN under diﬀerent network settings. We observe that the performance of the ally and adversary remains reasonably constant (and to EIGAN) we increase the number of nodes under D-EIGAN. the two experiments, we can
Restarted Nonconvex Accelerated Gradient Descent: No More Polylogarithmic Factor in the O ( (cid:15) − 7 / 4 ) Complexity

Nonconvex optimization with great demand of fast solvers is ubiquitous in modern machine learning. This paper studies two simple accelerated gradient methods, restarted accelerated gradient descent (AGD) and restarted heavy ball (HB) method, for general nonconvex problems under the gradient Lipschitz and Hessian Lipschitz conditions. We establish that the two algorithms ﬁnd an (cid:15) -approximate ﬁrst-order stationary point in O ( (cid:15) − 7 / 4 ) gradient computations with simple proofs. Our complexity does not hide any polylogarithmic factors, and thus it improves over the state-of-the-art one by the O (log 1 (cid:15) ) factor. Our algorithms are simple in the sense that they only consist of Nesterov’s classical AGD or Polyak’s HB iterations, as well as a restart mechanism. They do not need the negative curvature exploitation or the minimization of regularized surrogate functions. Our simple proofs only use very elementary analysis, and in contrast with existing analysis, we do not invoke the analysis of the strongly convex AGD or HB.
Born-Infeld (BI) for AI: Energy-Conserving Descent (ECD) for Optimization

We introduce a novel framework for optimization based on energy-conserving Hamiltonian dynamics in a strongly mixing (chaotic) regime and establish its key properties analytically and numerically. The prototype is a discretization of Born-Infeld dynamics, with a squared relativistic speed limit depending on the objective function. This class of frictionless, energy-conserving optimizers proceeds unobstructed until slowing naturally near the minimal loss, which dominates the phase space volume of the system. Building from studies of chaotic systems such as dynamical billiards, we formulate a specific algorithm with good performance on machine learning and PDE-solving tasks, including generalization. It cannot stop at a high local minimum, an advantage in non-convex loss functions, and proceeds faster than GD+momentum in shallow valleys.
The Implicit Bias of Benign Overfitting

The phenomenon of benign overfitting, where a predictor perfectly fits noisy training data while attaining near-optimal expected loss, has received much attention in recent years, but still remains not fully understood beyond well-specified linear regression setups. In this paper, we provide several new results on when one can or cannot expect benign overfitting to occur, for both regression and classification tasks. We consider a prototypical and rather generic data model for benign overfitting of linear predictors, where an arbitrary input distribution of some fixed dimension $k$ is concatenated with a high-dimensional distribution. For linear regression which is not necessarily well-specified, we show that the minimum-norm interpolating predictor (that standard training methods converge to) is biased towards an inconsistent solution in general, hence benign overfitting will generally not occur. Moreover, we show how this can be extended beyond standard linear regression, by an argument proving how the existence of benign overfitting on some regression problems precludes its existence on other regression problems. We then turn to classification problems, and show that the situation there is much more favorable. Specifically, we prove that the max-margin predictor (to which standard training methods are known to converge in direction) is asymptotically biased towards minimizing a weighted \emph{squared hinge loss}. This allows us to reduce the question of benign overfitting in classification to the simpler question of whether this loss is a good surrogate for the misclassification error, and use it to show benign overfitting in some new settings.
TCFHs, hidden symmetries and M-theory backgrounds

We present the twisted covariant form hierarchy (TCFH) of 11-dimensional supergravity and so demonstrate that the form bilinears of supersymmetric solutions satisfy a generalisation of the conformal Killing-Yano equation with resepct to the TCFH connection. We also compute the Killing-Stäckel, KY and closed conformal Killing-Yano tensors of all spherically symmetric M-branes that include the M2-brane, M5-brane, KK-monopole and pp-wave and demonstrate that their geodesic flows are completely integrable by giving all independent conserved charges in involution. We then find that all form bilinears of pp-wave and KK-monopole solutions generate (hidden) symmetries for spinning particle probes propagating on these backgrounds. Moreover, there are Killing spinors such that some of the 1-, 2- and 3-form bilinears of the M2-brane solution generate symmetries for spinning particle probes. We also explore the question on whether the form bilinears are sufficient to prove the integrability of particle probe dynamics on 11-dimensional supersymmetric backgrounds.
Towards Agnostic Feature-based Dynamic Pricing: Linear Policies vs Linear Valuation with Unknown Noise

In feature-based dynamic pricing, a seller sets appropriate prices for a sequence of products (described by feature vectors) on the fly by learning from the binary outcomes of previous sales sessions ("Sold"if valuation $\geq$ price, and"Not Sold"otherwise). Existing works either assume noiseless linear valuation or precisely-known noise distribution, which limits the applicability of those algorithms in practice when these assumptions are hard to verify. In this work, we study two more agnostic models: (a) a"linear policy"problem where we aim at competing with the best linear pricing policy while making no assumptions on the data, and (b) a"linear noisy valuation"problem where the random valuation is linear plus an unknown and assumption-free noise. For the former model, we show a $\tilde{\Theta}(d^{\frac13}T^{\frac23})$ minimax regret up to logarithmic factors. For the latter model, we present an algorithm that achieves an $\tilde{O}(T^{\frac34})$ regret, and improve the best-known lower bound from $\Omega(T^{\frac35})$ to $\tilde{\Omega}(T^{\frac23})$. These results demonstrate that no-regret learning is possible for feature-based dynamic pricing under weak assumptions, but also reveal a disappointing fact that the seemingly richer pricing feedback is not significantly more useful than the bandit-feedback in regret reduction.
Implicit Regularization in Hierarchical Tensor Factorization and Deep Convolutional Neural Networks

In the pursuit of explaining implicit regularization in deep learning, prominent focus was given to matrix and tensor factorizations, which correspond to simplified neural networks. It was shown that these models exhibit an implicit tendency towards low matrix and tensor ranks, respectively. Drawing closer to practical deep learning, the current paper theoretically analyzes the implicit regularization in hierarchical tensor factorization, a model equivalent to certain deep convolutional neural networks. Through a dynamical systems lens, we overcome challenges associated with hierarchy, and establish implicit regularization towards low hierarchical tensor rank. This translates to an implicit regularization towards locality for the associated convolutional networks. Inspired by our theory, we design explicit regularization discouraging locality, and demonstrate its ability to improve the performance of modern convolutional networks on non-local tasks, in defiance of conventional wisdom by which architectural changes are needed. Our work highlights the potential of enhancing neural networks via theoretical analysis of their implicit regularization.
Properties of the Non-Autonomous Lattice Sine-Gordon Equation: Consistency around a Broken Cube Property

A bstract . The lattice sine-Gordon equation is an integrable partial di ﬀ erence equation on Z 2 , which approaches the sine-Gordon equation in a continuum limit. In this paper, we show that the non-autonomous lattice sine-Gordon equation has the consistency around a broken cube property as well as its autonomous version. Moreover, we construct two new Lax pairs of the non-autonomous case by using the consistency property.
Efficient Hierarchical Storage Management Framework Empowered by Reinforcement Learning

With the rapid development of big data and cloud computing, data management has become increasingly challenging. Over the years, a number of frameworks for data management and storage with various characteristics and features have become available. Most of these are highly efficient, but ultimately create data silos. It becomes difficult to move and work coherently with data as new requirements emerge as no single framework can efficiently fulfill the data management needs of diverse applications. A possible solution is to design smart and efficient hierarchical (multi-tier) storage solutions. A hierarchical storage system (HSS) is a meta solution that consists of different storage frameworks organized as a jointly constructed large storage pool. It brings a number of benefits including better utilization of the storage, cost-efficiency, and use of different features provided by the underlying storage frameworks. In order to maximize the gains of hierarchical storage solutions, it is important that they include intelligent and autonomous mechanisms for data management grounded in the features of the different underlying frameworks. These decisions should be made according to the characteristics of the dataset, tier status, and access patterns. These are highly dynamic parameters and defining a policy based on the mentioned parameters is a non-trivial task. This paper presents an open-source hierarchical storage framework with a dynamic migration policy based on reinforcement learning (RL). We present a mathematical model, a software architecture, and an implementation based on both simulations and a live cloud-based environment. We compare the proposed RL-based strategy to a baseline of three rule-based policies, showing that the RL-based policy achieves significantly higher efficiency and optimal data distribution in different scenarios compared to the dynamic rule-based policies.
Phase Retrieval for Radar Waveform Design

The ability of a radar to discriminate in both range and Doppler velocity is completely characterized by the ambiguity function (AF) of its transmit waveform. Mathematically, it is obtained by correlating the waveform with its Doppler-shifted and delayed replicas. We consider the inverse problem of designing a radar transmit waveform that satisfies the specified AF magnitude. This process can be viewed as a signal reconstruction with some variation of phase retrieval methods. We provide a trust-region algorithm that minimizes a smoothed non-convex least-squares objective function to iteratively recover the underlying signal-of-interest for either time- or band-limited support. The method first approximates the signal using an iterative spectral algorithm and then refines the attained initialization based upon a sequence of gradient iterations. Our theoretical analysis shows that unique signal reconstruction is possible using signal samples no more than thrice the number of signal frequencies or time samples. Numerical experiments demonstrate that our method recovers both time- and band-limited signals from even sparsely and randomly sampled AFs with mean-square-error of $1\times 10^{-6}$ and $9\times 10^{-2}$ for the full noiseless samples and sparse noisy samples, respectively.
Fairness Implications of Encoding Protected Categorical Attributes

Past research has demonstrated that the explicit use of protected attributes in machine learning can improve both performance and fairness. Many machine learning algorithms, however, cannot directly process categorical attributes, such as country of birth or ethnicity. Because protected attributes frequently are categorical, they must be encoded as features that can be input to a chosen machine learning algorithm, e.g. support vector machines, gradient boosting decision trees or linear models. Thereby, encoding methods influence how and what the machine learning algorithm will learn, affecting model performance and fairness. This work compares the accuracy and fairness implications of the two most well-known encoding methods: one-hot encoding and target encoding. We distinguish between two types of induced bias that may arise from these encoding methods and may lead to unfair models. The first type, irreducible bias, is due to direct group category discrimination and the second type, reducible bias, is due to the large variance in statistically underrepresented groups. We investigate the interaction between categorical encodings and target encoding regularization methods that reduce unfairness. Furthermore, we consider the problem of intersectional unfairness that may arise when machine learning best practices improve performance measures by encoding several categorical attributes into a high-cardinality feature.
Simplicial Convolutional Filters

We study linear filters for processing signals supported on abstract topological spaces modeled as simplicial complexes, which may be interpreted as generalizations of graphs that account for nodes, edges, triangular faces, etc. To process such signals, we develop simplicial convolutional filters defined as matrix polynomials of the lower and upper Hodge Laplacians. First, we study the properties of these filters and show that they are linear and shift-invariant, as well as permutation and orientation equivariant. These filters can also be implemented in a distributed fashion with a low computational complexity, as they involve only (multiple rounds of) simplicial shifting between upper and lower adjacent simplices. Second, focusing on edge-flows, we study the frequency responses of these filters and examine how we can use the Hodge-decomposition to delineate gradient, curl and harmonic frequencies. We discuss how these frequencies correspond to the lower- and the upper-adjacent couplings and the kernel of the Hodge Laplacian, respectively, and can be tuned independently by our filter designs. Third, we study different procedures for designing simplicial convolutional filters and discuss their relative advantages. Finally, we corroborate our simplicial filters in several applications: to extract different frequency components of a simplicial signal, to denoise edge flows, and to analyze financial markets and traffic networks.
On the RTL Implementation of FINN Matrix Vector Compute Unit

FPGA-based accelerators are becoming more popular for deep neural network due to the ability to scale performance with increasing degree of specialization with dataflow architectures or custom data types. To reduce the barrier for software engineers and data scientists to adopt FPGAs, C++- and OpenCL-based design entries with high-level synthesis (HLS) have been introduced. They provide higher abstraction compared to register-transfer level (RTL)-based design. HLS offers faster development time, better maintainability and more flexibility in code exploration, when evaluating options for multi-dimension tensors, convolutional layers or parallelism. Thus, HLS has been adopted by DNN accelerator generation frameworks such as FINN and hls4ml. In this paper, we present an alternative backend RTL library for FINN. We investigate and evaluate, across a spectrum of design dimensions, an RTL-based implementation versus the original HLS variant. We show that for smaller design parameters, RTL produces significantly smaller circuits. For larger circuits, however, the look-up table (LUT) count of RTL-based design is slightly higher, up to around $15\%$. On the other hand, HLS consistently requires more flip-flops (FFs) (orders-of-magnitude increase) and block RAMs (BRAMs) ($2\times$ more). This also impacts the critical path delay, with RTL producing significantly faster circuits, up to $80\%$. Furthermore, RTL also benefits from at-least a $10\times$ reduction in synthesis time. Finally the results were practically validated using a real-world use case of a multi-layer perceptron (MLP) network used in network intrusion detection. Overall, since HLS frameworks code-generate the hardware design, the benefits of the ease in the design entry is less important as compared to synthesis time reduction togther with resource benefits, this might make the RTL abstraction an attractive alternative.
Predicting Succinylation Sites in Proteins with Improved Deep Learning Architecture

Post-translational modifications (PTMs) in proteins occur after the process of translation. PTMs account for many cellular processes such as deoxyribonucleic acid (DNA) repair, cell signaling and cell death. One of the recent PTMs is succinylation. Succinylation modifies lysine residue from $-1$ to $+1$. Locating succinylation sites using experimental methods, such as mass spectrometry is very laborious. Hence, computational methods are favored using machine learning techniques. This paper proposes a deep learning architecture to predict succinylation sites. The performance of the proposed architecture is compared to the state-of-the-art deep learning architecture and other traditional machine learning techniques for succinylation. It is shown from the performance metrics that the proposed architecture provides a good trade-off between speed of computation and classification accuracy.
Extending the Use of MDL for High-Dimensional Problems: Variable Selection, Robust Fitting, and Additive Modeling

In the signal processing and statistics literature, the minimum description length (MDL) principle is a popular tool for choosing model complexity. Successful examples include signal denoising and variable selection in linear regression, for which the corresponding MDL solutions often enjoy consistent properties and produce very promising empirical results. This paper demonstrates that MDL can be extended naturally to the high-dimensional setting, where the number of predictors p is larger than the number of observations n. It first considers the case of linear regression, then allows for outliers in the data, and lastly extends to the robust fitting of nonparametric additive models. Results from numerical experiments are presented to demonstrate the efficiency and effectiveness of the MDL approach.
Deep Confidence Guided Distance for 3D Partial Shape Registration

We present a novel non-iterative learnable method for partial-to-partial 3D shape registration. The partial alignment task is extremely complex, as it jointly tries to match between points, and identify which points do not appear in the corresponding shape, causing the solution to be non-unique and ill-posed in most cases. Until now, two main methodologies have been suggested to solve this problem: sample a subset of points that are likely to have correspondences, or perform soft alignment between the point clouds and try to avoid a match to an occluded part. These heuristics work when the partiality is mild or when the transformation is small but fails for severe occlusions, or when outliers are present. We present a unique approach named Confidence Guided Distance Network (CGD-net), where we fuse learnable similarity between point embeddings and spatial distance between point clouds, inducing an optimized solution for the overlapping points while ignoring parts that only appear in one of the shapes. The point feature generation is done by a self-supervised architecture that repels far points to have different embeddings, therefore succeeds to align partial views of shapes, even with excessive internal symmetries, or acute rotations. We compare our network to recently presented learning-based and axiomatic methods and report a fundamental boost in performance.
Vizing's and Shannon's Theorems for defective edge colouring

We call a multigraph $(k,d)$-edge colourable if its edge set can be partitioned into $k$ subgraphs of maximum degree at most $d$ and denote as $\chi'_{d}(G)$ the minimum $k$ such that $G$ is $(k,d)$-edge colourable. We prove that for every odd integer $d$, every multigraph $G$ with maximum degree $\Delta$ is $(\lceil \frac{3\Delta - 1}{3d - 1} \rceil, d)$-edge colourable and this bound is attained for all values of $\Delta$ and $d$. An easy consequence of Vizing's Theorem is that, for every (simple) graph $G,$ $\chi'_{d}(G) \in \{ \lceil \frac{\Delta}{d} \rceil, \lceil \frac{\Delta+1}{d} \rceil \}$. We characterize the values of $d$ and $\Delta$ for which it is NP-complete to compute $\chi'_d(G)$. These results generalize classic results on the chromatic index of a graph by Shannon, Holyer, Leven and Galil and extend a result of Amini, Esperet and van den Heuvel.
The electric double layer at the interface between a polyelectrolyte gel and salt bath

The electric double layer (EDL) that forms at the interface between a polyelectrolyte gel and a salt bath is studied using asymptotic and numerical methods. Specifically, matched asymptotic expansions, based on the smallness of the Debye length relative to the typical gel dimensions, are used to construct solutions of the governing equations and derive electroneutral models with consistent jump conditions across the gel-bath interface. A general approach for solving the equations of incompressible nonlinear elasticity in a curved boundary layer is developed and used to resolve the gel mechanics in the EDL. A critical feature of the model is that it accounts for phase separation within the gel, which gives rise to diffuse interfaces with a characteristic thickness described by the Kuhn length. We show that the solutions of the electroneutral model can only be asymptotically matched to the solutions in the EDL, in general, when the Kuhn length greatly exceeds the Debye length. Conversely, if the Debye length is similar to or larger than the Kuhn length, then the entire gel can selforganise into periodic, electrically charged domains via phase separation. The breakdown of electroneutrality demonstrates that the commonly invoked electroneutral assumption must be used with caution, as it generally only applies when the Debye length is much smaller than the Kuhn length.