Miracle-dz/UniFAR-FTUs huggingface dataset:
Identifying good directions to escape the NTK regime and efficiently learn low-degree plus sparse polynomials

A recent goal in the theory of deep learning is to identify how neural networks can escape the"lazy training,"or Neural Tangent Kernel (NTK) regime, where the network is coupled with its first order Taylor expansion at initialization. While the NTK is minimax optimal for learning dense polynomials (Ghorbani et al, 2021), it cannot learn features, and hence has poor sample complexity for learning many classes of functions including sparse polynomials. Recent works have thus aimed to identify settings where gradient based algorithms provably generalize better than the NTK. One such example is the"QuadNTK"approach of Bai and Lee (2020), which analyzes the second-order term in the Taylor expansion. Bai and Lee (2020) show that the second-order term can learn sparse polynomials efficiently; however, it sacrifices the ability to learn general dense polynomials. In this paper, we analyze how gradient descent on a two-layer neural network can escape the NTK regime by utilizing a spectral characterization of the NTK (Montanari and Zhong, 2020) and building on the QuadNTK approach. We first expand upon the spectral analysis to identify"good"directions in parameter space in which we can move without harming generalization. Next, we show that a wide two-layer neural network can jointly use the NTK and QuadNTK to fit target functions consisting of a dense low-degree term and a sparse high-degree term -- something neither the NTK nor the QuadNTK can do on their own. Finally, we construct a regularizer which encourages our parameter vector to move in the"good"directions, and show that gradient descent on the regularized loss will converge to a global minimizer, which also has low test error. This yields an end to end convergence and generalization guarantee with provable sample complexity improvement over both the NTK and QuadNTK on their own.
Correct structures and similarity measures of soft sets along with historic comments of Prof. D.A. Molodtsov

After the paper of Molodtsov [Molodtsov D. Soft set theory — First results, Computers and Mathematics with Applications, 1999, vol. 37, no. 4-5, pp. 19-31. ] first appeared, soft set theory grew at a breakneck pace. Several authors have introduced various operations, relations, results, etc. as well as other aspects in soft set theory and hybrid structures incorrectly, despite their widespread use in mathematics and allied areas. In his paper [Molodtsov D.A. Equivalence and correct operations for soft sets, International Robotics and Automation Journal, 2018, vol. 4, no. 1, pp. 18-21. ], Molodtsov, the father of soft set theory, pointed out several wrong results and notions. Molodtsov [Molodtsov D.A. Structure of soft sets, Nechetkie Sistemy i Myagkie Vychisleniya, 2017, vol. 12, no. 1, pp. 5-18. ] also stated that the concept of soft set had not been fully understood and used everywhere. As a result, it is important to revisit the quirks of those conceptions and provide a formal account of the notion of soft set equivalency. Molodtsov already explored many correct operations on soft sets. We use some notions and results of Molodtsov [Molodtsov D.A. Structure of soft sets, Nechetkie Sistemy i Myagkie Vychisleniya, 2017, vol. 12, no. 1, pp. 5-18. ] to create matrix representations as well as related operations of soft sets, and to quantify the similarity between two soft sets.
Efficient Resource Allocation with Fairness Constraints in Restless Multi-Armed Bandits

Restless Multi-Armed Bandits (RMAB) is an apt model to represent decision-making problems in public health interventions (e.g., tuberculosis, maternal, and child care), anti-poaching planning, sensor monitoring, personalized recommendations and many more. Existing research in RMAB has contributed mechanisms and theoretical results to a wide variety of settings, where the focus is on maximizing expected value. In this paper, we are interested in ensuring that RMAB decision making is also fair to different arms while maximizing expected value. In the context of public health settings, this would ensure that different people and/or communities are fairly represented while making public health intervention decisions. To achieve this goal, we formally define the fairness constraints in RMAB and provide planning and learning methods to solve RMAB in a fair manner. We demonstrate key theoretical properties of fair RMAB and experimentally demonstrate that our proposed methods handle fairness constraints without sacrificing significantly on solution quality.
215 Billion Intersecting Brane Models

The landscape of string vacua is very large, but generally expected to be ﬁnite in size. Enumerating the number and properties of the vacua is an important task for both the landscape and the swampland, in part to gain a deeper understanding of what is possible and “generic”. We obtain an exact counting of distinct intersecting brane vacua of type IIA string theory on the T 6 / Z 2 × Z 2 orientifold. Care is taken to only count gauge-inequivalent brane conﬁgurations. Leveraging the recursive nature by which branes may be added together one-by-one, we use dynamic programming to eﬃciently count the number of solutions of the tadpole, K-theory and supersymmetry consistency conditions. The distributions of 4D gauge group rank and complex structure moduli for the entire ensemble of intersecting brane vacua are presented. The methods we developed here may be useful in obtaining sharp upper and lower bounds on other corners of the landscape.
Nucleic Acids Research

Poly A containing RNA isolated from embryonic chick calvaria was transferred from 6% formaldehyde 0.75% agarose gels to diazobenzyloxymethyl paper and the paper then hybridized to either nick translated pro al collagen cDNA clones, pCgl or pCg54, or to the nick translated pro a2 collagen cDNA clone, pCg45. From the mobilities of the bands hybridizing most strongly to each, pro a2 collagen mRNA was shown to be slightly larger than pro al mRNA; they are 5100 and 4900 nucleotides long respectively. pCg54 also hybridized weakly to two bands of lower mobility, corresponding to RNAs 6.4 and 5.6 kb long. Neither pCg54 nor pCg45 hybridized to type II procollagen mRNA in poly A containing RNA isolated from embryonic chick sterna.
DISH: A Distributed Hybrid Primal-Dual Optimization Framework to Utilize System Heterogeneity

We consider solving distributed consensus optimization problems over multi-agent networks. Current distributed methods fail to capture the heterogeneity among agents’ local computation capacities. We propose DISH as a distributed hybrid primal-dual algorithmic framework to handle and utilize system heterogeneity. Specifically, DISH allows those agents with higher computational capabilities or cheaper computational costs to implement Newton-type updates locally, while other agents can adopt the much simpler gradient-type updates. We show that DISH is a general framework and includes EXTRA, DIGing, and ESOM-0 as special cases. Moreover, when all agents take both primal and dual Newton-type updates, DISH approximates Newton’s method by estimating both primal and dual Hessians. Theoretically, we show that DISH achieves a linear (Q-linear) convergence rate to the exact optimal solution for strongly convex functions, regardless of agents’ choices of gradient-type and Newton-type updates. Finally, we perform numerical studies to demonstrate the efficacy of DISH in practice. To the best of our knowledge, DISH is the first hybrid method allowing heterogeneous local updates for distributed consensus optimization under general network topology with provable convergence and rate guarantees.
Learning Digital Terrain Models From Point Clouds: ALS2DTM Dataset and Rasterization-Based GAN

Despite the popularity of deep neural networks in various domains, the extraction of digital terrain models (DTMs) from airborne laser scanning (ALS) point clouds is still challenging. This might be due to the lack of the dedicated large-scale annotated dataset and the data-structure discrepancy between point clouds and DTMs. To promote data-driven DTM extraction, this article collects from open sources a large-scale dataset of ALS point clouds and corresponding DTMs with various urban, forested, and mountainous scenes. A baseline method is proposed as the first attempt to train a deep neural network to extract DTMs directly from ALS point clouds via rasterization techniques, coined DeepTerRa. Extensive studies with well-established methods are performed to benchmark the dataset and analyze the challenges in learning to extract DTM from point clouds. The experimental results show the interest of the agnostic data-driven approach, with submetric error level compared to methods designed for DTM extraction. The data and source code are available online at https://lhoangan.github. io/deepterra/ for reproducibility and further similar research.
Optimal finite-time processes in weakly driven overdamped Brownian motion

The complete physical understanding of the optimization of the thermodynamic work still is an important open problem in stochastic thermodynamics. We address this issue using the Hamiltonian approach of linear response theory in finite time and weak processes. We derive the Euler–Lagrange equation associated and discuss its main features, illustrating them using the paradigmatic example of driven Brownian motion in overdamped regime. We show that the optimal protocols obtained either coincide, in the appropriate limit, with the exact solutions by stochastic thermodynamics or can be even identical to them, presenting the well-known jumps. However, our approach reveals that jumps at the extremities of the process are a good optimization strategy in the regime of fast but weak processes for any driven system. Additionally, we show that fast-but-weak optimal protocols are time-reversal symmetric, a property that has until now remained hidden in the exact solutions far from equilibrium.
Discontinuity Computing Using Physics-Informed Neural Networks

Simulating discontinuities has been a long-standing challenge, especially when dealing with shock waves characterized by strong nonlinear features. Despite their promise, the recently developed physics-informed neural networks (PINNs) have not yet fully demonstrated their effectiveness in handling discontinuities when compared to traditional shock-capturing methods. In this study, we reveal a paradoxical phenomenon during the training of PINNs when computing problems with strong nonlinear discontinuities. To address this issue and enhance the PINNs’ ability to capture shocks, we propose PINNs-WE (Physics-Informed Neural Networks with Equation Weight) method by introducing three novel strategies. Firstly, we attenuate the neural network’s expression locally at ‘transition points’ within the shock waves by introducing a physics-dependent weight into the governing equations. Consequently, the neural network will concentrate on training the smoother parts of the solutions. As a result, due to the compressible property, sharp discontinuities emerge, with transition points being compressed into well-trained smooth regions akin to passive particles. Secondly, we also introduce the Rankine–Hugoniot (RH) relation, which is equivalent to the weak form of the conservation laws near the discontinuity, in order to improve the shock-capturing preformance. Lastly, we construct a global physical conservation constraint to enhance the conservation properties of PINNs which is key to resolve the right position of the discontinuity. To illustrate the impact of our novel approach, we investigate the behavior of the one-dimensional Burgers’ equation, as well as the one- and two-dimensional Euler equations. In our numerical experiments, we compare our proposed PINNs-WE method with a traditional high-order weighted essential non-oscillatory (WENO) approach. The results of our study highlight the significant enhancement in discontinuity computing by the PINNs-WE method when compared to traditional PINNs.
Wavelet Regularization Benefits Adversarial Training

Adversarial training methods are state-of-the-art (SOTA) empirical defense methods against adversarial examples. Many regularization methods have been proven to be effective with the combination of adversarial training. Nevertheless, such regularization methods are implemented in the time domain. Since adversarial vulnerability can be regarded as a high-frequency phenomenon, it is essential to regulate the adversarially-trained neural network models in the frequency domain. Faced with these challenges, we make a theoretical analysis on the regularization property of wavelets which can enhance adversarial training. We propose a wavelet regularization method based on the Haar wavelet decomposition which is named Wavelet Average Pooling. This wavelet regularization module is integrated into the wide residual neural network so that a new WideWaveletResNet model is formed. On the datasets of CIFAR-10 and CIFAR-100, our proposed Adversarial Wavelet Training method realizes considerable robustness under different types of attacks. It verifies the assumption that our wavelet regularization method can enhance adversarial robustness especially in the deep wide neural networks. The visualization experiments of the Frequency Principle (F-Principle) and interpretability are implemented to show the effectiveness of our method. A detailed comparison based on different wavelet base functions is presented. The code is available at the repository: \url{https://github.com/momo1986/AdversarialWaveletTraining}.
EiX-GNN : Concept-level eigencentrality explainer for graph neural networks

Nowadays, deep prediction models, especially graph neural networks, have a majorplace in critical applications. In such context, those models need to be highlyinterpretable or being explainable by humans, and at the societal scope, this understandingmay also be feasible for humans that do not have a strong prior knowledgein models and contexts that need to be explained. In the literature, explainingis a human knowledge transfer process regarding a phenomenon between an explainerand an explainee. We propose EiX-GNN (Eigencentrality eXplainer forGraph Neural Networks) a new powerful method for explaining graph neural networksthat encodes computationally this social explainer-to-explainee dependenceunderlying in the explanation process. To handle this dependency, we introducethe notion of explainee concept assimibility which allows explainer to adapt itsexplanation to explainee background or expectation. We lead a qualitative studyto illustrate our explainee concept assimibility notion on real-world data as wellas a qualitative study that compares, according to objective metrics established inthe literature, fairness and compactness of our method with respect to performingstate-of-the-art methods. It turns out that our method achieves strong results inboth aspects.
Towards Schema Inference for Data Lakes

A data lake is a repository of data with potential for future analysis. However, both discovering what data is in a data lake and exploring related data sets can take signiﬁcant eﬀort, as a data lake can contain an intimidating amount of heterogeneous data. In this paper, we propose the use of schema inference to support the interpretation of the data in the data lake. If a data lake is to support a schema-on-read paradigm, understanding the existing schema of relevant portions of the data lake seems like a prerequisite. In this paper, we make use of approximate indexes that can be used for data discovery to inform the inference of a schema for a data lake, consisting of entity types and the relationships between them. The speciﬁc approach identiﬁes candidate entity types by clustering similar data sets from the data lake, and then relationships between data sets in diﬀerent clusters are used to inform the identiﬁcation of relationships between the entity types. The approach is evaluated using real-world data repositories, to identify where the proposal is eﬀective, and to inform the identiﬁcation of areas for further work.
Models of Millimeter and Radio Emission from Interacting Supernovae

This work utilizes established models of synchrotron-powered light curves for core-collapse supernovae in dense circumstellar environments, namely, type IIn and Ibn, to demonstrate the potential for detecting millimeter emission from these events. The progenitor types of these supernovae are still an open question, but using the synchrotron light curves as probes for the circumstellar environments could shed light on the mass-loss histories of the progenitors and discern between different theories. Observations in millimeter bands are particularly fruitful, as they probe regions at smaller radii and higher ambient densities, where centimeter emission tends to be self-absorbed. In our application of these light curves, we explore a diversity of progenitor types and mass-loss profiles to understand their effects on the light-curve shapes. Additionally, we fit model parameters to the 8 GHz light curve of type IIn supernova 2006jd and then create millimeter light curves using these parameters to show the possibility of detecting an early millimeter peak from such an event. We predict that next generation millimeter surveys will possess the capability to detect nearby and extreme events. However, there is a pressing need for millimeter follow-up of optically discovered interacting supernovae to more completely sample the true population.
SUPER-IVIM-DC: Intra-voxel incoherent motion based Fetal lung maturity assessment from limited DWI data using supervised learning coupled with data-consistency

Intra-voxel incoherent motion (IVIM) analysis of fetal lungs Diffusion-Weighted MRI (DWI) data shows potential in providing quantitative imaging bio-markers that reflect, indirectly, diffusion and pseudo-diffusion for non-invasive fetal lung maturation assessment. However, long acquisition times, due to the large number of different ``b-value'' images required for IVIM analysis, precluded clinical feasibility. We introduce SUPER-IVIM-DC a deep-neural-networks (DNN) approach which couples supervised loss with a data-consistency term to enable IVIM analysis of DWI data acquired with a limited number of b-values. We demonstrated the added-value of SUPER-IVIM-DC over both classical and recent DNN approaches for IVIM analysis through numerical simulations, healthy volunteer study, and IVIM analysis of fetal lung maturation from fetal DWI data. Our numerical simulations and healthy volunteer study show that SUPER-IVIM-DC estimates of the IVIM model parameters from limited DWI data had lower normalized root mean-squared error compared to previous DNN-based approaches. Further, SUPER-IVIM-DC estimates of the pseudo-diffusion fraction parameter from limited DWI data of fetal lungs correlate better with gestational age compared to both to classical and DNN-based approaches (0.555 vs. 0.463 and 0.310). SUPER-IVIM-DC has the potential to reduce the long acquisition times associated with IVIM analysis of DWI data and to provide clinically feasible bio-markers for non-invasive fetal lung maturity assessment.
Two Ways of Understanding Social Dynamics: Analyzing the Predictability of Emergence of Objects in Reddit r/place Dependent on Locality in Space and Time

Lately, studying social dynamics in interacting agents has been boosted by the power of computer models, which bring the richness of qualitative work, while offering the precision, transparency, extensiveness, and replicability of statistical and mathematical approaches. A particular set of phenomena for the study of social dynamics is Web collaborative platforms. A dataset of interest is r/place, a collaborative social experiment held in 2017 on Reddit, which consisted of a shared online canvas of 1000 pixels by 1000 pixels co-edited by over a million recorded users over 72 hours. In this paper, we designed and compared two methods to analyze the dynamics of this experiment. Our first method consisted in approximating the set of 2D cellular-automata-like rules used to generate the canvas images and how these rules change over time. The second method consisted in a convolutional neural network (CNN) that learned an approximation to the generative rules in order to generate the complex outcomes of the canvas. Our results indicate varying context-size dependencies for the predictability of different objects in r/place in time and space. They also indicate a surprising peak in difficulty to statistically infer behavioral rules towards the middle of the social experiment, while user interactions did not drop until before the end. The combination of our two approaches, one rule-based and the other statistical CNN-based, shows the ability to highlight diverse aspects of analyzing social dynamics.
Predict better with less training data using a QNN

Over the past decade, machine learning revolutionized vision-based quality assessment for which convolutional neural networks (CNNs) have now become the standard. In this paper, we consider a potential next step in this development and describe a quanvolutional neural network (QNN) algorithm that efficiently maps classical image data to quantum states and allows for reliable image analysis. We practically demonstrate how to leverage quantum devices in computer vision and how to introduce quantum convolutions into classical CNNs. Dealing with a real world use case in industrial quality control, we implement our hybrid QNN model within the PennyLane framework and empirically observe it to achieve better predictions using much fewer training data than classical CNNs. In other words, we empirically observe a genuine quantum advantage for an industrial application where the advantage is due to superior data encoding.
A Method for Application of a Quantum Search Algorithm to Classical Databases

Grover’s algorithm is normally presented as a method of searching a database, however, it would be more accurately described as a method of identifying elements of an interval of the integers which satisfy some logical clause - an example might be identifying binary strings which correspond to the solutions of a Sudoku problem. In this paper we present the ﬁrst method of performing a true database search using Grover’s search algorithm, by ﬁrst creating a mapping from a set of indices in the range 0 : 2 n ´ 1 to a set of database elements, then applying the clause to these elements. We then demonstrate the feasibility of an attack against the Diﬃe-Hellman cryptosystem based on a Grover’s search of a database of candidate solutions generated via the number ﬁeld sieve algorithm.
A Unified Convergence Theorem for Stochastic Optimization Methods

In this work, we provide a fundamental unified convergence theorem used for deriving expected and almost sure convergence results for a series of stochastic optimization methods. Our unified theorem only requires to verify several representative conditions and is not tailored to any specific algorithm. As a direct application, we recover expected and almost sure convergence results of the stochastic gradient method (SGD) and random reshuffling (RR) under more general settings. Moreover, we establish new expected and almost sure convergence results for the stochastic proximal gradient method (prox-SGD) and stochastic model-based methods (SMM) for nonsmooth nonconvex optimization problems. These applications reveal that our unified theorem provides a plugin-type convergence analysis and strong convergence guarantees for a wide class of stochastic optimization methods.