Miracle-dz/UniFAR-FTUs huggingface dataset:
Zeroth-Order Feedback-Based Optimization for Distributed Demand Response

Distributed demand response is a typical distributed optimization problem that requires coordination among multiple agents to satisfy demand response requirements. However, existing distributed algorithms for this problem still face challenges such as unknown system models, nonconvexity, privacy issues, etc. To address these challenges, we propose and analyze two distributed algorithms, in which the agents do not share their information and instead perform local updates using zeroth-order feedback information to estimate the gradient of the global objective function. One algorithm applies to problems with general convex and compact feasible sets but has higher oracle complexity bounded by $O(d/\epsilon^2)$, while the other algorithm achieves lower complexity bound $O(d/\epsilon)$ but is only applicable to problems with box constraints. We conduct empirical experiments to validate their performance.
Longer Fixations, More Computation: Gaze-Guided Recurrent Neural Networks

Humans read texts at a varying pace, while machine learning models treat each token in the same way in terms of a computational process. Therefore, we ask, does it help to make models act more like humans? In this paper, we convert this intuition into a set of novel models with fixation-guided parallel RNNs or layers and conduct various experiments on language modeling and sentiment analysis tasks to test their effectiveness, thus providing empirical validation for this intuition. Our proposed models achieve good performance on the language modeling task, considerably surpassing the baseline model. In addition, we find that, interestingly, the fixation duration predicted by neural networks bears some resemblance to humans' fixation. Without any explicit guidance, the model makes similar choices to humans. We also investigate the reasons for the differences between them, which explain why"model fixations"are often more suitable than human fixations, when used to guide language models.
Cumulant method for weighted random connection models

In this paper, we derive cumulant bounds for subgraph counts and power-weighted edge length in a class of spatial random networks known as weighted random connection models. This involves dealing with long-range spatial correlations induced by the profile function and the weight distribution. We start by deriving the bounds for the classical case of a Poisson vertex set, and then provide extensions to $\alpha$-determinantal processes.
Complexity of Single Loop Algorithms for Nonlinear Programming with Stochastic Objective and Constraints

We analyze the complexity of single-loop quadratic penalty and augmented Lagrangian algorithms for solving nonconvex optimization problems with functional equality constraints. We consider three cases, in all of which the objective is stochastic and smooth, that is, an expectation over an unknown distribution that is accessed by sampling. The nature of the equality constraints differs among the three cases: deterministic and linear in the first case, deterministic, smooth and nonlinear in the second case, and stochastic, smooth and nonlinear in the third case. Variance reduction techniques are used to improve the complexity. To find a point that satisfies $\varepsilon$-approximate first-order conditions, we require $\widetilde{O}(\varepsilon^{-3})$ complexity in the first case, $\widetilde{O}(\varepsilon^{-4})$ in the second case, and $\widetilde{O}(\varepsilon^{-5})$ in the third case. For the first and third cases, they are the first algorithms of"single loop"type (that also use $O(1)$ samples at each iteration) that still achieve the best-known complexity guarantees.
Gaussian smoothing stochastic gradient descent (GSmoothSGD)

Gaussian smoothing offers a nonlocal and gradient-free approach that has found successful application in the optimization of non-convex functions. This work formalizes and analyzes a Gaussian smoothing stochastic gradient descent (GSmoothSGD) method that aims at reducing noise and uncertainty in the gradient approximation and, thereby, enhancing the effectiveness of SGD by assisting it in efficiently navigating away from or circumventing local minima. This results in a notable performance boost when tackling non-convex optimization problems. To further improve convergence we also combine Gaussian smoothing with stochastic variance reduced gradient (GSmoothSVRG) and investigate its convergence properties. Our numerical examples involve optimizing two convex problems using a Monte-Carlo-based approximation of the nonlocal gradient that exhibit the advantages of the smoothing algorithms.
Flexible Tails for Normalising Flows, with Application to the Modelling of Financial Return Data

We propose a transformation capable of altering the tail properties of a distribution, motivated by extreme value theory, which can be used as a layer in a normalizing flow to approximate multivariate heavy tailed distributions. We apply this approach to model financial returns, capturing potentially extreme shocks that arise in such data. The trained models can be used directly to generate new synthetic sets of potentially extreme returns
Improved Planck constraints on axion-like early dark energy as a resolution of the Hubble tension

Axion-like early dark energy (EDE) as an extension to $\Lambda$CDM has been proposed as a possible solution to the 'Hubble tension'. We revisit this model using a new cosmic microwave background (CMB) temperature and polarization likelihood constructed from the {\it Planck} NPIPE data release. In a Bayesian analysis, we find that the maximum fractional contribution of EDE to the total energy density is $f_{\rm EDE}<0.061$ (without SH0ES) over the redshift range $z\in[10^3,10^4]$ and that the Hubble constant is constrained to lie within the range $ 66.9<H_0<69.5$ km/s/Mpc (both at 95 \% C.L.). The data therefore favour a model close to $\Lambda$CDM, leaving a residual tension of $3.7\sigma$ with the SH$0$ES Cepheid-based measurement of $H_0$. A comparison with the likelihood profile shows that our conclusions are robust to prior-volume effects. Our new CMB likelihood provides no evidence in favour of a significant EDE component.
BERTwich: Extending BERT's Capabilities to Model Dialectal and Noisy Text

Real-world NLP applications often deal with nonstandard text (e.g., dialectal, informal, or misspelled text). However, language models like BERT deteriorate in the face of dialect variation or noise. How do we push BERT's modeling capabilities to encompass nonstandard text? Fine-tuning helps, but it is designed for specializing a model to a task and does not seem to bring about the deeper, more pervasive changes needed to adapt a model to nonstandard language. In this paper, we introduce the novel idea of sandwiching BERT's encoder stack between additional encoder layers trained to perform masked language modeling on noisy text. We find that our approach, paired with recent work on including character-level noise in fine-tuning data, can promote zero-shot transfer to dialectal text, as well as reduce the distance in the embedding space between words and their noisy counterparts.
Intriguing Properties of Data Attribution on Diffusion Models

Data attribution seeks to trace model outputs back to training data. With the recent development of diffusion models, data attribution has become a desired module to properly assign valuations for high-quality or copyrighted training samples, ensuring that data contributors are fairly compensated or credited. Several theoretically motivated methods have been proposed to implement data attribution, in an effort to improve the trade-off between computational scalability and effectiveness. In this work, we conduct extensive experiments and ablation studies on attributing diffusion models, specifically focusing on DDPMs trained on CIFAR-10 and CelebA, as well as a Stable Diffusion model LoRA-finetuned on ArtBench. Intriguingly, we report counter-intuitive observations that theoretically unjustified design choices for attribution empirically outperform previous baselines by a large margin, in terms of both linear datamodeling score and counterfactual evaluation. Our work presents a significantly more efficient approach for attributing diffusion models, while the unexpected findings suggest that at least in non-convex settings, constructions guided by theoretical assumptions may lead to inferior attribution performance. The code is available at https://github.com/sail-sg/D-TRAK.
Federated Natural Policy Gradient Methods for Multi-task Reinforcement Learning

Federated reinforcement learning (RL) enables collaborative decision making of multiple distributed agents without sharing local data trajectories. In this work, we consider a multi-task setting, in which each agent has its own private reward function corresponding to different tasks, while sharing the same transition kernel of the environment. Focusing on infinite-horizon tabular Markov decision processes, the goal is to learn a globally optimal policy that maximizes the sum of the discounted total rewards of all the agents in a decentralized manner, where each agent only communicates with its neighbors over some prescribed graph topology. We develop federated vanilla and entropy-regularized natural policy gradient (NPG) methods under softmax parameterization, where gradient tracking is applied to the global Q-function to mitigate the impact of imperfect information sharing. We establish non-asymptotic global convergence guarantees under exact policy evaluation, which are nearly independent of the size of the state-action space and illuminate the impacts of network size and connectivity. To the best of our knowledge, this is the first time that global convergence is established for federated multi-task RL using policy optimization. Moreover, the convergence behavior of the proposed algorithms is robust against inexactness of policy evaluation.
Bandit-Driven Batch Selection for Robust Learning under Label Noise

We introduce a novel approach for batch selection in Stochastic Gradient Descent (SGD) training, leveraging combinatorial bandit algorithms. Our methodology focuses on optimizing the learning process in the presence of label noise, a prevalent issue in real-world datasets. Experimental evaluations on the CIFAR-10 dataset reveal that our approach consistently outperforms existing methods across various levels of label corruption. Importantly, we achieve this superior performance without incurring the computational overhead commonly associated with auxiliary neural network models. This work presents a balanced trade-off between computational efficiency and model efficacy, offering a scalable solution for complex machine learning applications.
Descriptive properties of I2-embeddings

We contribute to the study of generalizations of the Perfect Set Property and the Baire Property to subsets of spaces of higher cardinalities, like the power set $P(\lambda)$ of a singular cardinal $\lambda$ of countable cofinality or products $\prod_{i<\omega}\lambda_i$ for a strictly increasing sequence $\langle\lambda_i ~ \vert ~ i<\omega\rangle$ of cardinals. We consider the question under which large cardinal hypotheses classes of definable subsets of these spaces possess such regularity properties, focusing on rank-into-rank axioms and classes of sets definable by $\Sigma_1$-formulas with parameters from various collections of sets. We prove that $\omega$-many measurable cardinals, while sufficient to prove the Perfect Set Property of all $\Sigma_1$-definable sets with parameters in $V_\lambda\cup\{V_\lambda\}$, are not enough to prove it if there is a cofinal sequence in $\lambda$ in the parameters. For this conclusion, the existence of an I2-embedding is enough, but there are parameters in $V_{\lambda+1}$ for which I2 is still not enough. The situation is similar for the Baire Property: under I2 all sets that are $\Sigma_1$-definable using elements of $V_\lambda$ and a cofinal sequence as parameters have the Baire property, but I2 is not enough for some parameter in $V_{\lambda+1}$. Finally, the existence of an I0-embedding implies that all sets that are $\Sigma^1_n$-definable with parameters in $V_{\lambda+1}$ have the Baire property.
Obstructions to the existence of M{\o}ller maps

M{\o}ller maps are identifications between the observables of a perturbatively interacting physical system and the observables of its underlying free (i.e. non-interacting) system. This work studies and characterizes obstructions to the existence of such identifications. The main results are existence and importantly also non-existence theorems, which in particular imply that M{\o}ller maps do not exist for non-Abelian Chern-Simons and Yang-Mills theories on globally hyperbolic Lorentzian manifolds.
Mechanically-Inflatable Bio-Inspired Locomotion for Robotic Pipeline Inspection

Pipelines, vital for fluid transport, pose an important yet challenging inspection task, particularly in small, flexible biological systems, that robots have yet to master. In this study, we explored the development of an innovative robot inspired by the ovipositor of parasitic wasps to navigate and inspect pipelines. The robot features a flexible locomotion system that adapts to different tube sizes and shapes through a mechanical inflation technique. The flexible locomotion system employs a reciprocating motion, in which groups of three sliders extend and retract in a cyclic fashion. In a proof-of-principle experiment, the robot locomotion efficiency demonstrated positive linear correlation (r=0.6434) with the diameter ratio (ratio of robot diameter to tube diameter). The robot showcased a remarkable ability to traverse tubes of different sizes, shapes and payloads with an average of (70%) locomotion efficiency across all testing conditions, at varying diameter ratios (0.7-1.5). Furthermore, the mechanical inflation mechanism displayed substantial load-carrying capacity, producing considerable holding force of (13 N), equivalent to carrying a payload of approximately (5.8 Kg) inclusive the robot weight. This novel soft robotic system shows promise for inspection and navigation within tubular confined spaces, particularly in scenarios requiring adaptability to different tube shapes, sizes, and load-carrying capacities. This novel design serves as a foundation for a new class of pipeline inspection robots that exhibit versatility across various pipeline environments, potentially including biological systems.
FairLabel: Correcting Bias in Labels

There are several algorithms for measuring fairness of ML models. A fundamental assumption in these approaches is that the ground truth is fair or unbiased. In real-world datasets, however, the ground truth often contains data that is a result of historical and societal biases and discrimination. Models trained on these datasets will inherit and propagate the biases to the model outputs. We propose FairLabel, an algorithm which detects and corrects biases in labels. The goal of FairLabel is to reduce the Disparate Impact (DI) across groups while maintaining high accuracy in predictions. We propose metrics to measure the quality of bias correction and validate FairLabel on synthetic datasets and show that the label correction is correct 86.7% of the time vs. 71.9% for a baseline model. We also apply FairLabel on benchmark datasets such as UCI Adult, German Credit Risk, and Compas datasets and show that the Disparate Impact Ratio increases by as much as 54.2%.
Stochastic viability in an island model with partial dispersal : Approximation by a diffusion process in the limit of a large number of islands

In this paper, we study a finite population undergoing discrete, nonoverlapping generations, that is structured into $D$ demes, each containing $N$ individuals of two possible types, $A$ and $B$, whose viability coefficients, $s_A$ and $s_B$, respectively, vary randomly from one generation to the next. We assume that the means, variances and covariance of the viability coefficients are inversely proportional to the number of demes $D$, while higher-order moments are negligible in comparison to $1/D$. We use a discrete-time Markov chain with two time scales to model the evolutionary process, and we demonstrate that as the number of demes $D$ approaches infinity, the accelerated Markov chain converges to a diffusion process for any deme size $N\geq 2$. This diffusion process allows us to evaluate the fixation probability of type $A$ following its introduction as a single mutant in a population that was fixed for type $B$. We explore the impact of increasing the variability in the viability coefficients on this fixation probability. At least when $N$ is large enough, it is shown that increasing this variability for type $B$ or decreasing it for type $A$ leads to an increase in the fixation probability of a single $A$. The effect of the population-scaled variances, $\sigma^2_A$ and $\sigma^2_B$, can even cancel the effects of the population-scaled means, $\mu_A$ and $\mu_B$. We also show that the fixation probability of a single $A$ increases as the deme-scaled migration rate increases. Moreover, this probability is higher for type $A$ than for type $B$ if the population-scaled geometric mean is higher for type $A$ than for type $B$, which means that $\mu_A-\sigma_A^2/2>\mu_B-\sigma_B^2/2$.
Life and Death of a Thin Liquid Film

Thin films, bubbles and membranes are central to numerous natural and engineering processes, i.e., in thin-film solar cells, coatings, biosensors, electrowetting displays, foams, and emulsions. Yet, the characterization and an adequate understanding of their rupture is limited by the scarcity of atomic detail. We present here the complete life-cycle of freely suspended films using non-equilibrium molecular dynamics simulations of a simple atomic fluid free of surfactants and surface impurities, thus isolating the fundamental rupture mechanisms. Counter to the conventional notion that rupture occurs randomly, we discovered a short-term 'memory' by rewinding in time from a rupture event, extracting deterministic behaviors from apparent stochasticity. A comprehensive investigation of the key rupture-stages including both unrestrained and frustrated propagation is made - characterization of the latter leads to a first-order correction to the classical film-retraction theory. Furthermore, the highly resolved time window reveals that the different modes of the morphological development, typically characterized as heterogeneous nucleation and spinodal decomposition, continuously evolve seamlessly with time from one into the other.
On the Comparative Analysis of Average Treatment Effects Estimation via Data Combination

There is growing interest in exploring causal effects in target populations by combining multiple datasets. Nevertheless, most approaches are tailored to specific settings and lack comprehensive comparative analyses across different settings. In this article, within the typical scenario of a source dataset and a target dataset, we establish a unified framework for comparing various settings in causal inference via data combination. We first design six distinct settings, each with different available datasets and identifiability assumptions. The six settings cover a wide range of scenarios in the existing literature. We then conduct a comprehensive efficiency comparative analysis across these settings by calculating and comparing the semiparametric efficiency bounds for the average treatment effect (ATE) in the target population. Our findings reveal the key factors contributing to efficiency gains or losses across these settings. In addition, we extend our analysis to other estimands, including ATE in the source population and the average treatment effect on treated (ATT) in both the source and target populations. Furthermore, we empirically validate our findings by constructing locally efficient estimators and conducting extensive simulation studies. We demonstrate the proposed approaches using a real application to a MIMIC-III dataset as the target population and an eICU dataset as the source population.
Theory of Orbital Pumping

We develop a theory of orbital pumping, which corresponds to the emission of orbital currents from orbital dynamics. This phenomenon exhibits two distinct characteristics compared to spin pumping. Firstly, while spin pumping generates solely spin (angular momentum) currents, orbital pumping yields both orbital angular momentum currents and orbital angular position currents. Secondly, lattice vibrations induce orbital dynamics and associated orbital pumping as the orbital angular position is directly coupled to the lattice. These pumped orbital currents can be detected as transverse electric voltages via the inverse orbital(-torsion) Hall effect, stemming from orbital textures. Our work proposes a new avenue for generating orbital currents and provides a broader understanding of angular momentum dynamics encompassing spin, orbital, and phonon.
Spherical functions of symmetric forms and a conjecture of Hironaka

For all $r\ge1,$ we verify the following conjecture of Hironaka: for a $p$-adic field $F$ with $p$ odd, the space of spherical functions of $\mathrm{Sym}_{r\times r}(F)\cap\mathrm{GL}_r(F)$ is free of rank $4^r$ over the Hecke algebra.
On sums of Fourier coefficients of cusp forms twisted with additive characters

When $a_n$ is the $n$'th coefficient of some holomorphic cusp form, we prove a variety of Omega results for the twisted sum $\sum_{n\le x}a_ne^{2\pi in\alpha}$ and discuss their applications to the Ramanujan $\tau$-function and sums of $a_n$ over arithmetic progressions.
Augmenting deep neural networks with symbolic knowledge: Towards trustworthy and interpretable AI for education

Artificial neural networks (ANNs) have shown to be amongst the most important artificial intelligence (AI) techniques in educational applications, providing adaptive educational services. However, their educational potential is limited in practice due to three major challenges: i) difficulty in incorporating symbolic educational knowledge (e.g., causal relationships, and practitioners' knowledge) in their development, ii) learning and reflecting biases, and iii) lack of interpretability. Given the high-risk nature of education, the integration of educational knowledge into ANNs becomes crucial for developing AI applications that adhere to essential educational restrictions, and provide interpretability over the predictions. This research argues that the neural-symbolic family of AI has the potential to address the named challenges. To this end, it adapts a neural-symbolic AI framework and accordingly develops an approach called NSAI, that injects and extracts educational knowledge into and from deep neural networks, for modelling learners computational thinking. Our findings reveal that the NSAI approach has better generalizability compared to deep neural networks trained merely on training data, as well as training data augmented by SMOTE and autoencoder methods. More importantly, unlike the other models, the NSAI approach prioritises robust representations that capture causal relationships between input features and output labels, ensuring safety in learning to avoid spurious correlations and control biases in training data. Furthermore, the NSAI approach enables the extraction of rules from the learned network, facilitating interpretation and reasoning about the path to predictions, as well as refining the initial educational knowledge. These findings imply that neural-symbolic AI can overcome the limitations of ANNs in education, enabling trustworthy and interpretable applications.
Simplified polarization-encoding for BB84 QKD sourced by incoherent light of a silicon emitter

We investigate a polarization-encoded BB84-QKD transmitter that is simplified from an architectural and technological point-of-view, demonstrating a silicon emitter sourcing a low-complexity polarization modulator for secure-key generation at a raw-key rate of 2.8kb/s and QBER of 10.47%, underpinning the feasibility of an all-silicon QKD transmitter.