Miracle-dz/UniFAR-FTUs huggingface dataset:
Stabilizing and Improving Federated Learning with Non-IID Data and Client Dropout

The label distribution skew induced data heterogeniety has been shown to be a significant obstacle that limits the model performance in federated learning, which is particularly developed for collaborative model training over decentralized data sources while preserving user privacy. This challenge could be more serious when the participating clients are in unstable circumstances and dropout frequently. Previous work and our empirical observations demonstrate that the classifier head for classification task is more sensitive to label skew and the unstable performance of FedAvg mainly lies in the imbalanced training samples across different classes. The biased classifier head will also impact the learning of feature representations. Therefore, maintaining a balanced classifier head is of significant importance for building a better global model. To this end, we propose a simple yet effective framework by introducing a prior-calibrated softmax function for computing the cross-entropy loss and a prototype-based feature augmentation scheme to re-balance the local training, which are lightweight for edge devices and can facilitate the global model aggregation. The improved model performance over existing baselines in the presence of non-IID data and client dropout is demonstrated by conducting extensive experiments on benchmark classification tasks.
Contrastive Learning under Heterophily

Graph Neural Networks are powerful tools for learning node representations when task-specific node labels are available. However, obtaining labels for graphs is expensive in many applications. This is particularly the case for large graphs. To address this, there has been a body of work to learn node representations in a self-supervised manner without labels. Contrastive learning (CL), has been particularly popular to learn representations in a self-supervised manner. In general, CL methods work by maximizing the similarity between representations of augmented views of the same example, and minimizing the similarity between augmented views of different examples. However, existing graph CL methods cannot learn high-quality representations under heterophily, where connected nodes tend to belong to different classes. This is because under heterophily, augmentations of the same example may not be similar to each other. In this work, we address the above problem by proposing the first graph CL method, HLCL, for learning node representations, under heterophily. HLCL uses a high-pass and a low-pass graph filter to generate different views of the same node. Then, it contrasts the two filtered views to learn the final node representations. Effectively, the high-pass filter captures the dissimilarity between nodes in a neighborhood and the low-pass filter captures the similarity between neighboring nodes.Contrasting the two filtered views allows HLCL to learn rich node representations for graphs, under heterophily and homophily.Empirically, HLCL outperforms state-of-the-art graph CL methods on benchmark heterophily datasets and large-scale real-world datasets by up to 10%.
Leveraging Pre-trained AudioLDM for Text to Sound Generation: A Benchmark Study

Deep neural networks have recently achieved breakthroughs in sound generation with text prompts. Despite their promising performance, current text-to-sound generation models face issues on small-scale datasets (e.g., overfitting), significantly limiting their performance. In this paper, we investigate the use of pre-trained AudioLDM, the state-of-the-art model for text-to-audio generation, as the backbone for sound generation. Our study demonstrates the advantages of using pre-trained models for text-to-sound generation, especially in data-scarcity scenarios. In addition, experiments show that different training strategies (e.g., training conditions) may affect the performance of AudioLDM on datasets of different scales. To facilitate future studies, we also evaluate various text-to-sound generation systems on several frequently used datasets under the same evaluation protocols, which allow fair comparisons and benchmarking of these methods on the common ground.
A semiparametric approach for interactive fixed effects panel data models

This paper presents a new approach for the estimation and inference of the regression parameters in a panel data model with interactive fixed effects. It relies on the assumption that the factor loadings can be expressed as an unknown smooth function of the time average of covariates plus an idiosyncratic error term. Compared to existing approaches, our estimator has a simple partial least squares form and does neither require iterative procedures nor the previous estimation of factors. We derive its asymptotic properties by finding out that the limiting distribution has a discontinuity, depending on the explanatory power of our basis functions which is expressed by the variance of the error of the factor loadings. As a result, the usual ``plug-in"methods based on estimates of the asymptotic covariance are only valid pointwise and may produce either over- or under-coverage probabilities. We show that uniformly valid inference can be achieved by using the cross-sectional bootstrap. A Monte Carlo study indicates good performance in terms of mean squared error. We apply our methodology to analyze the determinants of growth rates in OECD countries.
Graph Learning and Its Advancements on Large Language Models: A Holistic Survey

Graph learning is a prevalent domain that endeavors to learn the intricate relationships among nodes and the topological structure of graphs. Over the years, graph learning has transcended from graph theory to graph data mining. With the advent of representation learning, it has attained remarkable performance in diverse scenarios. Owing to its extensive application prospects, graph learning attracts copious attention. While some researchers have accomplished impressive surveys on graph learning, they failed to connect related objectives, methods, and applications in a more coherent way. As a result, they did not encompass current ample scenarios and challenging problems due to the rapid expansion of graph learning. Particularly, large language models have recently had a disruptive effect on human life, but they also show relative weakness in structured scenarios. The question of how to make these models more powerful with graph learning remains open. Our survey focuses on the most recent advancements in integrating graph learning with pre-trained language models, specifically emphasizing their application within the domain of large language models. Different from previous surveys on graph learning, we provide a holistic review that analyzes current works from the perspective of graph structure, and discusses the latest applications, trends, and challenges in graph learning. Specifically, we commence by proposing a taxonomy and then summarize the methods employed in graph learning. We then provide a detailed elucidation of mainstream applications. Finally, we propose future directions.
Tur\'an Number of Subdivisions of Multipartite Graphs

In this paper, we initiate the study of the extremal exponent for the $1$-subdivisions of non-bipartite non-complete graphs. Let $K_{C_k,1}$ be the graph by connecting an additional vertex to every vertex of the cycle $C_k$, $k\geq 4$. We show that for the $1$-subdivision of the tripartite graph $K_{C_k,1}^{\text{sub}}$, its Tur\'an number is $\Theta(n^{4/3})$. Then we obtain the upper bound for the extremal number of a family of graphs consisting of (possibly degenerate) $1$-subdivisions of certain tripartite graphs. We also obtain the same upper bound for the 1-subdivision of $(K_{s+1,t}^+)^{sub}$, where $K_{s+1,t}^+$ is a graph obtained by joining $K_{s,t}$ with one new vertex $r$ by means of connecting $r$ with one vertex in the $s$-part and all the vertices in the $t$-part.
Representation Learning for Stack Overflow Posts: How Far are We?

The tremendous success of Stack Overflow has accumulated an extensive corpus of software engineering knowledge, thus motivating researchers to propose various solutions for analyzing its content. The performance of such solutions hinges significantly on the selection of representation models for Stack Overflow posts. As the volume of literature on Stack Overflow continues to burgeon, it highlights the need for a powerful Stack Overflow post representation model and drives researchers’ interest in developing specialized representation models that can adeptly capture the intricacies of Stack Overflow posts. The state-of-the-art (SOTA) Stack Overflow post representation models are Post2Vec and BERTOverflow, which are built upon neural networks such as convolutional neural network (CNN) and transformer architecture (e.g., BERT). Despite their promising results, these representation methods have not been evaluated in the same experimental setting. To fill the research gap, we first empirically compare the performance of the representation models designed specifically for Stack Overflow posts (Post2Vec and BERTOverflow) in a wide range of related tasks, i.e., tag recommendation, relatedness prediction, and API recommendation. The results show that Post2Vec cannot further improve the state-of-the-art techniques of the considered downstream tasks, and BERTOverflow shows surprisingly poor performance. To find more suitable representation models for the posts, we further explore a diverse set of transformer-based models, including (1) general domain language models (RoBERTa, Longformer, GPT2) and (2) language models built with software engineering-related textual artifacts (CodeBERT, GraphCodeBERT, seBERT, CodeT5, PLBart, and CodeGen). This exploration shows that models like CodeBERT and RoBERTa are suitable for representing Stack Overflow posts. However, it also illustrates the “No Silver Bullet” concept, as none of the models consistently wins against all the others. Inspired by the findings, we propose SOBERT, which employs a simple yet effective strategy to improve the representation models of Stack Overflow posts by continuing the pre-training phase with the textual artifact from Stack Overflow. The overall experimental results demonstrate that SOBERT can consistently outperform the considered models and increase the state-of-the-art performance significantly for all the downstream tasks.
Quantifying the Effects of Magnetic Field Line Curvature Scattering on Radiation Belt and Ring Current Particles

Magnetic field line curvature (FLC) scattering is a collisionless scattering mechanism that arises when a particle's gyro-radius is comparable to the magnetic field line's curvature radius, resulting in the breaking of the conservation of the first adiabatic invariant. Studies in recent years have explored the implications of FLC scattering on the precipitation of both ring current ions and radiation belt electrons. In this work, we first compare two previous FLC scattering coefficients using test particle calculations. Then, we systematically calculate diffusion coefficients from FLC scattering in radial and MLT directions for particles of various energy levels, as well as its sensitivity to the $Kp$ index. We find that the timescale of FLC scattering is sufficient to account for the sudden loss of MeV electrons near the geostationary orbit during disturbed times. Additionally, the decay time of ring current protons is on the order of hours to minutes, providing an explanation for the ring current decay throughout the recovery phase of magnetic storms. Lastly, we compare the effects of wave-particle resonant scattering and FLC scattering in the vicinity of the midnight equator. Our findings suggest that the impacts of FLC scattering on MeV electrons or hundreds keV protons with smaller pitch angle is comparable to, or even more significant than, the effects of whistler mode or EMIC wave resonant scattering. Our quantitative results should be useful to evaluate the importance of the effects of FLC scattering while modeling the dynamics of radiation belt and ring current.
Diagonal cycles and anticyclotomic Iwasawa theory of modular forms

We construct a new anticyclotomic Euler system (in the sense of Jetchev-Nekovar-Skinner) for the Galois representation $V_{f,\chi}$ attached to a newform $f$ of weight $k\geq 2$ twisted by an anticyclotomic Hecke character $\chi$. We then show some arithmetic applications of the constructed Euler system, including new results on the Bloch-Kato conjecture in ranks zero and one, and a divisibility towards the Iwasawa-Greenberg main conjecture for $V_{f,\chi}$. In particular, in the case where the base-change of $f$ to our imaginary quadratic field has root number $+1$ and $\chi$ has higher weight (which implies that the complex $L$-function $L(V_{f,\chi},s)$ vanishes at the center), our results show that the Bloch-Kato Selmer group of $V_{f,\chi}$ is nonzero, and if a certain distinguished class $\kappa_{f,\chi}$ is nonzero, then the Selmer group is one-dimensional. Such applications to the Bloch-Kato conjecture were left wide open by the earlier approaches using Heegner cycles and/or Beilinson-Flach classes. Our construction is based instead on a generalisation of the Gross-Kudla-Schoen diagonal cycles.
Incentives and co-evolution: Steering linear dynamical systems with noncooperative agents

Modern socio-technical systems typically consist of many interconnected users and competing service providers, where notions like market equilibrium are tightly connected to the ``evolution'' of the network of users. In this paper, we model the users' dynamics as a linear dynamical system, and the service providers as agents taking part to a generalized Nash game, whose outcome coincides with the input of the users' dynamics. We thus characterize the notion of co-evolution of the market and the network dynamics and derive dissipativity-based conditions leading to a pertinent notion of equilibrium. We then focus on the control design and adopt the light-touch policy to incentivize or penalize the service providers as little as possible, while steering the networked system to a desirable outcome. We also provide a dimensionality-reduction procedure, which offers network-size independent conditions. Finally, we illustrate our novel notions and algorithms on a simulation setup stemming from digital market regulations for influencers, a topic of growing interest.
On a set of diophantine equations

• A submitted manuscript is the version of the article upon submission and before peer-review. There can be important differences between the submitted version and the official published version of record. People interested in the research are advised to contact the author for the final version of the publication, or visit the DOI to the publisher's website. • The final author version and the galley proof are versions of the publication after peer review. • The final published version features the final layout of the paper including the volume, issue and page numbers.
Delay-dependent robust stability and H8 control of jump linear systems

This paper considers the class of continuous-time jump linear systems with time-delay and polytopic uncertain parameters. When the time-delay is a known constant, by using the linear matrix inequality (LMI) technique, we ® rst establish delay dependent su cient conditions for robust stability and robust H1 control of the class of systems under study. The problem of determining the maximum time-delay under which the system will remain stable is cast into a generalized eigenvalue problem and thus solved by LMI techniques. When the control input contains a time-delay, an algorithm to design a state feedback controller with constant gain matrix is developed. Two examples are given to show the validness of the theoretical results.
Eternal homogeneous gravitational collapse: A comprehensive analysis from $\Theta$ parametrization

A new class of self-gravitating collapsing star models with perfect fluid distributions is discussed in this work. The paper has a comprehensive analysis of a homogeneous gravitational collapsing system wherein using a parametrization scheme for the expansion-scalar $(\Theta )$, the solutions of the Einstein Field Equations (EFEs) are determined independently. The background geometry for the analysis is considered to be homogeneous and isotropic represented by a Friedmann-Leimatre-Robertson-Walker (FLRW) metric and by employing the boundary conditions, we have discussed the solution in more detail. Further, all the physical and geometrical parameters are obtained in terms of Schwarzschild mass $(M)$ that makes the model significant in astrophysical applications. The singularity analysis of the collapsing system is also discussed by the apparent-horizon, and it has seen that the homogenous gravitational collapse turns into a new kind of scenario- Eternal collapsing object.
Modelling and analysis of non-linear orifice type damping in vibration isolators

The effects of non-linear orifice type damping on the response of one- and two-degree of freedom (DOF) systems are studied. The damping is formulated through the hydraulic resistance of the fluid path as a function of the frequency of flow oscillation and mean flow velocity for the laminar and turbulent flows. A comparison is made to demonstrate the vibration control capability of linear and non-linear damping models in terms of displacement transmissibility ratios for a 1-DOF system and for a 2-DOF system with an optimally tuned absorber, providing the minimum peak displacement of the isolated mass over the complete frequency range of interest. The results show a significant shift in resonant frequency between the linear and non-linear damping approaches.
Thin-Film Evolution Equation for a Strained Solid Film on a Deformable Substrate: Numerical Steady States

We consider the nonlinear behavior of the thin-film evolution equation for a strained solid film on a substrate. The evolution equation describes morphological changes to the film by surface diffusion in response to elastic energy, surface energy, and wetting energy. Due to the thin-film approximation, the elastic response of the film is determined analytically, resulting in a self-contained evolution equation which does not require separate numerical solution of the full three-dimensional elasticity problem. Using a pseudospectral predictor-corrector method we numerically determine the family of steady state solutions to this evolution equation which correspond to quantum dot and quantum ridge morphologies.
One Transformer Fits All Distributions in Multi-Modal Diffusion at Scale

This paper proposes a unified diffusion framework (dubbed UniDiffuser) to fit all distributions relevant to a set of multi-modal data in one model. Our key insight is -- learning diffusion models for marginal, conditional, and joint distributions can be unified as predicting the noise in the perturbed data, where the perturbation levels (i.e. timesteps) can be different for different modalities. Inspired by the unified view, UniDiffuser learns all distributions simultaneously with a minimal modification to the original diffusion model -- perturbs data in all modalities instead of a single modality, inputs individual timesteps in different modalities, and predicts the noise of all modalities instead of a single modality. UniDiffuser is parameterized by a transformer for diffusion models to handle input types of different modalities. Implemented on large-scale paired image-text data, UniDiffuser is able to perform image, text, text-to-image, image-to-text, and image-text pair generation by setting proper timesteps without additional overhead. In particular, UniDiffuser is able to produce perceptually realistic samples in all tasks and its quantitative results (e.g., the FID and CLIP score) are not only superior to existing general-purpose models but also comparable to the bespoken models (e.g., Stable Diffusion and DALL-E 2) in representative tasks (e.g., text-to-image generation).
Amino acid balancing for the prediction and evaluation of protein concentrations in cell‐free protein synthesis systems

Cell‐free protein synthesis (CFPS) systems are an attractive method to complement the usual cell‐based synthesis of proteins, especially for screening approaches. The literature describes a wide variety of CFPS systems, but their performance is difficult to compare since the reaction components are often used at different concentrations. Therefore, we have developed a calculation tool based on amino acid balancing to evaluate the performance of CFPS by determining the fractional yield as the ratio between theoretically achievable and experimentally achieved protein molar concentration. This tool was applied to a series of experiments from our lab and to various systems described in the literature to identify systems that synthesize proteins very efficiently and those that still have potential for higher yields. The well‐established Escherichia coli system showed a high efficiency in the utilization of amino acids, but interestingly, less considered systems, such as those based on Vibrio natriegens or Leishmania tarentolae, also showed exceptional fractional yields of over 70% and 90%, respectively, implying very efficient conversions of amino acids. The methods and tools described here can quickly identify when a system has reached its maximum or has limitations. We believe that this approach will facilitate the evaluation and optimization of existing CFPS systems and provides the basis for the systematic development of new CFPS systems.
Stabilizing Transformer Training by Preventing Attention Entropy Collapse

Training stability is of great importance to Transformers. In this work, we investigate the training dynamics of Transformers by examining the evolution of the attention layers. In particular, we track the attention entropy for each attention head during the course of training, which is a proxy for model sharpness. We identify a common pattern across different architectures and tasks, where low attention entropy is accompanied by high training instability, which can take the form of oscillating loss or divergence. We denote the pathologically low attention entropy, corresponding to highly concentrated attention scores, as $\textit{entropy collapse}$. As a remedy, we propose $\sigma$Reparam, a simple and efficient solution where we reparametrize all linear layers with spectral normalization and an additional learned scalar. We demonstrate that $\sigma$Reparam successfully prevents entropy collapse in the attention layers, promoting more stable training. Additionally, we prove a tight lower bound of the attention entropy, which decreases exponentially fast with the spectral norm of the attention logits, providing additional motivation for our approach. We conduct experiments with $\sigma$Reparam on image classification, image self-supervised learning, machine translation, speech recognition, and language modeling tasks. We show that $\sigma$Reparam provides stability and robustness with respect to the choice of hyperparameters, going so far as enabling training (a) a Vision Transformer {to competitive performance} without warmup, weight decay, layer normalization or adaptive optimizers; (b) deep architectures in machine translation and (c) speech recognition to competitive performance without warmup and adaptive optimizers. Code is available at \url{https://github.com/apple/ml-sigma-reparam}.