Miracle-dz/UniFAR-FTUs huggingface dataset:
Federated Fairness without Access to Sensitive Groups

Current approaches to group fairness in federated learning assume the existence of predefined and labeled sensitive groups during training. However, due to factors ranging from emerging regulations to dynamics and location-dependency of protected groups, this assumption may be unsuitable in many real-world scenarios. In this work, we propose a new approach to guarantee group fairness that does not rely on any predefined definition of sensitive groups or additional labels. Our objective allows the federation to learn a Pareto efficient global model ensuring worst-case group fairness and it enables, via a single hyper-parameter, trade-offs between fairness and utility, subject only to a group size constraint. This implies that any sufficiently large subset of the population is guaranteed to receive at least a minimum level of utility performance from the model. The proposed objective encompasses existing approaches as special cases, such as empirical risk minimization and subgroup robustness objectives from centralized machine learning. We provide an algorithm to solve this problem in federation that enjoys convergence and excess risk guarantees. Our empirical results indicate that the proposed approach can effectively improve the worst-performing group that may be present without unnecessarily hurting the average performance, exhibits superior or comparable performance to relevant baselines, and achieves a large set of solutions with different fairness-utility trade-offs.
Which Model to Transfer? A Survey on Transferability Estimation

Transfer learning methods endeavor to leverage relevant knowledge from existing source pre-trained models or datasets to solve downstream target tasks. With the increase in the scale and quantity of available pre-trained models nowadays, it becomes critical to assess in advance whether they are suitable for a specific target task. Model transferability estimation is an emerging and growing area of interest, aiming to propose a metric to quantify this suitability without training them individually, which is computationally prohibitive. Despite extensive recent advances already devoted to this area, they have custom terminological definitions and experimental settings. In this survey, we present the first review of existing advances in this area and categorize them into two separate realms: source-free model transferability estimation and source-dependent model transferability estimation. Each category is systematically defined, accompanied by a comprehensive taxonomy. Besides, we address challenges and outline future research directions, intending to provide a comprehensive guide to aid researchers and practitioners.
Algorithmically Fair Maximization of Multiple Submodular Objective Functions

Constrained maximization of submodular functions poses a central problem in combinatorial optimization. In many realistic scenarios, a number of agents need to maximize multiple submodular objectives over the same ground set. We study such a setting, where the different solutions must be disjoint, and thus, questions of fairness arise. Inspired from the fair division literature, we suggest a simple round-robin protocol, where agents are allowed to build their solutions one item at a time by taking turns. Unlike what is typical in fair division, however, the prime goal here is to provide a fair algorithmic environment; each agent is allowed to use any algorithm for constructing their respective solutions. We show that just by following simple greedy policies, agents have solid guarantees for both monotone and non-monotone objectives, and for combinatorial constraints as general as $p$-systems (which capture cardinality and matroid intersection constraints). In the monotone case, our results include approximate EF1-type guarantees and their implications in fair division may be of independent interest. Further, although following a greedy policy may not be optimal in general, we show that consistently performing better than that is computationally hard.
Practical Insights into Knowledge Distillation for Pre-Trained Models

This research investigates the enhancement of knowledge distillation (KD) processes in pre-trained models, an emerging field in knowledge transfer with significant implications for distributed training and federated learning environments. These environments benefit from reduced communication demands and accommodate various model architectures. Despite the adoption of numerous KD approaches for transferring knowledge among pre-trained models, a comprehensive understanding of KD's application in these scenarios is lacking. Our study conducts an extensive comparison of multiple KD techniques, including standard KD, tuned KD (via optimized temperature and weight parameters), deep mutual learning, and data partitioning KD. We assess these methods across various data distribution strategies to identify the most effective contexts for each. Through detailed examination of hyperparameter tuning, informed by extensive grid search evaluations, we pinpoint when adjustments are crucial to enhance model performance. This paper sheds light on optimal hyperparameter settings for distinct data partitioning scenarios and investigates KD's role in improving federated learning by minimizing communication rounds and expediting the training process. By filling a notable void in current research, our findings serve as a practical framework for leveraging KD in pre-trained models within collaborative and federated learning frameworks.
Quantum Theory and Application of Contextual Optimal Transport

Optimal Transport (OT) has fueled machine learning (ML) applications across many domains. In cases where paired data measurements ($\mu$, $\nu$) are coupled to a context variable $p_i$ , one may aspire to learn a global transportation map that can be parameterized through a potentially unseen con-text. Existing approaches utilize Neural OT and largely rely on Brenier's theorem. Here, we propose a first-of-its-kind quantum computing formulation for amortized optimization of contextualized transportation plans. We exploit a direct link between doubly stochastic matrices and unitary operators thus finding a natural connection between OT and quantum computation. We verify our method on synthetic and real data, by predicting variations in cell type distributions parameterized through drug dosage as context. Our comparisons to several baselines reveal that our method can capture dose-induced variations in cell distributions, even to some extent when dosages are extrapolated and sometimes with performance similar to the best classical models. In summary, this is a first step toward learning to predict contextualized transportation plans through quantum.
Second-Order Fine-Tuning without Pain for LLMs: A Hessian Informed Zeroth-Order Optimizer

Fine-tuning large language models (LLMs) with classic first-order optimizers entails prohibitive GPU memory due to the backpropagation process. Recent works have turned to zeroth-order optimizers for fine-tuning, which save substantial memory by using two forward passes. However, these optimizers are plagued by the heterogeneity of parameter curvatures across different dimensions. In this work, we propose HiZOO, a diagonal Hessian informed zeroth-order optimizer which is the first work to leverage the diagonal Hessian to enhance zeroth-order optimizer for fine-tuning LLMs. What's more, HiZOO avoids the expensive memory cost and only increases one forward pass per step. Extensive experiments on various models (350M~66B parameters) indicate that HiZOO improves model convergence, significantly reducing training steps and effectively enhancing model accuracy. Moreover, we visualize the optimization trajectories of HiZOO on test functions, illustrating its effectiveness in handling heterogeneous curvatures. Lastly, we provide theoretical proofs of convergence for HiZOO. Code is publicly available at https://anonymous.4open. science/r/HiZOO27F8.
Artificial Bee Colony optimization of Deep Convolutional Neural Networks in the context of Biomedical Imaging

Most efforts in Computer Vision focus on natural images or artwork, which differ significantly both in size and contents from the kind of data biomedical image processing deals with. Thus, Transfer Learning models often prove themselves suboptimal for these tasks, even after manual finetuning. The development of architectures from scratch is oftentimes unfeasible due to the vastness of the hyperparameter space and a shortage of time, computational resources and Deep Learning experts in most biomedical research laboratories. An alternative to manually defining the models is the use of Neuroevolution, which employs metaheuristic techniques to optimize Deep Learning architectures. However, many algorithms proposed in the neuroevolutive literature are either too unreliable or limited to a small, predefined region of the hyperparameter space. To overcome these shortcomings, we propose the Chimera Algorithm, a novel, hybrid neuroevolutive algorithm that integrates the Artificial Bee Colony Algorithm with Evolutionary Computation tools to generate models from scratch, as well as to refine a given previous architecture to better fit the task at hand. The Chimera Algorithm has been validated with two datasets of natural and medical images, producing models that surpassed the performance of those coming from Transfer Learning.
Font Impression Estimation in the Wild

This paper addresses the challenging task of estimating font impressions from real font images. We use a font dataset with annotation about font impressions and a convolutional neural network (CNN) framework for this task. However, impressions attached to individual fonts are often missing and noisy because of the subjective characteristic of font impression annotation. To realize stable impression estimation even with such a dataset, we propose an exemplar-based impression estimation approach, which relies on a strategy of ensembling impressions of exemplar fonts that are similar to the input image. In addition, we train CNN with synthetic font images that mimic scanned word images so that CNN estimates impressions of font images in the wild. We evaluate the basic performance of the proposed estimation method quantitatively and qualitatively. Then, we conduct a correlation analysis between book genres and font impressions on real book cover images; it is important to note that this analysis is only possible with our impression estimation method. The analysis reveals various trends in the correlation between them - this fact supports a hypothesis that book cover designers carefully choose a font for a book cover considering the impression given by the font.
Comparing Inferential Strategies of Humans and Large Language Models in Deductive Reasoning

Deductive reasoning plays a pivotal role in the formulation of sound and cohesive arguments. It allows individuals to draw conclusions that logically follow, given the truth value of the information provided. Recent progress in the domain of large language models (LLMs) has showcased their capability in executing deductive reasoning tasks. Nonetheless, a significant portion of research primarily assesses the accuracy of LLMs in solving such tasks, often overlooking a deeper analysis of their reasoning behavior. In this study, we draw upon principles from cognitive psychology to examine inferential strategies employed by LLMs, through a detailed evaluation of their responses to propositional logic problems. Our findings indicate that LLMs display reasoning patterns akin to those observed in humans, including strategies like $\textit{supposition following}$ or $\textit{chain construction}$. Moreover, our research demonstrates that the architecture and scale of the model significantly affect its preferred method of reasoning, with more advanced models tending to adopt strategies more frequently than less sophisticated ones. Importantly, we assert that a model's accuracy, that is the correctness of its final conclusion, does not necessarily reflect the validity of its reasoning process. This distinction underscores the necessity for more nuanced evaluation procedures in the field.
The Logic of Correct Models

For each $n\in\mathbb{N}$, let $[n]\phi$ mean"the sentence $\phi$ is true in all $\Sigma_{n+1}$-correct transitive sets. "Assuming G\"odel's axiom $V = L$, we prove the following graded variant of Solovay's completeness theorem: the set of formulas valid under this interpretation is precisely the set of theorems of the linear provability logic GLP.3. We also show that this result is not provable in ZFC, so the hypothesis V = L cannot be removed. As part of the proof, we derive (in ZFC) the following purely modal-logical results which are of independent interest: the logic GLP.3 coincides with the logic of closed substitutions of GLP, and is the maximal non-degenerate, normal extension of GLP.
Stable Neural Stochastic Differential Equations in Analyzing Irregular Time Series Data

Irregular sampling intervals and missing values in real-world time series data present challenges for conventional methods that assume consistent intervals and complete data. Neural Ordinary Differential Equations (Neural ODEs) offer an alternative approach, utilizing neural networks combined with ODE solvers to learn continuous latent representations through parameterized vector fields. Neural Stochastic Differential Equations (Neural SDEs) extend Neural ODEs by incorporating a diffusion term, although this addition is not trivial, particularly when addressing irregular intervals and missing values. Consequently, careful design of drift and diffusion functions is crucial for maintaining stability and enhancing performance, while incautious choices can result in adverse properties such as the absence of strong solutions, stochastic destabilization, or unstable Euler discretizations, significantly affecting Neural SDEs' performance. In this study, we propose three stable classes of Neural SDEs: Langevin-type SDE, Linear Noise SDE, and Geometric SDE. Then, we rigorously demonstrate their robustness in maintaining excellent performance under distribution shift, while effectively preventing overfitting. To assess the effectiveness of our approach, we conduct extensive experiments on four benchmark datasets for interpolation, forecasting, and classification tasks, and analyze the robustness of our methods with 30 public datasets under different missing rates. Our results demonstrate the efficacy of the proposed method in handling real-world irregular time series data.
Original orthorhombic tetrahedral and trigonal hybrid allotropes Cn with n= 8, 10, 12, 14 possessing ethene -like and propadiene -like units: Crystal chemistry and first principles

Original carbon allotropes, orthorhombic C8, C10, C12 and C14 presenting mixed sp2 and sp3 carbon hybridizations exhibiting ethene like and propadiene like embedded units are proposed from crystal chemistry and calculations within the quantum density functional theory DFT. The carbon allotropes with topologies related with jeb, mog, as well as new topologies, show alternating tetrahedral and trigonal carbon stacking along the a orthorhombic direction (vertical) close to but different from isoglitter. The carbon allotropes were shown to be cohesive and stable both mechanically (elastic properties) and dynamically (phonons), with calculated Vickers hardness (HV) magnitudes ranging between 25 GPa and 52 GPa, the latter magnitude assigned to C12 possessing the largest number of tetrahedral versus trigonal stacking. High phonon frequencies close to 50 THz were attributed to stretching mode of ethene (in C8 and C12) and propadiene (in C10 and C14) components with good agreement with experiment for Raman molecular C double bond C intense stretching mode. Frequencies close to 40 THz were proposed as signatures of C.C simple bonds in the tetrahedra. The electronic band structure and electronic density of states DOS shown exemplarily for C8 point to metallic-like behavior assigned to the itinerant role of trigonal carbon pi-electrons.
Technical Report on the Checkfor.ai AI-Generated Text Classifier

We present the CheckforAI text classifier, a transformer-based neural network trained to distinguish text written by large language models from text written by humans. CheckforAI outperforms zero-shot methods such as DetectGPT as well as leading commercial AI detection tools with over 9 times lower error rates on a comprehensive benchmark comprised of ten text domains (student writing, creative writing, scientific writing, books, encyclopedias, news, email, scientific papers, short-form Q&A) and 8 open- and closed-source large language models. We propose a training algorithm, hard negative mining with synthetic mirrors, that enables our classifier to achieve orders of magnitude lower false positive rates on high-data domains such as reviews. Finally, we show that CheckforAI is not biased against nonnative English speakers and generalizes to domains and models unseen during training.
Interpreting Context Look-ups in Transformers: Investigating Attention-MLP Interactions

In this paper, we investigate the interplay between attention heads and specialized"next-token"neurons in the Multilayer Perceptron that predict specific tokens. By prompting an LLM like GPT-4 to explain these model internals, we can elucidate attention mechanisms that activate certain next-token neurons. Our analysis identifies attention heads that recognize contexts relevant to predicting a particular token, activating the associated neuron through the residual connection. We focus specifically on heads in earlier layers consistently activating the same next-token neuron across similar prompts. Exploring these differential activation patterns reveals that heads that specialize for distinct linguistic contexts are tied to generating certain tokens. Overall, our method combines neural explanations and probing isolated components to illuminate how attention enables context-dependent, specialized processing in LLMs.
Let's Rectify Step by Step: Improving Aspect-based Sentiment Analysis with Diffusion Models

Aspect-Based Sentiment Analysis (ABSA) stands as a crucial task in predicting the sentiment polarity associated with identified aspects within text. However, a notable challenge in ABSA lies in precisely determining the aspects' boundaries (start and end indices), especially for long ones, due to users' colloquial expressions. We propose DiffusionABSA, a novel diffusion model tailored for ABSA, which extracts the aspects progressively step by step. Particularly, DiffusionABSA gradually adds noise to the aspect terms in the training process, subsequently learning a denoising process that progressively restores these terms in a reverse manner. To estimate the boundaries, we design a denoising neural network enhanced by a syntax-aware temporal attention mechanism to chronologically capture the interplay between aspects and surrounding text. Empirical evaluations conducted on eight benchmark datasets underscore the compelling advantages offered by DiffusionABSA when compared against robust baseline models. Our code is publicly available at https://github.com/Qlb6x/DiffusionABSA.
Enhancing Power Quality Event Classification with AI Transformer Models

Recently, there has been a growing interest in utilizing machine learning for accurate classification of power quality events (PQEs). However, most of these studies are performed assuming an ideal situation, while in reality, we can have measurement noise, DC offset, and variations in the voltage signal's amplitude and frequency. Building on the prior PQE classification works using deep learning, this paper proposes a deep-learning framework that leverages attention-enabled Transformers as a tool to accurately classify PQEs under the aforementioned considerations. The proposed framework can operate directly on the voltage signals with no need for a separate feature extraction or calculation phase. Our results show that the proposed framework outperforms recently proposed learning-based techniques. It can accurately classify PQEs under the aforementioned conditions with an accuracy varying between 99.81%$-$91.43% depending on the signal-to-noise ratio, DC offsets, and variations in the signal amplitude and frequency.
The Impacts of Amazon Deforestation on Pacific Climate

The Impacts of Amazon Deforestation on Pacific Climate Variability in eastern Pacific sea surface temperatures (SSTs) associated with the El Niño Southern Oscillation are known to affect Amazonian precipitation, but to what extent do changing Amazonian vegetation and rainfall impact eastern Pacific SST? The Amazon rainforest is threatened by many factors including climate change and clearing for agricultural reasons. Forest fires and dieback are more likely due to increased frequency and intensity of droughts in the region. It is possible that extensive Amazon deforestation can enhance El Niño conditions by weakening the Walker circulation. Correlations between annual rainfall rates over the Amazon and other atmospheric parameters (global precipitation, surface air temperature, low cloud amount, 500 hPa vertical velocity, surface winds, and 200 hPa winds) over the eastern Pacific indicate strong relationships among these fields. Maps of these correlations (teleconnection maps) reveal that when the Amazon is rainy, SSTs in the central and eastern Pacific are cold, rainfall is suppressed over the central and eastern Pacific, low clouds are prominent over the eastern and southeastern Pacific, and subsidence over the central and eastern Pacific is enhanced. Precipitation in the Amazon is also consistent with a strong Walker circulation (La Niña conditions), manifest as strong correlations with the easterly surface and westerly 200 hPa zonal winds. Coupling between Amazon rainfall and these fields are seen in observations and model data. Correlations were calculated using data from observations, reanalysis data, two models under the Coupled Model Intercomparison Project/Atmospheric Model Intercomparison Project (CMIP5/AMIP), and an AMIP run with the model used in this study, the
Retinotopic Mapping Enhances the Robustness of Convolutional Neural Networks

Foveated vision, a trait shared by many animals, including humans, has not been fully utilized in machine learning applications, despite its significant contributions to biological visual function. This study investigates whether retinotopic mapping, a critical component of foveated vision, can enhance image categorization and localization performance when integrated into deep convolutional neural networks (CNNs). Retinotopic mapping was integrated into the inputs of standard off-the-shelf convolutional neural networks (CNNs), which were then retrained on the ImageNet task. As expected, the logarithmic-polar mapping improved the network's ability to handle arbitrary image zooms and rotations, particularly for isolated objects. Surprisingly, the retinotopically mapped network achieved comparable performance in classification. Furthermore, the network demonstrated improved classification localization when the foveated center of the transform was shifted. This replicates a crucial ability of the human visual system that is absent in typical convolutional neural networks (CNNs). These findings suggest that retinotopic mapping may be fundamental to significant preattentive visual processes.
Super Caldero--Chapoton map for type $A$

One can explicitly compute the generators of a surface cluster algebra either combinatorially, through dimer covers of snake graphs, or homologically, through the CC-map applied to indecomposable modules over the appropriate algebra. Recent work by Musiker, Ovenhouse and Zhang used Penner and Zeitlin's decorated super Teichm{\"u}ller theory to define a super version of the cluster algebra of type $A$ and gave a combinatorial formula to compute the even generators. We extend this theory by giving a homological way of explicitly computing these generators by defining a super CC-map for type $A$.