Miracle-dz/UniFAR-FTUs huggingface dataset:
Turning the Curse of Heterogeneity in Federated Learning into a Blessing for Out-of-Distribution Detection

Deep neural networks have witnessed huge successes in many challenging prediction tasks and yet they often suffer from out-of-distribution (OoD) samples, misclassifying them with high conﬁdence. Recent advances show promising OoD detection performance for centralized training, and however, OoD detection in federated learning (FL) is largely overlooked, even though many security sensitive applications such as autonomous driving and voice recognition authorization are commonly trained using FL for data privacy concerns. The main challenge that prevents previous state-of-the-art OoD detection methods from being incorporated to FL is that they require large amount of real OoD samples. However, in real-world scenarios, such large-scale OoD training data can be costly or even infeasible to obtain, especially for resource-limited local devices. On the other hand, a notorious challenge in FL is data heterogeneity where each client collects non-identically and independently distributed (non-iid) data. We propose to take advantage of such heterogeneity and turn the curse into a blessing that facilitates OoD detection in FL. The key is that for each client, non-iid data from other clients (unseen external classes) can serve as an alternative to real OoD samples. Speciﬁcally, we propose a novel Federated Out-of-Distribution Synthesizer (F OSTER ), which learns a class-conditional generator to synthesize virtual external-class OoD samples, and maintains data conﬁdentiality and communication efﬁciency required by FL. Experimental results show that our method outperforms the state-of-the-art for
How Does Semi-supervised Learning with Pseudo-labelers Work? A Case Study

Semi-supervised learning is a popular machine learning paradigm that utilizes a large amount of unlabeled data as well as a small amount of labeled data to facilitate learning tasks. While semi-supervised learning has achieved great success in training neural networks, its theoretical understanding remains largely open. In this paper, we aim to theoretically understand a semi-supervised learning approach based on pre-training and linear probing. In particular, the semi-supervised learning approach we consider first trains a two-layer neural network based on the unlabeled data with the help of pseudo-labelers. Then it linearly probes the pretrained network on a small amount of labeled data. We prove that, under a certain toy data generation model and two-layer convolutional neural network, the semisupervised learning approach can achieve nearly zero test loss, while a neural network directly trained by supervised learning on the same amount of labeled data can only achieve constant test loss. Through this case study, we demonstrate a separation between semi-supervised learning and supervised learning in terms of test loss provided the same amount of labeled data.
Targeted Hyperparameter Optimization with Lexicographic Preferences Over Multiple Objectives

Motivated by various practical applications, we propose a novel and general formulation of targeted multi-objective hyperparameter optimization. Our formulation allows a clear specification of an automatable optimization goal using lexicographic preference over multiple objectives. We then propose a randomized directed search method named LexiFlow to solve this problem. We demonstrate the strong empirical performance of the proposed algorithm in multiple hyperparameter optimization tasks.
Graph Domain Adaptation via Theory-Grounded Spectral Regularization

Transfer learning on graphs drawn from varied distributions (domains) is in great demand across many applications. Emerging methods attempt to learn domaininvariant representations using graph neural networks (GNNs), yet the empirical performances vary and the theoretical foundation is limited. This paper aims at designing theory-grounded algorithms for graph domain adaptation (GDA). (i) As the first attempt, we derive a model-based GDA bound closely related to two GNN spectral properties: spectral smoothness (SS) and maximum frequency response (MFR). This is achieved by cross-pollinating between the OT-based (optimal transport) DA and graph filter theories. (ii) Inspired by the theoretical results, we propose algorithms regularizing spectral properties of SS and MFR to improve GNN transferability. We further extend the GDA theory into the more challenging scenario of conditional shift, where spectral regularization still applies. (iii) More importantly, our analyses of the theory reveal which regularization would improve performance of what transfer learning scenario, (iv) with numerical agreement with extensive real-world experiments: SS and MFR regularizations bring more benefits to the scenarios of node transfer and link transfer, respectively. In a nutshell, our study paves the way toward explicitly constructing and training GNNs that can capture more transferable representations across graph domains. Codes are released at https://github.com/Shen-Lab/GDA-SpecReg.
Learning to Solve Constraint Satisfaction Problems with Recurrent Transformer

Constraint satisfaction problems (CSPs) are about finding values of variables that satisfy the given constraints. We show that Transformer extended with recurrence is a viable approach to learning to solve CSPs in an end-to-end manner, having clear advantages over state-of-the-art methods such as Graph Neural Networks, SATNet, and some neuro-symbolic models. With the ability of Transformer to handle visual input, the proposed Recurrent Transformer can straightforwardly be applied to visual constraint reasoning problems while successfully addressing the symbol grounding problem. We also show how to leverage deductive knowledge of discrete constraints in the Transformer's inductive learning to achieve sample-efficient learning and semi-supervised learning for CSPs.
α-Avoidance

When substitutions and bindings interact, there is a risk of undesired side effects if the substitution is applied naïvely. The λ -calculus captures this phenomenon concretely, as β -reduction may require the renaming of bound variables to avoid variable capture. In this paper we introduce α -paths as an estimation for α -avoidance, roughly expressing that α -conversions are not required to prevent variable capture. These paths provide a novel method to analyse and predict the potential need for α in different calculi. In particular, we show how α -path characterises α -avoidance for several sub-calculi of the λ -calculus like (i) developments, (ii) affine/linear λ -calculi, (iii) the weak λ -calculus, (iv) µ -unfolding and (iv) finally the safe λ -calculus. Furthermore, we study the unavoidability of α -conversions in untyped and simply-typed λ -calculi and prove undecidability of the need of α -conversions for (leftmost–outermost reductions) in the untyped λ -calculus. To ease the work with α -paths, we have implemented the method and the tool is publicly available.
Stochastic No-regret Learning for General Games with Variance Reduction

We show that a stochastic version of optimistic mirror descent (OMD), a variant of mirror descent with recency bias, converges fast in general games. More specifically, with our algorithm, the individual regret of each player vanishes at a speed of O (1 /T 3 / 4 ) and the sum of all players’ regret vanishes at a speed of O (1 /T ) , which is an improvement upon the O (1 / √ T ) convergence rate of prior stochastic algorithms, where T is the number of interaction rounds. Due to the advantage of stochastic methods in the computational cost
Bit-Pruning: A Sparse Multiplication-Less Dot-Product

Dot-product is a central building block in neural networks. However, multiplication (mult) in dot-product consumes intensive energy and space costs that challenge deployment on resource-constrained edge devices. In this study, we realize energy-efficient neural networks by exploiting a mult-less, sparse dotproduct. We first reformulate a dot-product between an integer weight and activation into an equivalent operation comprised of additions followed by bit-shifts (add-shift-add). In this formulation, the number of add operations equals the number of bits of the integer weight in binary format. Leveraging this observation, we propose Bit-Pruning, which removes unnecessary bits in each weight value during training to reduce the energy consumption of add-shift-add. Bit-Pruning can be seen as soft Weight-Pruning as it prunes bits, not the whole weight element. In extensive experiments, we demonstrate that sparse multless networks trained with Bit-Pruning show a better accuracy-energy trade-off than sparse mult networks trained with Weight-Pruning. (Code is available at https://github.com/DensoITLab/bitprune)
Fisher-Legendre (FishLeg) optimization of deep neural networks

Incorporating second-order gradient information (curvature) into optimization can dramatically reduce the number of iterations required to train machine learning models. In natural gradient descent, such information comes from the Fisher information matrix which yields a number of desirable properties. As exact natural gradient updates are intractable for large models, successful methods such as KFAC and sequels approximate the Fisher in a structured form that can easily be inverted. However, this requires model/layer-specific tensor algebra and certain approximations that are often difficult to justify. Here, we use ideas from Legendre-Fenchel duality to learn a direct and efficiently evaluated model for the product of the inverse Fisher with any vector, in an online manner, leading to natural gradient steps that get more accurate over time despite noisy gradients. We prove that the resulting “Fisher-Legendre” (FishLeg) optimizer converges to a (global) minimum of non-convex functions satisfying the PL condition, which applies in particular to deep linear networks. On standard auto-encoder benchmarks, we show empirically that FishLeg outperforms standard first-order optimization methods, and performs on par with or better than other second-order methods, especially when using small batches. Thanks to its generality, we expect our approach to facilitate the handling of a variety neural network layers in future work.
Discovering Generalizable Multi-agent Coordination Skills from Multi-task Offline Data

Cooperative multi-agent reinforcement learning (MARL) faces the challenge of adapting to multiple tasks with varying agents and targets. Previous multi-task MARL approaches require costly interactions to simultaneously learn or fine-tune policies in different tasks. However, the situation that an agent should generalize to multiple tasks with only offline data from limited tasks is more in line with the needs of real-world applications. Since offline multi-task data contains a variety of behaviors, an effective data-driven approach is to extract informative latent variables that can represent universal skills for realizing coordination across tasks. In this paper, we propose a novel Offline MARL algorithm to Discover coordInation Skills (ODIS) from multi-task data. ODIS first extracts task-invariant coordination skills from offline multi-task data and learns to delineate different agent behaviors with the discovered coordination skills. Then we train a coordination policy to choose optimal coordination skills with the centralized training and decentralized execution paradigm. We further demonstrate that the discovered coordination skills can assign effective coordinative behaviors, thus significantly enhancing generalization to unseen tasks. Empirical results in cooperative MARL benchmarks, including the StarCraft multi-agent challenge, show that ODIS obtains superior performance in a wide range of tasks only with offline data from limited sources.
Alternative approaches to electronic damage by ion‐beam irradiation: Exciton models

The paper briefly describes the main features of the damage produced by swift heavy ion (SHI) irradiation. After a short revision of the widely used thermal spike concept, it focuses on cumulative mechanisms of track formation which are alternative to those based on lattice melting (thermal spike models). These cumulative mechanisms rely on the production of point defects around the ion trajectory, and their accumulation up to a final lattice collapse or amorphization. As to the formation of point defects, the paper considers those mechanisms relying on direct local conversion of the excitation energy into atomic displacements (exciton models). A particular attention is given to processes based on the non‐radiative recombination of excitons that have become self‐trapped as a consequence of a strong electron–phonon interaction (STEs). These mechanisms, although operative under purely ionizing radiation in some dielectric materials, have been rarely invoked, so far, to discuss SHI damage. They are discussed in this paper together with relevant examples to materials such as Cu3N, alkali halides, SiO2, and LiNbO3.
Multi-domain image generation and translation with identifiability guarantees

Multi-domain image generation and unpaired image-to-to-image translation are two important and related computer vision problems. The common technique for the two tasks is the learning of a joint distribution from multiple marginal distributions. However, it is well known that there can be infinitely many joint distributions that can derive the same marginals. Hence, it is necessary to formulate suitable constraints to address this highly ill-posed problem. Inspired by the recent advances in nonlinear Independent Component Analysis (ICA) theory, we propose a new method to learn the joint distribution from the marginals by enforcing a specific type of minimal change across domains. We report one of the first results connecting multi-domain generative models to identifiability and shows why identifiability is essential and how to achieve it theoretically and practically. We apply our method to five multi-domain image generation and six image-toimage translation tasks. The superior performance of our model supports our theory and demonstrates the effectiveness of our method. The training code are available at https://github.com/Mid-Push/i-stylegan.
Neural ePDOs: Spatially Adaptive Equivariant Partial Differential Operator Based Networks

Endowing deep learning models with symmetry priors can lead to a considerable performance improvement. As an interesting bridge between physics and deep learning, the equivariant partial differential operators (PDOs) have drawn much researchers’ attention recently. However, to ensure the PDOs translation equivariance, previous works have to require coefficient matrices to be constant and spatially shared for their linearity, which could lead to the sub-optimal feature learning at each position. In this work, we propose a novel nonlinear PDOs scheme that is both spatially adaptive and translation equivariant. The coefficient matrices are obtained by local features through a generator rather than spatially shared. Besides, we establish a new theory on incorporating more equivariance like rotations for such PDOs. Based on our theoretical results, we efficiently implement the generator with an equivariant multilayer perceptron (EMLP). As such equivariant PDOs are generated by neural networks, we call them Neural ePDOs. In experiments, we show that our method can significantly improve previous works with smaller model size in various datasets. Especially, we achieve the state-ofthe-art performance on the MNIST-rot dataset with only tenth of parameters of the previous best model.
Causality Compensated Attention for Contextual Biased Visual Recognition

Visual attention does not always capture the essential object representation desired for robust predictions. Attention modules tend to underline not only the target object but also the common co-occurring context that the module thinks helpful in the training. The problem is rooted in the confounding effect of the context leading to incorrect causalities between objects and predictions, which is further exacerbated by visual attention. In this paper, to learn causal object features robust for contextual bias, we propose a novel attention module named Interventional Dual Attention (IDA) for visual recognition. Specifically, IDA adopts two attention layers with multiple sampling intervention, which compensates the attention against the confounder context. Note that our method is model-agnostic and thus can be implemented on various backbones. Extensive experiments show our model obtains significant improvements in classification and detection with lower computation. In particular, we achieve the state-of-the-art results in multi-label classification on MS-COCO and PASCAL-VOC.
Out-of-Distribution Detection based on In-Distribution Data Patterns Memorization with Modern Hopfield Energy

Out-of-Distribution (OOD) detection is essential for safety-critical applications of deep neural networks. OOD detection is challenging since DNN models may produce very high logits value even for OOD samples. Hence, it is of great difficulty to discriminate OOD data by directly adopting Softmax on output logits as the confidence score. Differently, we detect the OOD sample with Hopfield energy in a store-then-compare paradigm. In more detail, penultimate layer outputs on the training set are considered as the representations of in-distribution (ID) data. Thus they can be transformed into stored patterns that serve as anchors to measure the discrepancy of unseen data for OOD detection. Starting from the energy function defined in Modern Hopfield Network for the discrepancy score calculation, we derive a simplified version SHE with theoretical analysis. In SHE, we utilize only one stored pattern to present each class, and these patterns can be obtained by simply averaging the penultimate layer outputs of training samples within this class. SHE has the advantages of hyperparameterfree and high computational efficiency. The evaluations of nine widely-used OOD datasets show the promising performance of such a simple yet effective approach and its superiority over State-of-the-Art models. Code is available at https://github.com/zjs975584714/SHE ood detection.
Towards Interpretable Deep Reinforcement Learning with Human-Friendly Prototypes

Despite recent success of deep learning models in research settings, their application in sensitive domains remains limited because of their opaque decision-making processes. Taking to this challenge, people have proposed various eXplainable AI (XAI) techniques designed to calibrate trust and understandability of black-box models, with the vast majority of work focused on supervised learning. Here, we focus on making an “interpretable-by-design” deep reinforcement learning agent which is forced to use human-friendly prototypes in its decisions, thus making its reasoning process clear. Our proposed method, dubbed Prototype-Wrapper Network (PW-Net), wraps around any neural agent backbone, and results indicate that it does not worsen performance relative to black-box models. Most importantly, we found in a user study that PW-Nets supported better trust calibration and task performance relative to standard interpretability approaches and black-boxes.
Winning Both the Accuracy of Floating Point Activation and the Simplicity of Integer Arithmetic

Even though floating point (FP) numbers have been adopted as a de facto standard data format for deep learning computing, the complexity of FP arithmetic impedes a broader deployment of Deep Neural Networks (DNNs). Recent works such as quantization have attempted to replace the FP matrix multiplication (MatMul) of DNNs with simple integer MatMul by transforming the datatypes of both weights and activations into integers. Unfortunately, unlike weight values that are static, it is challenging to represent dynamic activations with integers. In this paper, to simultaneously achieve the accuracy of FP activation and the simplicity of integer arithmetic, we present a method for replacing FP arithmetic with integer one without changing FP activations in the storage format while weights are quantized. The proposed method pre-aligns the significands of FP activations just ahead of the MatMul on-the-fly so that the aligned significands (integers) can be used for the computation. Inspired by an observation that conventional FP arithmetic does not produce precise results due to rounding, we demonstrate that our proposed integer arithmetic-based scheme can produce the same level of errors as that of the FP arithmetic in case DNNs use FP activations and quantized weights. Experimental results show that the hardware based on the proposed scheme shows significant improvement over FP arithmetic-based designs in terms of energy efficiency and throughput-per-area while maintaining a similar level of accuracy.
Resveratrol Treatment Ameliorates Leptin Resistance and Adiposity Programed by the Combined Effect of Maternal and Post-Weaning High-Fat Diet.

SCOPE Prenatal high-fat (HF) and postnatal HF diet are both associated with obesity and metabolic disturbances in adults. Leptin resistance induced by obesity limits its biological effects. The anti-obesity mechanism of resveratrol in visceral adiposity is investigated here. METHODS AND RESULTS During mating and lactation, Sprague-Dawley dams are fed either control or a HF diet. Subsequently, the offspring are fed chow or an HF diet. A fifth group that received maternal/postnatal HF diet and resveratrol after weaning (HHR) is used to study the effects of resveratrol treatment. Resveratrol treatment alleviates adiposity programed by maternal and postnatal HF diet by decreasing feed intake or inducing metabolic changes. Resveratrol treatment is also found to ameliorate the decrease in SIRT1 abundance observed in retroperitoneal adipose tissue, programed by maternal and postnatal HF diet. Moreover, resveratrol therapy decreases plasma leptin level and increases leptin receptor expression in retroperitoneal adipose tissue through DNA methylation modification. CONCLUSION These results suggest that resveratrol can alleviate peripheral leptin resistance programed by the combined effect of prenatal and postnatal HF diet through epigenetic regulation of genes coding leptin and its receptor. It provides insights into a novel mechanism explaining the beneficial effects of resveratrol in obesity management.
LPCAT2 Regulates CD14 Expression During Macrophage Inflammatory Response to E. coli O111:B4

LPCAT2 is a lipid-modifying enzyme that co-localises in lipid rafts with TLR4 and regulates macrophage inflammatory response; however, its effect on TLR4 co-receptor–CD14 is unknown. RAW264.7 cells, a common murine macrophage experimental model, were used to study the effect of LPCAT2 on CD14 expression. The expression of LPCAT2 in RAW264.7 cells was silenced using RNA interference and treated with 100ng/ml of various lipopolysaccharide chemotypes. We found that CD14 expression induced by smooth lipopolysaccharide was significantly decreased (p < 0.05) in RAW264.7 macrophages with LPCAT2 silenced. This study suggests that LPCAT2 regulates CD14 gene and protein expression. This implies that LPCAT2 can regulate CD14-dependent cellular activities.