Miracle-dz/UniFAR-FTUs huggingface dataset:
PerFedMask: Personalized Federated Learning with Optimized Masking Vectors

Recently, various personalized federated learning (FL) algorithms have been proposed to tackle data heterogeneity. To mitigate device heterogeneity, a common approach is to use masking. In this paper, we first show that using random masking can lead to a bias in the obtained solution of the learning model. To this end, we propose a personalized FL algorithm with optimized masking vectors called PerFedMask. In particular, PerFedMask facilitates each device to obtain its optimized masking vector based on its computational capability before training. Finetuning is performed after training. PerFedMask is a generalization of a recently proposed personalized FL algorithm, FedBABU (Oh et al., 2022). PerFedMask can be combined with other FL algorithms including HeteroFL (Diao et al., 2021) and Split-Mix FL (Hong et al., 2022). Results based on CIFAR-10 and CIFAR-100 datasets show that the proposed PerFedMask algorithm provides a higher test accuracy after fine-tuning and lower average number of trainable parameters when compared with six existing state-of-the-art FL algorithms in the literature. The codes are available at https://github.com/MehdiSet/PerFedMask.
DensePure: Understanding Diffusion Models for Adversarial Robustness

Diffusion models have been recently employed to improve certified robustness through the process of denoising. However, the theoretical understanding of why diffusion models are able to improve the certified robustness is still lacking, preventing from further improvement. In this study, we close this gap by analyzing the fundamental properties of diffusion models and establishing the conditions under which they can enhance certified robustness. This deeper understanding allows us to propose a new method DensePure , designed to improve the certified robustness of a pretrained model (i.e. classifier). Given an (adversarial) input, DensePure consists of multiple runs of denoising via the reverse process of the diffusion model (with different random seeds) to get multiple reversed samples, which are then passed through the classifier, followed by majority voting of inferred labels to make the final prediction. This design of using multiple runs of denoising is informed by our theoretical analysis of the conditional distribution of the reversed sample. Specifically, when the data density of a clean sample is high, its conditional density under the reverse process in a diffusion model is also high; thus sampling from the latter conditional distribution can purify the adversarial example and return the corresponding clean sample with a high probability. By using the highest density point in the conditional distribution as the reversed sample, we identify the robust region of a given instance under the diffusion model’s reverse process. We show that this robust region is a union of multiple convex sets, and is potentially much larger than the robust regions identified in previous works. In practice, DensePure can approximate the label of the high density region in the conditional distribution so that it can enhance certified robustness. We conduct extensive experiments to demonstrate the effectiveness of DensePure by evaluating its certified robustness given a standard model via randomized smoothing. We show that DensePure is consistently better than existing methods on ImageNet, with 7% improvement on average. Project
Copy is All You Need

The dominant text generation models compose the output by sequentially selecting words from a fixed vocabulary. In this paper, we formulate text generation as progressively copying text segments (e.g., words or phrases) from an existing text collection. We compute the contextualized representations of meaningful text segments and index them using efficient vector search toolkits. The task of text generation is then decomposed into a series of copy-and-paste operations: at each time step, we seek suitable text spans from the text collection rather than selecting from a standalone vocabulary. Experiments on the standard language modeling benchmark (WikiText-103) show that our approach achieves better generation quality according to both automatic and human evaluations. Besides, its inference efficiency is comparable to token-level autoregressive models thanks to the reduction of decoding steps. We also show that our approach allows for effective domain adaptation by simply switching to domain-specific text collection without extra training. Finally, we observe that our approach attains additional performance gains by simply scaling up to larger text collections, again without further training. \footnote{Our source codes are publicly available at \url{https://github.com/gmftbyGMFTBY/Copyisallyouneed}. }
Is a Caption Worth a Thousand Images? A Study on Representation Learning

The development of CLIP (Radford et al., 2021) has sparked a debate on whether adding language supervision can yield vision models with more transferable representations than traditional image-only methods. Our work studies this question through a carefully controlled comparison of two approaches, in terms of their ability to learn representations that generalize to downstream classification tasks. We find that when the pre-training data meets certain criteria—it is sufficiently large and contains descriptive captions with low variability—-image-only meth-ods do not match CLIP’s performance even when they are trained with more image data. However, contrary to what one might expect, there are practical settings in which these criteria are not met, wherein added supervision through captions is actually detrimental . Motivated by our findings, we devise simple data and algorithmic interventions to improve the transfer performance of CLIP-style models
Single-shot General Hyper-parameter Optimization for Federated Learning

We address the problem of hyper-parameter optimization (HPO) for federated learning ( FL-HPO ). We introduce F ederated Lo ss Su R face A ggregation ( FLoRA ), a general FL-HPO solution framework that can address use cases of tabular data and any Machine Learning (ML) model including gradient boosting training algorithms, SVMs, neural networks, among others and thereby further expands the scope of FL-HPO . FLoRA enables single-shot FL-HPO : identifying a single set of good hyper-parameters that are subsequently used in a single FL training. Thus, it enables FL-HPO solutions with minimal additional communication overhead compared to FL training without HPO. Utilizing standard smoothness assumptions, we theoretically characterize the optimality gap of FLoRA for any convex and non-convex loss functions, which explicitly accounts for the heterogeneous nature of the parties’ local data distributions, a dominant characteristic of FL systems. Our empirical evaluation of FLoRA for multiple FL algorithms on seven OpenML datasets demonstrates signiﬁcant model accuracy improvements over the baselines, and robustness to increasing number of parties involved in FL-HPO training.
Eva: Practical Second-order Optimization with Kronecker-vectorized Approximation

Second-order optimization algorithms exhibit excellent convergence properties for training deep learning models, but often incur significant computation and memory overheads. This can result in lower training efficiency than the first-order counterparts such as stochastic gradient descent (SGD). In this work, we present a memory-and time-efficient second-order algorithm named Eva with two novel techniques: 1) we construct the second-order information with the Kronecker factorization of small stochastic vectors over a mini-batch of training data to reduce memory consumption, and 2) we derive an efficient update formula without explicitly computing the inverse of matrices using the Sherman-Morrison formula. We further provide a theoretical interpretation of Eva from a trust-region optimization point of view to understand how it works. Extensive experimental results on different models and datasets show that Eva reduces the end-to-end training time up to 2 . 05 × and 2 . 42 × compared to first-order SGD and second-order algorithms (K-FAC and Shampoo), respectively.
Task-customized Masked Autoencoder via Mixture of Cluster-conditional Experts

Masked Autoencoder~(MAE) is a prevailing self-supervised learning method that achieves promising results in model pre-training. However, when the various downstream tasks have data distributions different from the pre-training data, the semantically irrelevant pre-training information might result in negative transfer, impeding MAE's scalability. To address this issue, we propose a novel MAE-based pre-training paradigm, Mixture of Cluster-conditional Experts (MoCE), which can be trained once but provides customized pre-training models for diverse downstream tasks. Different from the mixture of experts (MoE), our MoCE trains each expert only with semantically relevant images by using cluster-conditional gates. Thus, each downstream task can be allocated to its customized model pre-trained with data most similar to the downstream data. Experiments on a collection of 11 downstream tasks show that MoCE outperforms the vanilla MAE by 2.45\% on average. It also obtains new state-of-the-art self-supervised learning results on detection and segmentation.
A new characterization of the edge of stability based on a sharpness measure aware of batch gradient distribution

For full-batch gradient descent (GD), it has been empirically shown that the sharpness, the top eigenvalue of the Hessian, increases and then hovers above 2 / (learning rate), and this is called “the edge of stability” phenomenon. However, it is unclear why the sharpness is somewhat larger than 2 / (learning rate) and how this can be extended to general mini-batch stochastic gradient descent (SGD). We propose a new sharpness measure (interaction-aware-sharpness) aware of the interaction between the batch gradient distribution and the loss landscape geometry. This leads to a more refined and general characterization of the edge of stability for SGD. Moreover, based on the analysis of a concentration measure of the batch gradient, we propose a more accurate scaling rule, Linear and Saturation Scaling Rule (LSSR), between batch size and learning rate.
Re-calibrating Feature Attributions for Model Interpretation

The ability to interpret machine learning models is critical for high-stakes applications. Due to its desirable theoretical properties, path integration is a widely used scheme for feature attribution to interpret model predictions. However, the methods implementing this scheme currently rely on absolute attribution scores to eventually provide sensible interpretations. This not only contradicts the premise that the features with larger attribution scores are more relevant to the model prediction, but also conflicts with the theoretical settings for which the desirable properties of the attributions are proven. We address this by devising a method to first compute an appropriate reference for the path integration scheme. This reference further helps in identifying valid interpolation points on a desired integration path. The reference is computed in a gradient ascending direction on the model’s loss surface, while the interpolations are performed by analyzing the model gradients and variations between the reference and the input. The eventual integration is effectively performed along a non-linear path. Our scheme can be incorporated into the existing integral-based attribution methods. We also devise an effective sampling and integration procedure that enables employing our scheme with multi-reference path integration efficiently. We achieve a marked performance boost for a range of integral-based attribution methods on both local and global evaluation metrics by enhancing them with our scheme. Our extensive results also show improved sensitivity, sanity preservation and model robustness with the proposed re-calibration of the attribution techniques with our method. 1
Replay Memory as An Empirical MDP: Combining Conservative Estimation with Experience Replay

Experience replay, which stores transitions in a replay memory for repeated use, plays an important role of improving sample efficiency in reinforcement learning. Existing techniques such as reweighted sampling, episodic learning and reverse sweep update process the information in the replay memory to make experience replay more efficient. In this work, we further exploit the information in the replay memory by treating it as an empirical Replay Memory MDP (RM-MDP). By solving it with dynamic programming, we learn a conservative value estimate that only considers transitions observed in the replay memory. Both value and policy regularizers based on this conservative estimate are developed and integrated with model-free learning algorithms. We design the memory density metric to measure the quality of RM-MDP. Our empirical studies quantitatively find a strong correlation between performance improvement and memory density. Our method combines Conservative Estimation with Experience Replay (CEER), improving sample efficiency by a large margin, especially when the memory density is high. Even when the memory density is low, such a conservative estimate can still help to avoid suicidal actions and thereby improve performance.
Reviewing the properties and prototype structure of metonymy

Building on previous research (Barcelona, 2002, 2003), the author carefully discusses and takes a stand on the issues he finds problematic in the standard cognitive linguistic notion of metonymy. These include, among others, the status of metonymy as a mapping, as a stand-for relationship, as a type of activation, and as a type of “domain highlighting”; prototype-based vs. unitary definitions of metonymy; and the distinction of metonymy from metaphor, “modulation”, “facets”, and active zones. On the basis of his discussion of those issues, he puts forward a technical unitary (“schematic”) notion of metonymy, which he claims to coexist with three tightly interrelated prototype-based notions of metonymy that he proposes: “purely schematic”, “typical”, and “prototypical” metonymies, all parts of a continuum.
Invited Review: Cepheid Variables as Extra-Galactic Distance Indicators

The role of cepheid variables in establishing the inner distance scale to nearby galaxies is discussed. Emphasis is placed on the necessity for broad wavelength coverage in attempting to account for metalliclty differences and reddening internal to the parent galaxies. In addition linear detectors are essential in minimizing the effects of any unresolved background contibution to the photometry. Recent infrared observations of Cepheids in Local Group galaxies are surveyed and all published data on extragalactic Cepheids are presented for convenient access.
Impaired Chylomicron Assembly Modifies Hepatic Metabolism Through Bile Acid–Dependent and Transmissible Microbial Adaptations

The mechanisms by which alterations in intestinal bile acid (BA) metabolism improve systemic glucose tolerance and hepatic metabolic homeostasis are incompletely understood. We examined metabolic adaptations in mice with conditional intestinal deletion of the abetalipoproteinemia (ABL) gene microsomal triglyceride transfer protein (Mttp‐IKO), which blocks chylomicron assembly and impairs intestinal lipid transport. Mttp‐IKO mice exhibit improved hepatic glucose metabolism and augmented insulin signaling, without weight loss. These adaptations included decreased BA excretion, increased pool size, altered BA composition, and increased fibroblast growth factor 15 production. Mttp‐IKO mice absorb fructose normally but are protected against dietary fructose‐induced hepatic steatosis, without weight loss or changes in energy expenditure. In addition, Mttp‐IKO mice exhibit altered cecal microbial communities, both at baseline and following fructose feeding, including increased abundance of Bacteroides and Lactobacillus genera. Transplantation of cecal microbiota from chow‐fed Mttp‐IKO mice into antibiotic‐treated wild‐type recipients conferred transmissible protection against fructose‐induced hepatic steatosis in association with a bloom in Akkermansia and increased Clostridium XIVa genera, whose abundance was positively correlated with fecal coprostanol and total neutral sterol excretion in recipient mice. However, antibiotic‐treated Mttp‐IKO mice were still protected against fructose‐induced hepatic steatosis, suggesting that changes in microbiota are not required for this phenotype. Nevertheless, we found increased abundance of fecal Akkermansia from two adult ABL subjects with MTTP mutations compared to their heterozygous parents and within the range noted in six healthy control subjects. Furthermore, Akkermansia abundance across all subjects was positively correlated with fecal coprostanol excretion. Conclusion: The findings collectively suggest multiple adaptive pathways of metabolic regulation following blocked chylomicron assembly, including shifts in BA signaling and altered microbial composition that confer a transmissible phenotype.
Association between restless legs syndrome and other movement disorders

Objective This review focuses on the possible association between restless legs syndrome (RLS) and movement disorders, including Parkinson disease (PD), other parkinsonian syndromes, essential tremor, choreic and dystonic syndromes, Tourette syndrome, and heredodegenerative ataxias. Methods Review of PubMed from 1966 to September 2018 and identification of references of interest for the topic. A meta-analysis of eligible studies on the frequency of RLS in patients with PD and controls using Meta-DiSc1.1. 1 software and using the PRISMA guidelines was performed. Results and conclusions Although there are substantial clinical, neuroimaging, neuropathologic, and genetic differences between RLS and PD, many reports describe a higher than expected prevalence of RLS in patients with PD, when compared with the general population or with matched control groups; several studies have also suggested that RLS could be an early clinical feature of PD. RLS symptoms are frequent in multiple system atrophy, essential tremor, Tourette syndrome, Friedreich ataxia, and spinocerebellar ataxia type 3 as well. Finally, possible genetic links between PD and RLS (the presence of allele 2 of the complex microsatellite repeat Rep1 within the α-synuclein gene promoter) and between Tourette syndrome and RLS (several variants in the BTBD9 gene) have been reported in 2 case–control association studies, although these data, based on preliminary data with small sample sizes, need to be replicated in further studies.
Formation of the Ionospheric G-condition Following the

17 A total solar eclipse (TSE) crossed the continental US (CONUS) from west to east on 18 21 August 2017. Ionosondes located under the eclipse totality at Lusk (Wyoming) and 19 Boulder (Colorado) observed the ionospheric G-condition 20 minutes after totality. The 20 Millstone Hill mid-latitude incoherent scatter radar recorded an anomalous low altitude 21 F2 peak during the recovery phase of the eclipse, which can be attributed to an iono22 spheric G-condition. We perform WACCM-X simulations to investigate the physical pro23 cesses that drive the ionospheric G-condition. Specifically, we conduct a diagnostic anal24 ysis of the simulated atomic oxygen ion continuity equation to examine the source of the 25 G-condition. Results indicate that (a) perturbations in plasma density of E and F1-layers 26 closely follow the TSE occultation, whereas the F2-layer density depletion lags the oc27 cultation by 20-minutes; (b) this delay increases with altitude and is caused by slower 28 ion recombination in the diffusion-dominated F2-layer; (c) the delay creates a time pe29 riod during eclipse recovery when plasma density of the F1-layer is larger than that of 30 the F2-layer, which manifests as the G-condition. The simulation study showed an in31 crease in the strength of the ionospheric G-condition with latitude, which disagrees with 32 previously reported studies. 33 Plain Language Summary 34 Solar radiation is the primary source of the ionosphere. The eclipse-driven sudden 35 disappearance of solar irradiance provides an opportunity to study how the ionosphere 36 behaves under this controlled environment. Ground-based observations during the 21 37 August 2017 Great American Eclipse (GAE) provide evidence of the existence of an iono38 spheric G-condition, where the plasma density of the ionospheric F1-layer exceeds that 39 of the F2-layer. This paper provides the first explanation of the formation mechanism 40 of the ionospheric G-condition following a total solar eclipse. We use simulations from 41 a physics-based model to infer the physics behind the ionospheric G-condition. The com42 bination of observations and model predictions provides insight into the formation mech43 anism of these unusual conditions following the 2017 GAE. 44
Effects of Streptococcus thermophilus Fermentation on the Flavors and Antioxidant Properties of Barley Juice

Lactic fermentation can improve the metabolic profile and functionality of juice, which is beneficial to human health. This study aimed to investigate the effect of Streptococcus thermophilus fermentation on the metabolic profiles and antioxidant activity of barley juice. The results demonstrated that S. thermophilus 7G10 dramatically increased the total titratable acidity and total phenolic and flavonoid contents in the barley juice after 24 h of fermentation. Only sixteen volatile compounds were detected in the fermented barley juice, including six acids, four ketones, three alcohols, and one aldehyde. In addition, based on non-targeted metabolomics, 30 important differential metabolites were screened among the 1460 non-volatile compounds. Notably, the barley juice fermented with S. thermophilus 7G10 had increased free radical (ABTS, DPPH, and O2−) scavenging activities. Furthermore, sensory evaluation showed that the barley juice fermented with S. thermophilus 7G10 was most attractive to consumers. These results show that LAB fermentation promotes the formation of volatile compounds and potentially enhances the antioxidant properties of barley juice.
BTT_A_290286 47..55

An emerging theory regarding the potentially autoimmune nature of painful bladder syndrome/interstitial cystitis (PBS/IC) had led to several studies being conducted to assess the possible therapeutic effect of immunotherapeutic options for PBS/IC. This review presents the available evidence regarding the potential autoimmunity-based pathogenesis of PBS/IC and focuses on a main representative of the immunotherapeutic modalities for PBS/IC, aiming to summarize, evaluate, and present available data regarding the potential therapeutic role of monoclonal antibodies for PBS/IC patients. A non-systematic narrative and interpretative literature review was performed. The monoclonal antibodies included in the review were the anti-tumor necrosis factor-α (anti-TNF-α) agents adalimumab, which showed no difference compared to placebo, and certolizumab pegol, which showed statistically important differences in all outcome measures compared to placebo at the 18-week follow-up visit. Anti-nerve growth factor (anti-NGF) agents were also reviewed, including tanezumab, which showed both positive and negative efficacy results compared to placebo, and fulranumab, the study of which was discontinued owing to adverse events. In summary, monoclonal antibody therapy remains to be further researched in order for it to be proposed as a promising future treatment option for PBS/IC.