Miracle-dz/UniFAR-FTUs huggingface dataset:
Model Compression and Efficient Inference for Large Language Models: A Survey

Transformer based large language models have achieved tremendous success. However, the significant memory and computational costs incurred during the inference process make it challenging to deploy large models on resource-constrained devices. In this paper, we investigate compression and efficient inference methods for large language models from an algorithmic perspective. Regarding taxonomy, similar to smaller models, compression and acceleration algorithms for large language models can still be categorized into quantization, pruning, distillation, compact architecture design, dynamic networks. However, Large language models have two prominent characteristics compared to smaller models: (1) Most of compression algorithms require finetuning or even retraining the model after compression. The most notable aspect of large models is the very high cost associated with model finetuning or training. Therefore, many algorithms for large models, such as quantization and pruning, start to explore tuning-free algorithms. (2) Large models emphasize versatility and generalization rather than performance on a single task. Hence, many algorithms, such as knowledge distillation, focus on how to preserving their versatility and generalization after compression. Since these two characteristics were not very pronounced in early large models, we further distinguish large language models into medium models and ``real'' large models. Additionally, we also provide an introduction to some mature frameworks for efficient inference of large models, which can support basic compression or acceleration algorithms, greatly facilitating model deployment for users.
Unlocking the Potential of Transformers in Time Series Forecasting with Sharpness-Aware Minimization and Channel-Wise Attention

Transformer-based architectures achieved breakthrough performance in natural language processing and computer vision, yet they remain inferior to simpler linear baselines in multivariate long-term forecasting. To better understand this phenomenon, we start by studying a toy linear forecasting problem for which we show that transformers are incapable of converging to their true solution despite their high expressive power. We further identify the attention of transformers as being responsible for this low generalization capacity. Building upon this insight, we propose a shallow lightweight transformer model that successfully escapes bad local minima when optimized with sharpness-aware optimization. We empirically demonstrate that this result extends to all commonly used real-world multivariate time series datasets. In particular, SAMformer surpasses the current state-of-the-art model TSMixer by 14.33% on average, while having ~4 times fewer parameters. The code is available at https://github.com/romilbert/samformer.
EventF2S: Asynchronous and Sparse Spiking AER Framework using Neuromorphic-Friendly Algorithm

Bio-inspired Address Event Representation (AER) sensors have attracted significant popularity owing to their low power consumption, high sparsity, and high temporal resolution. Spiking Neural Network (SNN) has become the inherent choice for AER data processing. However, the integration of the AER-SNN paradigm has not adequately explored asynchronous processing, neuromorphic compatibility, and sparse spiking, which are the key requirements of resource-constrained applications. To address this gap, we introduce a brain-inspired AER-SNN object recognition solution, which includes a data encoder integrated with a First-To-Spike recognition network. Being fascinated by the functionality of neurons in the visual cortex, we designed the solution to be asynchronous and compatible with neuromorphic hardware. Furthermore, we have adapted the principle of denoising and First-To-Spike coding to achieve optimal spike signaling, significantly reducing computation costs. Experimental evaluation has demonstrated that the proposed method incurs significantly less computation cost to achieve state-of-the-art competitive accuracy. Overall, the proposed solution offers an asynchronous and cost-effective AER recognition system that harnesses the full potential of AER sensors.
Representation Learning Using a Single Forward Pass

We propose a neuroscience-inspired Solo Pass Embedded Learning Algorithm (SPELA). SPELA is a prime candidate for training and inference applications in Edge AI devices. At the same time, SPELA can optimally cater to the need for a framework to study perceptual representation learning and formation. SPELA has distinctive features such as neural priors (in the form of embedded vectors), no weight transport, no update locking of weights, complete local Hebbian learning, single forward pass with no storage of activations, and single weight update per sample. Juxtaposed with traditional approaches, SPELA operates without the need for backpropagation. We show that our algorithm can perform nonlinear classification on a noisy boolean operation dataset. Additionally, we exhibit high performance using SPELA across MNIST, KMNIST, and Fashion MNIST. Lastly, we show the few-shot and 1-epoch learning capabilities of SPELA on MNIST, KMNIST, and Fashion MNIST, where it consistently outperforms backpropagation.
Diffusion Model with Cross Attention as an Inductive Bias for Disentanglement

Disentangled representation learning strives to extract the intrinsic factors within observed data. Factorizing these representations in an unsupervised manner is notably challenging and usually requires tailored loss functions or specific structural designs. In this paper, we introduce a new perspective and framework, demonstrating that diffusion models with cross-attention can serve as a powerful inductive bias to facilitate the learning of disentangled representations. We propose to encode an image to a set of concept tokens and treat them as the condition of the latent diffusion for image reconstruction, where cross-attention over the concept tokens is used to bridge the interaction between the encoder and diffusion. Without any additional regularization, this framework achieves superior disentanglement performance on the benchmark datasets, surpassing all previous methods with intricate designs. We have conducted comprehensive ablation studies and visualization analysis, shedding light on the functioning of this model. This is the first work to reveal the potent disentanglement capability of diffusion models with cross-attention, requiring no complex designs. We anticipate that our findings will inspire more investigation on exploring diffusion for disentangled representation learning towards more sophisticated data analysis and understanding.
Efficient $\Phi$-Regret Minimization with Low-Degree Swap Deviations in Extensive-Form Games

Recent breakthrough results by Dagan, Daskalakis, Fishelson and Golowich [2023] and Peng and Rubinstein [2023] established an efficient algorithm attaining at most $\epsilon$ swap regret over extensive-form strategy spaces of dimension $N$ in $N^{\tilde O(1/\epsilon)}$ rounds. On the other extreme, Farina and Pipis [2023] developed an efficient algorithm for minimizing the weaker notion of linear-swap regret in $\mathsf{poly}(N)/\epsilon^2$ rounds. In this paper, we take a step toward bridging the gap between those two results. We introduce the set of $k$-mediator deviations, which generalize the untimed communication deviations recently introduced by Zhang, Farina and Sandholm [2024] to the case of having multiple mediators. We develop parameterized algorithms for minimizing the regret with respect to this set of deviations in $N^{O(k)}/\epsilon^2$ rounds. This closes the gap in the sense that $k=1$ recovers linear swap regret, while $k=N$ recovers swap regret. Moreover, by relating $k$-mediator deviations to low-degree polynomials, we show that regret minimization against degree-$k$ polynomial swap deviations is achievable in $N^{O(kd)^3}/\epsilon^2$ rounds, where $d$ is the depth of the game, assuming constant branching factor. For a fixed degree $k$, this is polynomial for Bayesian games and quasipolynomial more broadly when $d = \mathsf{polylog} N$ -- the usual balancedness assumption on the game tree.
One to rule them all? Assessing the performance of Forest Europe’s biodiversity indicators against multitaxonomic data

Most broad-scale forest biodiversity indicators are based on data from national forest inventories and are used to assess the state of biodiversity through several regional initiatives and reporting. Although valuable, these indicators are essentially indirect and evaluate habitat quantity and quality rather than biodiversity per se. Besides, most of these indicators are applicable at regional or national scales, while their use at a more local level is difficult. Therefore, their link to biodiversity may be weak, which decreases their usefulness for decision-making. For several decades, Forest Europe indicators assessed the state of European forests, in particular its biodiversity. However, no extensive study has been conducted to date to assess the performance of these indicators against multitaxonomic data. We hypothesized that – as implied by the reporting process – no single biodiversity indicator from Forest Europe can represent overall forest biodiversity, but that several – eventually combined – indicators would reflect habitat quality for at least some taxa in a comprehensive way. We tested the set of indicators proposed by Forest Europe against the species richness of six taxonomic and functional groups (tracheophytes, epixylic and epiphytic bryophytes, birds, saproxylic beetles, saproxylic non-lichenized fungi and epixylic and epiphytic lichenized fungi) across several hundreds of plots over Europe. We showed that, while some indicators perform relatively well across groups (e.g. deadwood volume), no single indicator represented all biodiversity at once, and that a combination of several indicators performed better. Surprisingly, some indicators showed weak links with the biodiversity of the six taxonomic and functional groups. Forest Europe indicators were chosen for their availability and ease of understanding for most people. However, our analyses showed that there are still gaps in the monitoring framework, and that surveying certain taxa along with stand structure is necessary to support policymaking and tackle forest biodiversity loss at the large scale. Impact statement Biodiversity indicators used to assess the state of Europe’s forests perform unequally; a combination of indicators is more informative
Sapap3 knockout mice show threat bias under conflict during platform mediated avoidance task: implications for obsessive compulsive disorder

Obsessive compulsive disorder (OCD) typically involves cycling between symptoms of intrusive aversive thoughts (obsessions) and repetitive rituals aimed at avoiding these aversive outcomes (compulsions), which interferes with patients’ engagement with other important aspects of their lives. This cycling relationship between obsessions and compulsions highlights a potential role of impaired threat processing and avoidance behavior in OCD symptoms. The most effective behavioral therapy for OCD, exposure with response prevention (ERP), aims to break this cycle. However, it can be difficult for patients to access and engage with, suggesting a need for improved understanding of the neural mechanisms of threat processing and avoidance behavior in OCD to guide development of new and more effective treatments. Platform mediated avoidance (PMA) has proven to be a useful translational paradigm for use in rodents to examine avoidance neurobiology and models relevant to OCD, such as ERP and overtraining-induced persistent avoidance. However, to date this protocol has only been used in rats, and studies in transgenic mouse models relevant to OCD may shed further light on neural mechanisms relevant to disturbances in avoidance and threat processing in the disorder. To address this gap, we tested Sapap3 knockout (KO) mice, a leading preclinical model in OCD research, in the PMA task. Using this paradigm, we examined avoidance acquisition, expression, and extinction, as well as reward seeking under motivational conflict in two separate cohorts conditioned using higher (0.4 mA) or lower (0.24 mA) intensity shock. Surprisingly, the most striking difference observed in Sapap3-KOs vs control mice was heightened suppression of lever pressing for rewards during a tone signaling impending threat, suggesting a shift in action selection under motivational conflict (genotype effect avoidance conditioning: 0.24 mA cohort p=0.011, 0.4 mA cohort p=0.07; avoidance extinction: 0.24 mA p=0.057, 0.4 mA p=0.042). Avoidance responding was also acquired more slowly in Sapap3-KOs trained with a low intensity shock (time x genotype interaction p=0.025) and was extinguished more robustly following ERP (genotype effect p=0.043). In contrast, avoidance was similar between Sapap3 KOs and WT littermate controls trained using higher intensity shock (0.4 mA cohort time effect p<0.0001). Expression of the immediate early gene c-Fos associated with reinstatement of avoidance after ERP showed preliminary evidence for decreased activity of medial orbitofrontal cortex (mOFC) in KOs which may contribute to observed differences in PMA performance (KO vs WT mOFC c-Fos t-test p=0.0456). Together these findings suggest that mOFC dysfunction may contribute to increases in the influence of threats vs rewards over action selection in an animal model with relevance to OCD, and that the Sapap3-KO model presents valuable opportunities for deeper mechanistic investigation of avoidance and threat processing relevant to the human disorder.
Bisphenol A induces sex-dependent alterations in the dynamics of neuroendocrine seasonal adaptation in Djungarian hamsters

In nature, species synchronize reproduction and energy metabolism with seasons to optimize survival and growth. While the effects of endocrine-disrupting chemicals (EDCs) exposure on conventional laboratory rodents are increasingly studied, their impacts on mammalian seasonal adaptation remain unexplored. This study investigates the effect of oral exposure to bisphenol A (BPA) on physiological and neuroendocrine seasonal adaptation in Djungarian hamsters. Adult female and male hamsters were orally exposed to BPA (5, 50, or 500 µg/kg/d) or vehicle during a 10-week transition from a long (LP) to short (SP) photoperiod (winter transition) or vice versa (summer transition). Changes in body weight, food intake, and pelage color were monitored weekly and, at the end of the exposure, gene expression of hypothalamic markers of photoperiodic, reproductive and metabolic integration, reproductive organ activity, and glycemia were assessed. Our results revealed sex-specific effects of BPA on acquiring SP and LP phenotypes. During LP to SP transition, females exposed to 500 µg/kg/d BPA exhibited delayed body weight loss and reduced feed efficiency associated with a lower expression of somatostatin in the arcuate nucleus (ARC), while males exposed to 5 µg/kg/d BPA showed an accelerated acquisition of SP-induced metabolic parameters. During SP to LP transition, females exposed to 5 µg/kg/d BPA displayed a faster LP adaptation in reproductive and metabolic parameters, along with quicker ARC kisspeptin downregulation and delayed ARC Pomc upregulation, while males exposed to BPA exhibited decreased expression of central photoperiodic integrators without changes in the physiological LP acquisition. This pioneering study investigating EDC impacts on mammalian seasonal physiology shows that BPA alters the dynamic of metabolic adaptation to both SP and LP transitions with marked sex dimorphism, causing temporal discordance in seasonal adaptation between males and females. These findings emphasize the importance of investigating EDCs’ impact on non-conventional animal models, providing insights into wildlife physiology. Highlights Djungarian hamster’s seasonal adaptation is disrupted by BPA oral exposure BPA delays in females and accelerates in males the metabolic adaptation to short days BPA accelerates in females, not in males, metabolic/reproductive adaptation to long days BPA affects the photoperiodic expression of central reproductive and metabolic genes Graphical abstract
Identifying Reproducibly Important EEG Markers of Schizophrenia with an Explainable Multi-Model Deep Learning Approach

The diagnosis of schizophrenia (SZ) can be challenging due to its diverse symptom presentation. As such, many studies have sought to identify diagnostic biomarkers of SZ using explainable machine learning methods. However, the generalizability of identified biomarkers in many machine learning-based studies is highly questionable given that most studies only analyze explanations from a small number of models. In this study, we present (1) a novel feature interaction-based explainability approach and (2) several new approaches for summarizing multi-model explanations. We implement our approach within the context of electroencephalogram (EEG) spectral power data. We further analyze both training and test set explanations with the goal of extracting generalizable insights from the models. Importantly, our analyses identify effects of SZ upon the α, β, and θ frequency bands, the left hemisphere of the brain, and interhemispheric interactions across a majority of folds. We hope that our analysis will provide helpful insights into SZ and inspire the development of robust approaches for identifying neuropsychiatric disorder biomarkers from explainable machine learning models.
Bats of a feather: Range characteristics and wing morphology predict phylogeographic breaks in volant vertebrates

Intraspecific genetic variation and phylogeographic structure can be influenced by factors such as landscape features, environmental gradients, historical biogeography, and organismal traits such as dispersal ability. Since deep genetic structure is often considered a precursor to speciation, identifying the factors that are associated with genetic structure can contribute to a greater understanding about diversification. Here, we use repurposed data to perform a global analysis of volant vertebrates (i.e., bats and birds) to estimate where intraspecific phylogeographic breaks occur and identify the factors that are important predictors of these breaks. We estimate phylogeographic breaks using Monmonier’s maximum difference barrier algorithm and conduct a Random Forests analysis using the presence of a phylogeographic break as a response variable. In bats, phylogeographic breaks are concentrated in biodiversity hotspots while breaks estimated in bird species are more widespread across temperate and tropical zones. However, for both clades geographical features such as maximum latitude, measures of wing morphology, and organismal traits associated with feeding ecology were found to be important predictors of phylogeographic breaks. Our analysis identifies geographical areas as wells as suites of organismal traits that could serve as a starting point for more detailed studies of biodiversity processes.
Tripole precipitation pattern and SST variations linked with extreme zonal activities of the western Pacific subtropical high

Using the daily variables of National Centers for Environmental Prediction reanalysis 1 from 1981 to 2014, this study first constructs a new index to describe the western Pacific subtropical high (WPSH) zonal activities by the normalized relative vorticity series averaged over a three‐dimensional core domain (112.5°–142.5°E, 10°–30°N and 925–500 hPa). Its horizontal scope is referred to as core region. Ten positive and six negative extreme episodes lasting from 10 to 17 days were identified based on the new index. They demonstrated prominent variabilities in intraseasonal and interannual timescales. Composite results show a tripole precipitation pattern with a wet (dry) zone over the tropical western Pacific (TWP) separating two dry (wet) zones in the meridional direction. The northern one covers the mei‐yu belt while the southern one extends along the tropical Indian Ocean (TIO) and the Maritime Continent. The activities of WPSH and monsoon trough form the height anomalies over the core region, while its northern height anomalies are due to the westerly trough in middle latitudes. Further, the three systems exert joint effects on the tripole dry‐wet pattern.
Association of Obesity and Bariatric Surgery on Hair Health

Obesity and obesity-related conditions today constitute a public health problem worldwide. Obesity is an “epidemic” chronic disorder, which is defined by the WHO as normal or excessive fat accumulation that may impair health. It is also defined for adults as a BMI that is greater than or equal to 30. The most common obesity-related diseases are type 2 diabetes mellitus, cardiovascular diseases, metabolic syndrome, chronic kidney disease, hyperlipidemia, hypertension, nonalcoholic fatty liver disease, and certain types of cancer. It has been also proven that obesity can have a negative effect on hair. It can lead to hair thinning. Patients with obesity can undergo bariatric surgery if they meet the inclusion criteria. The four common types of weight loss surgery include a duodenal switch with biliopancreatic diversion, laparoscopic adjustable gastric banding, Roux-en-Y gastric bypass, and sleeve gastrectomy. Bariatric surgery can affect skin and hair and is associated with telogen effluvium due to weight loss, microelement deficiency, anesthesia, low calorie intake, and low protein intake. Patients who undergo bariatric surgery can experience post-bariatric surgery depression. Hair loss can have a major impact on self-esteem, negatively affecting one’s self-image. The purpose of this narrative review is to critically review how obesity, obesity-related diseases, and bariatric surgery affect hair health in general and the hair development cycle, and how they influence hair loss.