Miracle-dz/UniFAR-FTUs huggingface dataset:
Multinomial belief networks

A Bayesian approach to machine learning is attractive when we need to quantify uncertainty, deal with missing observations, when samples are scarce, or when the data is sparse. All of these commonly apply when analysing healthcare data. To address these analytical requirements, we propose a deep generative model for multinomial count data where both the weights and hidden units of the network are Dirichlet distributed. A Gibbs sampling procedure is formulated that takes advantage of a series of augmentation relations, analogous to the Zhou--Cong--Chen model. We apply the model on small handwritten digits, and a large experimental dataset of DNA mutations in cancer, and we show how the model is able to extract biologically meaningful meta-signatures in a fully data-driven way.
MobileCLIP: Fast Image-Text Models through Multi-Modal Reinforced Training

Contrastive pretraining of image-text foundation models, such as CLIP, demonstrated excellent zero-shot performance and improved robustness on a wide range of downstream tasks. However, these models utilize large transformer-based encoders with significant memory and latency overhead which pose challenges for deployment on mobile devices. In this work, we introduce MobileCLIP -- a new family of efficient image-text models optimized for runtime performance along with a novel and efficient training approach, namely multi-modal reinforced training. The proposed training approach leverages knowledge transfer from an image captioning model and an ensemble of strong CLIP encoders to improve the accuracy of efficient models. Our approach avoids train-time compute overhead by storing the additional knowledge in a reinforced dataset. MobileCLIP sets a new state-of-the-art latency-accuracy tradeoff for zero-shot classification and retrieval tasks on several datasets. Our MobileCLIP-S2 variant is 2.3$\times$ faster while more accurate compared to previous best CLIP model based on ViT-B/16. We further demonstrate the effectiveness of our multi-modal reinforced training by training a CLIP model based on ViT-B/16 image backbone and achieving +2.9% average performance improvement on 38 evaluation benchmarks compared to the previous best. Moreover, we show that the proposed approach achieves 10$\times$-1000$\times$ improved learning efficiency when compared with non-reinforced CLIP training.
Graph Partitioning with Fujitsu Digital Annealer

Graph partitioning, or community detection, is the cornerstone of many fields, such as logistics, transportation and smart power grids. Efficient computation and efficacious evaluation of communities are both essential, especially in commercial and industrial settings. However, the solution space of graph partitioning increases drastically with the number of vertices and subgroups. With an eye to solving large scale graph partitioning and other optimization problems within a short period of time, the Digital Annealer (DA), a specialized CMOS hardware also featuring improved algorithms, has been devised by Fujitsu Ltd. This study gauges Fujitsu DA's performance and running times. The modularity was implemented as both the objective function and metric for the solutions. The graph partitioning problems were formatted into Quadratic Unconstrained Binary Optimization (QUBO) structures so that they could be adequately imported into the DA. The DA yielded the highest modularity among other studies when partitioning Karate Club, Les Miserables, American Football, and Dolphin. Moreover, the DA was able to partition the Case 1354pegase power grid network into 45 subgroups, calling for 60,930 binary variables, whilst delivering optimal modularity results within a solving time of roughly 80 seconds. Our results suggest that the Fujitsu DA can be applied for rapid and efficient optimization for graph partitioning.
Rescuing referral failures during automated diagnosis of domain-shifted medical images

The success of deep learning models deployed in the real world depends critically on their ability to generalize well across diverse data domains. Here, we address a fundamental challenge with selective classification during automated diagnosis with domain-shifted medical images. In this scenario, models must learn to avoid making predictions when label confidence is low, especially when tested with samples far removed from the training set (covariate shift). Such uncertain cases are typically referred to the clinician for further analysis and evaluation. Yet, we show that even state-of-the-art domain generalization approaches fail severely during referral when tested on medical images acquired from a different demographic or using a different technology. We examine two benchmark diagnostic medical imaging datasets exhibiting strong covariate shifts: i) diabetic retinopathy prediction with retinal fundus images and ii) multilabel disease prediction with chest X-ray images. We show that predictive uncertainty estimates do not generalize well under covariate shifts leading to non-monotonic referral curves, and severe drops in performance (up to 50%) at high referral rates (>70%). We evaluate novel combinations of robust generalization and post hoc referral approaches, that rescue these failures and achieve significant performance improvements, typically>10%, over baseline methods. Our study identifies a critical challenge with referral in domain-shifted medical images and finds key applications in reliable, automated disease diagnosis.
Diffusive evaporation dynamics in polymer solutions is ubiquitous

Recent theory and experiments have shown how the buildup of a high-concentration polymer layer at a one-dimensional solvent-air interface can lead to an evaporation rate that scales with time as $t^{-1/2}$ and that is insensitive to the ambient humidity. Using phase field modelling we show that this scaling law constitutes a naturally emerging robust regime, Diffusion-Limited Evaporation (DLE). This regime dominates the dynamical state diagram of the system, which also contains regions of constant and arrested evaporation, confirming and extending understanding of recent experimental observations and theoretical predictions. We provide a theoretical argument to show that the scaling observed in the DLE regime occurs for a wide range of parameters, and our simulations predict that it can occur in two-dimensional geometries as well. Finally, we discuss possible extensions to more complex systems.
Mitigating Hallucination in Visual Language Models with Visual Supervision

Large vision-language models (LVLMs) suffer from hallucination a lot, generating responses that apparently contradict to the image content occasionally. The key problem lies in its weak ability to comprehend detailed content in a multi-modal context, which can be mainly attributed to two factors in training data and loss function. The vision instruction dataset primarily focuses on global description, and the auto-regressive loss function favors text modeling rather than image understanding. In this paper, we bring more detailed vision annotations and more discriminative vision models to facilitate the training of LVLMs, so that they can generate more precise responses without encounter hallucination. On one hand, we generate image-text pairs with detailed relationship annotations in panoptic scene graph dataset (PSG). These conversations pay more attention on detailed facts in the image, encouraging the model to answer questions based on multi-modal contexts. On the other hand, we integrate SAM and mask prediction loss as auxiliary supervision, forcing the LVLMs to have the capacity to identify context-related objects, so that they can generate more accurate responses, mitigating hallucination. Moreover, to provide a deeper evaluation on the hallucination in LVLMs, we propose a new benchmark, RAH-Bench. It divides vision hallucination into three different types that contradicts the image with wrong categories, attributes or relations, and introduces False Positive Rate as detailed sub-metric for each type. In this benchmark, our approach demonstrates an +8.4% enhancement compared to original LLaVA and achieves widespread performance improvements across other models.
DI-Net : Decomposed Implicit Garment Transfer Network for Digital Clothed 3D Human

3D virtual try-on enjoys many potential applications and hence has attracted wide attention. However, it remains a challenging task that has not been adequately solved. Existing 2D virtual try-on methods cannot be directly extended to 3D since they lack the ability to perceive the depth of each pixel. Besides, 3D virtual try-on approaches are mostly built on the fixed topological structure and with heavy computation. To deal with these problems, we propose a Decomposed Implicit garment transfer network (DI-Net), which can effortlessly reconstruct a 3D human mesh with the newly try-on result and preserve the texture from an arbitrary perspective. Specifically, DI-Net consists of two modules: 1) A complementary warping module that warps the reference image to have the same pose as the source image through dense correspondence learning and sparse flow learning; 2) A geometry-aware decomposed transfer module that decomposes the garment transfer into image layout based transfer and texture based transfer, achieving surface and texture reconstruction by constructing pixel-aligned implicit functions. Experimental results show the effectiveness and superiority of our method in the 3D virtual try-on task, which can yield more high-quality results over other existing methods.
Geometric thermodynamics of reaction-diffusion systems: Thermodynamic trade-off relations and optimal transport for pattern formation

The geometric approach to nonequilibrium thermodynamics is promising for understanding the role of dissipation and thermodynamic trade-off relations. This paper proposes a geometric framework for studying the nonequilibrium thermodynamics of reaction-diffusion systems. Based on this framework, we obtain several decompositions of the entropy production rate with respect to conservativeness, spatial structure, and wavenumber. We also generalize optimal transport theory to reaction-diffusion systems and derive several trade-off relations, including thermodynamic speed limits and thermodynamic uncertainty relations. The thermodynamic trade-off relations obtained in this paper shed light on thermodynamic dissipation in pattern formation. We numerically demonstrate our results using the Fisher$\unicode{x2013}$Kolmogorov$\unicode{x2013}$Petrovsky$\unicode{x2013}$Piskunov equation and the Brusselator model and discuss how the spatial pattern affects unavoidable dissipation.
DiffAttack: Evasion Attacks Against Diffusion-Based Adversarial Purification

Diffusion-based purification defenses leverage diffusion models to remove crafted perturbations of adversarial examples and achieve state-of-the-art robustness. Recent studies show that even advanced attacks cannot break such defenses effectively, since the purification process induces an extremely deep computational graph which poses the potential problem of gradient obfuscation, high memory cost, and unbounded randomness. In this paper, we propose a unified framework DiffAttack to perform effective and efficient attacks against diffusion-based purification defenses, including both DDPM and score-based approaches. In particular, we propose a deviated-reconstruction loss at intermediate diffusion steps to induce inaccurate density gradient estimation to tackle the problem of vanishing/exploding gradients. We also provide a segment-wise forwarding-backwarding algorithm, which leads to memory-efficient gradient backpropagation. We validate the attack effectiveness of DiffAttack compared with existing adaptive attacks on CIFAR-10 and ImageNet. We show that DiffAttack decreases the robust accuracy of models compared with SOTA attacks by over 20% on CIFAR-10 under $\ell_\infty$ attack $(\epsilon=8/255)$, and over 10% on ImageNet under $\ell_\infty$ attack $(\epsilon=4/255)$. We conduct a series of ablations studies, and we find 1) DiffAttack with the deviated-reconstruction loss added over uniformly sampled time steps is more effective than that added over only initial/final steps, and 2) diffusion-based purification with a moderate diffusion length is more robust under DiffAttack.
The twisted partial group algebra and (co)homology of partial crossed products

Given a group $G$ and a partial factor set $\sigma $ of $G,$ we introduce the twisted partial group algebra $\kappa_{par}^{\sigma}G,$ which governs the partial projective $\sigma$-representations of $G$ into algebras over a filed $\kappa. $ Using the relation between partial projective representations and twisted partial actions we endow $\kappa_{par}^\sigma G$ with the structure of a crossed product by a twisted partial action of $G$ on a commutative subalgebra of $\kappa_{par}^{\sigma} G. $ Then, we use twisted partial group algebras to obtain a first quadrant Grothendieck spectral sequence converging to the Hochschild homology of the crossed product $A\ast_{\Theta} G,$ involving the Hochschild homology of $A$ and the partial homology of $G,$ where ${\Theta}$ is a unital twisted partial action of $G$ on a $\kappa$-algebra $A$ with a $\kappa $-based twist. An analogous third quadrant cohomological spectral sequence is also obtained.
Gradient-based Local Next-best-view Planning for Improved Perception of Targeted Plant Nodes

Robots are increasingly used in tomato greenhouses to automate labour-intensive tasks such as selective harvesting and de-leafing. To perform these tasks, robots must be able to accurately and efficiently perceive the plant nodes that need to be cut, despite the high levels of occlusion from other plant parts. We formulate this problem as a local next-best-view (NBV) planning task where the robot has to plan an efficient set of camera viewpoints to overcome occlusion and improve the quality of perception. Our formulation focuses on quickly improving the perception accuracy of a single target node to maximise its chances of being cut. Previous methods of NBV planning mostly focused on global view planning and used random sampling of candidate viewpoints for exploration, which could suffer from high computational costs, ineffective view selection due to poor candidates, or non-smooth trajectories due to inefficient sampling. We propose a gradient-based NBV planner using differential ray sampling, which directly estimates the local gradient direction for viewpoint planning to overcome occlusion and improve perception. Through simulation experiments, we showed that our planner can handle occlusions and improve the 3D reconstruction and position estimation of nodes equally well as a sampling-based NBV planner, while taking ten times less computation and generating 28% more efficient trajectories.
Making Self-supervised Learning Robust to Spurious Correlation via Learning-speed Aware Sampling

Self-supervised learning (SSL) has emerged as a powerful technique for learning rich representations from unlabeled data. The data representations are able to capture many underlying attributes of data, and be useful in downstream prediction tasks. In real-world settings, spurious correlations between some attributes (e.g. race, gender and age) and labels for downstream tasks often exist, e.g. cancer is usually more prevalent among elderly patients. In this paper, we investigate SSL in the presence of spurious correlations and show that the SSL training loss can be minimized by capturing only a subset of the conspicuous features relevant to those sensitive attributes, despite the presence of other important predictive features for the downstream tasks. To address this issue, we investigate the learning dynamics of SSL and observe that the learning is slower for samples that conflict with such correlations (e.g. elder patients without cancer). Motivated by these findings, we propose a learning-speed aware SSL (LA-SSL) approach, in which we sample each training data with a probability that is inversely related to its learning speed. We evaluate LA-SSL on three datasets that exhibit spurious correlations between different attributes, demonstrating that it improves the robustness of pretrained representations on downstream classification tasks.
Understanding the Effectiveness of Large Language Models in Detecting Security Vulnerabilities

Security vulnerabilities in modern software are prevalent and harmful. While automated vulnerability detection tools have made promising progress, their scalability and applicability remain challenging. Recently, Large Language Models (LLMs), such as GPT-4 and CodeLlama, have demonstrated remarkable performance on code-related tasks. However, it is unknown whether such LLMs can do complex reasoning over code. In this work, we explore whether pre-trained LLMs can detect security vulnerabilities and address the limitations of existing tools. We evaluate the effectiveness of pre-trained LLMs on a set of five diverse security benchmarks spanning two languages, Java and C/C++, and including code samples from synthetic and real-world projects. We evaluate the effectiveness of LLMs in terms of their performance, explainability, and robustness. By designing a series of effective prompting strategies, we obtain the best results on the synthetic datasets with GPT-4: F1 scores of 0.79 on OWASP, 0.86 on Juliet Java, and 0.89 on Juliet C/C++. Expectedly, the performance of LLMs drops on the more challenging real-world datasets: CVEFixes Java and CVEFixes C/C++, with GPT-4 reporting F1 scores of 0.48 and 0.62, respectively. We show that LLMs can often perform better than existing static analysis and deep learning-based vulnerability detection tools, especially for certain classes of vulnerabilities. Moreover, LLMs also often provide reliable explanations, identifying the vulnerable data flows in code. We find that fine-tuning smaller LLMs can outperform the larger LLMs on synthetic datasets but provide limited gains on real-world datasets. When subjected to adversarial attacks on code, LLMs show mild degradation, with average accuracy reduction of up to 12.67%. Finally, we share our insights and recommendations for future work on leveraging LLMs for vulnerability detection.
Ribosomal RNA synthesis by RNA polymerase I is regulated by premature termination of transcription

The RNA polymerase I (Pol I) enzyme that synthesizes large rRNA precursors, exhibits high rate of pauses during elongation, indicative of a discontinuous process. We propose here that Premature Termination of Transcription (PTT) by Pol I is a critical regulatory step limiting rRNA production in vivo. The Pol I mutant, SuperPol (RPA135-F301S), produces 1.5-fold more rRNA than the wild type (WT). Combined CRAC and rRNA analysis link increased rRNA production in SuperPol to reduced PTT, resulting in shifting polymerase distribution toward the 3’ end of rDNA genes. In vitro, SuperPol shows defective nascent transcript cleavage. Notably, SuperPol is resistant to BMH-21, a drug impairing Pol I elongation and inducing proteasome-mediated degradation of Pol I subunits. Compared to WT, SuperPol maintains subunit stability and sustains high transcription levels upon BMH-21 treatment. These comparative results show that PTT is alleviated in SuperPol while it is stimulated by BMH-21 in WT Pol I.
Parotid glands have a dysregulated immune response following radiation therapy

Head and neck cancer treatment often consists of surgical resection of the tumor followed by ionizing radiation (IR), which can damage surrounding tissues and cause adverse side effects. The underlying mechanisms of radiation-induced salivary gland dysfunction are not fully understood, and treatment options are scarce and ineffective. The wound healing process is a necessary response to tissue injury, and broadly consists of inflammatory, proliferative, and redifferentiation phases with immune cells playing key roles in all three phases. In this study, select immune cells were phenotyped and quantified, and certain cytokine and chemokine concentrations were measured in mouse parotid glands after IR. Further, we used a model where glandular function is restored to assess the immune phenotype in a regenerative response. These data suggest that irradiated parotid tissue does not progress through a typical inflammatory response observed in wounds that heal. Specifically, total immune cells (CD45+) decrease at days 2 and 5 following IR, macrophages (F4/80+CD11b+) decrease at day 2 and 5 and increase at day 30, while neutrophils (Ly6G+CD11b+) significantly increase at day 30 following IR. Additionally, radiation treatment reduces CD3-cells at all time points, significantly increases CD3+/CD4+CD8+ double positive cells, and significantly reduces CD3+/CD4-CD8-double negative cells at day 30 after IR. Previous data indicate that post-IR treatment with IGF-1 restores salivary gland function at day 30, and IGF-1 injections attenuate the increase in macrophages, neutrophils, and CD4+CD8+ T cells observed at day 30 following IR. Taken together, these data indicate that parotid salivary tissue exhibits a dysregulated immune response following radiation treatment which may contribute to chronic loss of function phenotype in head and neck cancer survivors.
Connectivity underlying motor cortex activity during naturalistic goal-directed behavior

Neural representations of information are shaped by local network interactions. Previous studies linking neural coding and cortical connectivity focused on stimulus selectivity in the sensory cortex 1–4. Here we study neural activity in the motor cortex during naturalistic behavior in which mice gathered rewards with multidirectional tongue reaching. This behavior does not require training and thus allowed us to probe neural coding and connectivity in motor cortex before its activity is shaped by learning a specific task. Neurons typically responded during and after reaching movements and exhibited conjunctive tuning to target location and reward outcome. We used an all-optical 5,4,6,7 method for large-scale causal functional connectivity mapping in vivo. Mapping connectivity between > 20,000,000 excitatory neuronal pairs revealed fine-scale columnar architecture in layer 2/3 of the motor cortex. Neurons displayed local (< 100 µm) like-to-like connectivity according to target-location tuning, and inhibition over longer spatial scales. Connectivity patterns comprised a continuum, with abundant weakly connected neurons and sparse strongly connected neurons that function as network hubs. Hub neurons were weakly tuned to target-location and reward-outcome but strongly influenced neighboring neurons. This network of neurons, encoding location and outcome of movements to different motor goals, may be a general substrate for rapid learning of complex, goal-directed behaviors.