Miracle-dz/UniFAR-FTUs huggingface dataset:
CXR-LLAVA: a multimodal large language model for interpreting chest X-ray images

Purpose: This study aimed to develop an open-source multimodal large language model (CXR-LLAVA) for interpreting chest X-ray images (CXRs), leveraging recent advances in large language models (LLMs) to potentially replicate the image interpretation skills of human radiologists Materials and Methods: For training, we collected 592,580 publicly available CXRs, of which 374,881 had labels for certain radiographic abnormalities (Dataset 1) and 217,699 provided free-text radiology reports (Dataset 2). After pre-training a vision transformer with Dataset 1, we integrated it with an LLM influenced by the LLAVA network. Then, the model was fine-tuned, primarily using Dataset 2. The model's diagnostic performance for major pathological findings was evaluated, along with the acceptability of radiologic reports by human radiologists, to gauge its potential for autonomous reporting. Results: The model demonstrated impressive performance in test sets, achieving an average F1 score of 0.81 for six major pathological findings in the MIMIC internal test set and 0.62 for seven major pathological findings in the external test set. The model's F1 scores surpassed those of GPT-4-vision and Gemini-Pro-Vision in both test sets. In human radiologist evaluations of the external test set, the model achieved a 72.7% success rate in autonomous reporting, slightly below the 84.0% rate of ground truth reports. Conclusion: This study highlights the significant potential of multimodal LLMs for CXR interpretation, while also acknowledging the performance limitations. Despite these challenges, we believe that making our model open-source will catalyze further research, expanding its effectiveness and applicability in various clinical contexts. CXR-LLAVA is available at https://github.com/ECOFRI/CXR_LLAVA.
Perturbation Theory of Single Particle Spectrum of Antiferromagnetic Mott Insulating States in the Hubbard Models

In this work, we present an analytical framework for studying antiferromagnetic (AFM) Mott insulating states in the Hubbard model. We first derive an analytical solution for the single-particle Green's functions in the atomic limit. Within a second-order perturbation approach, we compute the ground state energy and show that the ground state is antiferromagnetically ordered. Then we derive an analytical solution for single-particle Green's functions when effects of the hopping term are considered in the N\'{e}el state. With the analytical solution, we compute and explain various properties of antiferromagnetic Mott insulators observed both experimentally and numerically: i) magnetic blueshift of the Mott gap; ii) spectral functions with features comparable to observations by angle-resolved photoemission spectroscopy on parental compounds of cuprate high $T_c$ superconductors. This work comprehends the electronic properties of antiferromagnetic Mott states analytically and provides a foundation for future investigations of doped antiferromagnetic Mott insulators, aiming for the mechanism of cuprates high-$T_c$ superconductivity.
Undermassive Host Galaxies of Five z ∼ 6 Luminous Quasars Detected with JWST

We measure host galaxy stellar masses for a sample of five luminous quasars at z ∼ 5–7. Using JWST/NIRCam medium-band images of nearby point-spread function reference stars, we carefully subtract the contribution from the quasar light to place upper and lower limits on the flux of each host galaxy. We find that the members of our sample of quasar host galaxies have mass upper limits of 109.7–1010.8 M ⊙, significantly less than expected from their supermassive black hole (SMBH) masses and the local M BH–M * relation. We additionally obtain JWST/NIRSpec integral field unit spectra of three of our quasars to calculate black hole masses, which we find are consistent with those in the literature, and to search for the presence of bright but compact galaxies via a Balmer break, for which we do not find evidence. We discuss the potential effects of dust extinction on our measured fluxes and the impact of selection effects on high-redshift quasar samples. We conclude that the masses of the SMBHs relative to the host galaxy stellar masses have a much larger scatter than locally, large enough that these selection effects cannot be responsible. This result is reinforced by other studies. Finally, we explore the potential implications of these results on the picture of SMBH–galaxy coeval growth in the early Universe.
Methodological challenges of scenario generation validation: a rear-end crash-causation model for virtual safety assessment

Safety assessment of crash and conflict avoidance systems is important for both the automotive industry and other stakeholders. One type of system that needs such an assessment is a driver monitoring system (DMS) with some intervention (e.g., warning or nudging) when the driver looks off-road for too long. Although using computer simulation to assess safety systems is becoming increasingly common, it is not yet commonly used for systems that affect driver behavior, such as DMSs. Models that generate virtual crashes, taking crash-causation mechanisms into account, are needed to assess these systems. However, few such models exist, and those that do have not been thoroughly validated on real-world data. This study aims to address this research gap by validating a rear-end crash-causation model which is based on four crash-causation mechanisms related to driver behavior: a) off-road glances, b) too-short headway, c) not braking with the maximum deceleration possible, and d) sleepiness (not reacting before the crash). The pre-crash kinematics were obtained from the German GIDAS in-depth crash database. Challenges with the validation process were identified and addressed. Most notably, a process was developed to transform the generated crashes to mimic the crash severity distribution in GIDAS. This step was necessary because GIDAS does not include property-damage-only (PDO) crashes, while the generated crashes cover the full range of severities (including low-severity crashes, of which many are PDOs). Our results indicate that the proposed model is a reasonably good crash generator. We further demonstrated that the model is a valid method for assessing DMSs in virtual simulations; it shows the safety impact of shorter"longest"off-road glances. As expected, cutting away long off-road glances substantially reduces the number of crashes that occur and reduces the average delta-v.
$L^p$-Hardy identities and inequalities with respect to the distance and mean distance to the boundary

Firstly, this paper establishes useful forms of the remainder term of Hardy-type inequalities on general domains where the weights are functions of the distance to the boundary. For weakly mean convex domains we use the resulting identities to establish nonexistence of extremizers for and improve known sharp Hardy inequalities. Secondly, we establish geometrically interesting remainders for the Davies-Hardy-Tidblom inequalities for the mean distance function, as well as generalize and improve several Hardy type inequalities in the spirit of Brezis and Marcus and spectral estimates of Davies. Lastly, we apply our results to obtain Sobolev inequalities for non-regular Riemannian metrics on geometric exterior domains.
Switching Temporary Teachers for Semi-Supervised Semantic Segmentation

The teacher-student framework, prevalent in semi-supervised semantic segmentation, mainly employs the exponential moving average (EMA) to update a single teacher's weights based on the student's. However, EMA updates raise a problem in that the weights of the teacher and student are getting coupled, causing a potential performance bottleneck. Furthermore, this problem may become more severe when training with more complicated labels such as segmentation masks but with few annotated data. This paper introduces Dual Teacher, a simple yet effective approach that employs dual temporary teachers aiming to alleviate the coupling problem for the student. The temporary teachers work in shifts and are progressively improved, so consistently prevent the teacher and student from becoming excessively close. Specifically, the temporary teachers periodically take turns generating pseudo-labels to train a student model and maintain the distinct characteristics of the student model for each epoch. Consequently, Dual Teacher achieves competitive performance on the PASCAL VOC, Cityscapes, and ADE20K benchmarks with remarkably shorter training times than state-of-the-art methods. Moreover, we demonstrate that our approach is model-agnostic and compatible with both CNN- and Transformer-based models. Code is available at \url{https://github.com/naver-ai/dual-teacher}.
Hopping and crawling DNA-coated colloids

Understanding the motion of particles with ligand-receptors is important for biomedical applications and material design. Yet, even among a single design, the prototypical DNA-coated colloids, seemingly similar micrometric particles hop or roll, depending on the study. We shed light on this problem by observing DNA-coated colloids diffusing near surfaces coated with complementary strands for a wide array of coating designs. We find colloids rapidly switch between 2 modes: they hop - with long and fast steps - and crawl - with short and slow steps. Both modes occur at all temperatures around the melting point and over a wide array of designs. The particles become increasingly subdiffusive as temperature decreases, in line with subsequent velocity steps becoming increasingly anti-correlated. Overall, crawling (or hopping) phases are more predominant at low (or high) temperatures; crawling is also more efficient at low temperatures than hopping to cover large distances. We rationalize this behavior within a simple model: at lower temperatures, the number of bound strands increases, and detachment of all bonds is unlikely, hence, hopping is prevented and crawling favored. We thus reveal the mechanism behind a common design rule relying on increased strand density for long-range self-assembly: dense strands on surfaces are required to enable crawling, possibly facilitating particle rearrangements.
Nature abhors a vacuum: A simple rigorous example of thermalization in an isolated macroscopic quantum system

We show, without relying on any unproven assumptions, that a low-density free fermion chain exhibits thermalization in the following (restricted) sense. We choose the initial state as a pure state drawn randomly from the Hilbert space in which all particles are in half of the chain. This represents a nonequilibrium state such that the half chain containing all particles is in equilibrium at infinite temperature, and the other half chain is a vacuum. We let the system evolve according to the unitary time evolution determined by the Hamiltonian and, at a sufficiently large typical time, measure the particle number in an arbitrary macroscopic region in the chain. In this setup, it is proved that the measured number is close to the equilibrium value with probability very close to one. Our result establishes the presence of thermalization in a concrete model in a mathematically rigorous manner. The most important theoretical ingredient for the proof of thermalization is the demonstration that a nonequilibrium initial state generated as above typically has a sufficiently large effective dimension. Here, we first give general proof of thermalization based on two assumptions, namely, the absence of degeneracy in energy eigenvalues and a property about the particle distribution in energy eigenstates. We then justify these assumptions in a concrete free-fermion model, where the absence of degeneracy is established by using number-theoretic results. This means that our general result also applies to any lattice gas models in which the above two assumptions are justified. To confirm the potential wide applicability of our theory, we discuss some other models for which the essential assumption about the particle distribution is easily verified, and some non-random initial states whose effective dimensions are sufficiently large.
Apollo: Zero-shot MultiModal Reasoning with Multiple Experts

We propose a modular framework that leverages the expertise of different foundation models over different modalities and domains in order to perform a single, complex, multi-modal task, without relying on prompt engineering or otherwise tailor-made multi-modal training. Our approach enables decentralized command execution and allows each model to both contribute and benefit from the expertise of the other models. Our method can be extended to a variety of foundation models (including audio and vision), above and beyond only language models, as it does not depend on prompts. We demonstrate our approach on two tasks. On the well-known task of stylized image captioning, our experiments show that our approach outperforms semi-supervised state-of-the-art models, while being zero-shot and avoiding costly training, data collection, and prompt engineering. We further demonstrate this method on a novel task, audio-aware image captioning, in which an image and audio are given and the task is to generate text that describes the image within the context of the provided audio. Our code is available on GitHub.
Pre-trained Speech Processing Models Contain Human-Like Biases that Propagate to Speech Emotion Recognition

Previous work has established that a person's demographics and speech style affect how well speech processing models perform for them. But where does this bias come from? In this work, we present the Speech Embedding Association Test (SpEAT), a method for detecting bias in one type of model used for many speech tasks: pre-trained models. The SpEAT is inspired by word embedding association tests in natural language processing, which quantify intrinsic bias in a model's representations of different concepts, such as race or valence (something's pleasantness or unpleasantness) and capture the extent to which a model trained on large-scale socio-cultural data has learned human-like biases. Using the SpEAT, we test for six types of bias in 16 English speech models (including 4 models also trained on multilingual data), which come from the wav2vec 2.0, HuBERT, WavLM, and Whisper model families. We find that 14 or more models reveal positive valence (pleasantness) associations with abled people over disabled people, with European-Americans over African-Americans, with females over males, with U.S. accented speakers over non-U.S. accented speakers, and with younger people over older people. Beyond establishing that pre-trained speech models contain these biases, we also show that they can have real world effects. We compare biases found in pre-trained models to biases in downstream models adapted to the task of Speech Emotion Recognition (SER) and find that in 66 of the 96 tests performed (69%), the group that is more associated with positive valence as indicated by the SpEAT also tends to be predicted as speaking with higher valence by the downstream model. Our work provides evidence that, like text and image-based models, pre-trained speech based-models frequently learn human-like biases. Our work also shows that bias found in pre-trained models can propagate to the downstream task of SER.
SEASONAL VARIATION IN ECOLOGICAL AND TAPHONOMIC PROCESSES RECORDED IN SHELLY DEATH ASSEMBLAGES

Abstract Modern valves of Lucina pensylvanica (Bivalvia: Lucinidae) were analyzed for 18O/16O ratios, drilling predation traces, biometric measurements, and taphonomic descriptors to explore seasonal variations in ecological and taphonomic processes within a death assemblage from Grand Bahama Island. The &dgr;18O values at the shell margin ( = last growth episode) were used as a proxy for temperature at the time closest to the organism's death. Temperature estimates suggest that most individuals died in warmer months, whereas mortality appeared to have been lower during cooler seasons (<24 °C). Only drilled valves yielded the coldest temperatures at the shell margin (∼18.1–21.3 °C), whereas exclusively undrilled valves exhibited the hottest temperatures at the last growth episode (∼25.5–26.5 °C). Drilled valves were significantly smaller and exhibited higher taphonomic alteration than undrilled valves, pointing to postmortem transport and sorting. While seasonal variations in predation and growth rates may have affected the observed patterns to some extent, the results, at least partly, reflect hydrodynamic variations throughout the year, which may have induced seasonal variation in postmortem sorting of drilled and undrilled valves. This study illustrates that, by combining geochemical proxies and quantitative paleoecological data, more informed ecologic and taphonomic interpretations can be achieved for subfossil and fossil assemblages.
Evaluating the effects of high-throughput structural neuroimaging predictors on whole-brain functional connectome outcomes via network-based vector-on-matrix regression

The joint analysis of multimodal neuroimaging data is critical in the field of brain research because it reveals complex interactive relationships between neurobiological structures and functions. In this study, we focus on investigating the effects of structural imaging (SI) features, including white matter micro-structure integrity (WMMI) and cortical thickness, on the whole brain functional connectome (FC) network. To achieve this goal, we propose a network-based vector-on-matrix regression model to characterize the FC-SI association patterns. We have developed a novel multi-level dense bipartite and clique subgraph extraction method to identify which subsets of spatially specific SI features intensively influence organized FC sub-networks. The proposed method can simultaneously identify highly correlated structural-connectomic association patterns and suppress false positive findings while handling millions of potential interactions. We apply our method to a multimodal neuroimaging dataset of 4,242 participants from the UK Biobank to evaluate the effects of whole-brain WMMI and cortical thickness on the resting-state FC. The results reveal that the WMMI on corticospinal tracts and inferior cerebellar peduncle significantly affect functional connections of sensorimotor, salience, and executive sub-networks with an average correlation of 0.81 (p<0.001).
Setting the Trap: Capturing and Defeating Backdoors in Pretrained Language Models through Honeypots

In the field of natural language processing, the prevalent approach involves fine-tuning pretrained language models (PLMs) using local samples. Recent research has exposed the susceptibility of PLMs to backdoor attacks, wherein the adversaries can embed malicious prediction behaviors by manipulating a few training samples. In this study, our objective is to develop a backdoor-resistant tuning procedure that yields a backdoor-free model, no matter whether the fine-tuning dataset contains poisoned samples. To this end, we propose and integrate a honeypot module into the original PLM, specifically designed to absorb backdoor information exclusively. Our design is motivated by the observation that lower-layer representations in PLMs carry sufficient backdoor features while carrying minimal information about the original tasks. Consequently, we can impose penalties on the information acquired by the honeypot module to inhibit backdoor creation during the fine-tuning process of the stem network. Comprehensive experiments conducted on benchmark datasets substantiate the effectiveness and robustness of our defensive strategy. Notably, these results indicate a substantial reduction in the attack success rate ranging from 10\% to 40\% when compared to prior state-of-the-art methods.
Feature Selection in the Contrastive Analysis Setting

Contrastive analysis (CA) refers to the exploration of variations uniquely enriched in a target dataset as compared to a corresponding background dataset generated from sources of variation that are irrelevant to a given task. For example, a biomedical data analyst may wish to find a small set of genes to use as a proxy for variations in genomic data only present among patients with a given disease (target) as opposed to healthy control subjects (background). However, as of yet the problem of feature selection in the CA setting has received little attention from the machine learning community. In this work we present contrastive feature selection (CFS), a method for performing feature selection in the CA setting. We motivate our approach with a novel information-theoretic analysis of representation learning in the CA setting, and we empirically validate CFS on a semi-synthetic dataset and four real-world biomedical datasets. We find that our method consistently outperforms previously proposed state-of-the-art supervised and fully unsupervised feature selection methods not designed for the CA setting. An open-source implementation of our method is available at https://github.com/suinleelab/CFS.
The Evolution of the Interplay Between Input Distributions and Linear Regions in Networks

It is commonly recognized that the expressiveness of deep neural networks is contingent upon a range of factors, encompassing their depth, width, and other relevant considerations. Currently, the practical performance of the majority of deep neural networks remains uncertain. For ReLU (Rectified Linear Unit) networks with piecewise linear activations, the number of linear convex regions serves as a natural metric to gauge the network's expressivity. In this paper, we count the number of linear convex regions in deep neural networks based on ReLU. In particular, we prove that for any one-dimensional input, there exists a minimum threshold for the number of neurons required to express it. We also empirically observe that for the same network, intricate inputs hinder its capacity to express linear regions. Furthermore, we unveil the iterative refinement process of decision boundaries in ReLU networks during training. We aspire for our research to serve as an inspiration for network optimization endeavors and aids in the exploration and analysis of the behaviors exhibited by deep networks.
Node Cardinality Estimation in the Internet of Things Using Privileged Feature Distillation

The Internet of Things (IoT) is emerging as a critical technology to connect resource-constrained devices such as sensors and actuators as well as appliances to the Internet. In this paper, we propose a novel methodology for node cardinality estimation in wireless networks such as the IoT and Radio-Frequency IDentification (RFID) systems, which uses the privileged feature distillation (PFD) technique and works using a neural network with a teacher-student model. The teacher is trained using both privileged and regular features, and the student is trained with predictions from the teacher and regular features. We propose node cardinality estimation algorithms based on the PFD technique for homogeneous as well as heterogeneous wireless networks. We show via extensive simulations that the proposed PFD based algorithms for homogeneous as well as heterogeneous networks achieve much lower mean squared errors in the computed node cardinality estimates than state-of-the-art protocols proposed in prior work, while taking the same number of time slots for executing the node cardinality estimation process as the latter protocols.
On the $L^r$-differentiability of Two Lusin Classes and a Full Descriptive Characterization of the $HK_r$-integral

It is proved that any function of a Lusin-type class, the class of $ACG_r$-functions, is differentiable almost everywhere in the sense of a derivative defined in the space~$L^r$, $1\le r<\infty$. This leads to obtaining a full descriptive characterization of a Henstock-Kurzweil-type integral, the $HK_r$-integral, which serves to recover functions from their $L^r$-derivatives. The class $ACG_r$ is compared with the classical Lusin class $ACG$ and it is shown that a continuous $ACG$-function can fail to be $L^r$-differentiable almost everywhere.