Miracle-dz/UniFAR-FTUs huggingface dataset:
Image Captioners Are Scalable Vision Learners Too

Contrastive pretraining on image-text pairs from the web is one of the most popular large-scale pretraining strategies for vision backbones, especially in the context of large multimodal models. At the same time, image captioning on this type of data is commonly considered an inferior pretraining strategy. In this paper, we perform a fair comparison of these two pretraining strategies, carefully matching training data, compute, and model capacity. Using a standard encoder-decoder transformer, we find that captioning alone is surprisingly effective: on classification tasks, captioning produces vision encoders competitive with contrastively pretrained encoders, while surpassing them on vision&language tasks. We further analyze the effect of the model architecture and scale, as well as the pretraining data on the representation quality, and find that captioning exhibits the same or better scaling behavior along these axes. Overall our results show that plain image captioning is a more powerful pretraining strategy than was previously believed.
The main role of fractal-like nature of conformational space in subdiffusion in protein

Protein dynamics is a fundamental element to comprehend their biological functions. However, a theoretical picture providing microscopic-detail explanation of its relevant features is still missing. One of the outmost relevant properties exhibited by this dynamic is its subdiffusivity, whose origins are still unknown. Here, by directly comparing all-atom molecular dynamics simulations and theory we show that this behavior mainly arises from the fractal nature of the network of metastable state of conformational state over which protein dynamics, thought as diffusion process, takes place. This process is assumed to be Markovian by the employed theoretical picture. Therefore, to further support its validity, we built a simple Markov state model starting from the simulations outcome and show that it exhibits a subdiffusive behavior, in quantitative agreement with the one associated to the molecular dynamics. Moreover, Molecular dynamics gives direct access to relevant quantities which allowed us to rule out the possibility the Continuous Time Random Walk can explain the protein subdiffusivity.
BOUNDED COHOMOLOGY PROPERTY ON SMOOTH PROJECTIVE SURFACES WITH ARBITRARY PICARD NUMBER

Let X be a smooth projective surface and the closed Mori cone NE( X ) = P R ≥ 0 [ C i ] with some curves C i on X . An interesting question is asked that whether X satisﬁes the bounded cohomology property, which is that there exists a positive constant c X such that h 1 ( O X ( C )) ≤ c X h 0 ( O X ( C )) for every curve C on X . We prove X satisﬁes the bounded cohomology property if each curve C i has either C 2 i < 0 or the Iitaka dimension κ ( X, C i ) = 1. When the Picard number ρ ( X ) = 2, we prove that X satisﬁes the bounded cohomology property if either (i) the Kodaira dimension κ ( X ) ≤ 1 or (ii) κ ( X, C 1 ) = 1 and C 22 < 0.
Randomized least-squares with minimal oversampling and interpolation in general spaces

In approximation of functions based on point values, least-squares methods provide more stability than interpolation, at the expense of increasing the sampling budget. We show that near-optimal approximation error can nevertheless be achieved, in an expected $L^2$ sense, as soon as the sample size $m$ is larger than the dimension $n$ of the approximation space by a constant ratio. On the other hand, for $m=n$, we obtain an interpolation strategy with a stability factor of order $n$. The proposed sampling algorithms are greedy procedures based on arXiv:0808.0163 and arXiv:1508.03261, with polynomial computational complexity.
MOFI: Learning Image Representations from Noisy Entity Annotated Images

We present MOFI, Manifold OF Images, a new vision foundation model designed to learn image representations from noisy entity annotated images. MOFI differs from previous work in two key aspects: (i) pre-training data, and (ii) training recipe. Regarding data, we introduce a new approach to automatically assign entity labels to images from noisy image-text pairs. Our approach involves employing a named entity recognition model to extract entities from the alt-text, and then using a CLIP model to select the correct entities as labels of the paired image. It's a simple, cost-effective method that can scale to handle billions of web-mined image-text pairs. Through this method, we have created Image-to-Entities (I2E), a new dataset with 1 billion images and 2 million distinct entities, covering rich visual concepts in the wild. Building upon the I2E dataset, we study different training recipes like supervised pre-training, contrastive pre-training, and multi-task learning. For contrastive pre-training, we treat entity names as free-form text, and further enrich them with entity descriptions. Experiments show that supervised pre-training with large-scale fine-grained entity labels is highly effective for image retrieval tasks, and multi-task training further improves the performance. The final MOFI model achieves 86.66% mAP on the challenging GPR1200 dataset, surpassing the previous state-of-the-art performance of 72.19% from OpenAI's CLIP model. Further experiments on zero-shot and linear probe image classification also show that MOFI outperforms a CLIP model trained on the original image-text data, demonstrating the effectiveness of the I2E dataset in learning strong image representations. We release our code and model weights at https://github.com/apple/ml-mofi.
Questioning the Survey Responses of Large Language Models

As large language models increase in capability, researchers have started to conduct surveys of all kinds on these models in order to investigate the population represented by their responses. In this work, we critically examine language models' survey responses on the basis of the well-established American Community Survey by the U.S. Census Bureau and investigate whether they elicit a faithful representations of any human population. Using a de-facto standard multiple-choice prompting technique and evaluating 39 different language models using systematic experiments, we establish two dominant patterns: First, models' responses are governed by ordering and labeling biases, leading to variations across models that do not persist after adjusting for systematic biases. Second, models' responses do not contain the entropy variations and statistical signals typically found in human populations. As a result, a binary classifier can almost perfectly differentiate model-generated data from the responses of the U.S. census. At the same time, models' relative alignment with different demographic subgroups can be predicted from the subgroups' entropy, irrespective of the model's training data or training strategy. Taken together, our findings suggest caution in treating models' survey responses as equivalent to those of human populations.
Maximization of the olfactory receptor neuron selectivity in the sub-threshold regime

It is known that if odors are presented to an olfactory receptor neuron (ORN) in a sub-threshold concentration -- i.e., when the average value of the number of the ORN bound receptor proteins (RPs) is insufficient for the generation of spikes, but such a generation is still possible due to fluctuations around the average value -- the ORN selectivity can be higher than the selectivity at higher concentrations and, in particular, higher than the selectivity of the ORN's RPs. In this work, the optimal odorant concentration providing the highest ORN selectivity is found in the framework of a simplified ORN model, and the dependence of the highest selectivity on the total number of RPs in the ORN, $N$, and its threshold value $N_{0}$ is derived. The effect of enhanced selectivity in the sub-threshold regime is best manifested, if $N_{0}$ is close to either unity or $N$. It is also more pronounced at large $N$-values.
Incentivizing High-Quality Content in Online Recommender Systems

For content recommender systems such as TikTok and YouTube, the platform's decision algorithm shapes the incentives of content producers, including how much effort the content producers invest in the quality of their content. Many platforms employ online learning, which creates intertemporal incentives, since content produced today affects recommendations of future content. In this paper, we study the incentives arising from online learning, analyzing the quality of content produced at a Nash equilibrium. We show that classical online learning algorithms, such as Hedge and EXP3, unfortunately incentivize producers to create low-quality content. In particular, the quality of content is upper bounded in terms of the learning rate and approaches zero for typical learning rate schedules. Motivated by this negative result, we design a different learning algorithm -- based on punishing producers who create low-quality content -- that correctly incentivizes producers to create high-quality content. At a conceptual level, our work illustrates the unintended impact that a platform's learning algorithm can have on content quality and opens the door towards designing platform learning algorithms that incentivize the creation of high-quality content.
Stochastic Optimization Algorithms for Problems with Controllable Biased Oracles

Motivated by multiple emerging applications in machine learning, we consider an optimization problem in a general form where the gradient of the objective is available through a biased stochastic oracle. We assume the bias magnitude can be reduced by a bias-control parameter, however, a lower bias requires more computation/samples. For instance, for two applications on stochastic composition optimization and policy optimization for infinite-horizon Markov decision processes, we show that the bias follows a power law and exponential decay, respectively, as functions of their corresponding bias control parameters. For problems with such gradient oracles, the paper proposes stochastic algorithms that adjust the bias-control parameter throughout the iterations. We analyze the nonasymptotic performance of the proposed algorithms in the nonconvex regime and establish their sample or bias-control computation complexities to obtain a stationary point. Finally, we numerically evaluate the performance of the proposed algorithms over the two applications.
Numerical Simulation of Power-Law Fluid Flow in a Trapezoidal Cavity using the Incompressible Finite-Difference Lattice Boltzmann Method

In this paper, a numerical investigation of power-law fluid flow in the trapezoidal cavity has been conducted by incompressible finite-difference lattice Boltzmann method (IFDLBM). By designing the equilibrium distribution function, the Navier-Stokes equations (NSEs) can be recovered exactly. Through the coordinate transformation method, the body-fitted grid in physical region is transformed into a uniform grid in computational region. The effect of Reynolds (Re) number, the power-law index $n$ and the vertical angle {\theta} on the trapezoidal cavity are investigated. According to the numerical results, we come to some conclusions. For low Re number Re=100, it can be found that the behavior of power-law fluid flow becomes more complicated with the increase of n. And as vertical angle {\theta} decreases, the flow becomes smooth and the number of vortices decreases. For high Re numbers, the flow development becomes more complex, the number and strength of vortices increase. If the Reynolds number increases further, the power-law fluid will changes from steady flow to periodic flow and then to turbulent flow. For the steady flow, the lager the {\theta}, the more complicated the vortices. And the critical Re number from steady to periodic state decreases with the decrease of power-law index n.
Large Language Models Sometimes Generate Purely Negatively-Reinforced Text

When using adversarial training, it is common practice to train against the most egregious failures. However, this might imply using examples with sensitive information (such as leaked passwords or security vulnerabilities) as training data. One might assume that language models trained with gradient descent never generate text snippets which were only present in examples associated with the lowest possible reward. In this paper, we show that this assumption is wrong: in some situations, large language models do learn from such negatively-reinforced examples. We present a specific training setup that enables Pythia-160M to guess passwords 13% more often than it would by guessing randomly, despite only showing it these passwords on examples where the model is incentivized to not output these passwords. Our code is available at www.github. com/FabienRoger/Learning-From-Negative-Examples
Discrete Graph Auto-Encoder

Despite advances in generative methods, accurately modeling the distribution of graphs remains a challenging task primarily because of the absence of predefined or inherent unique graph representation. Two main strategies have emerged to tackle this issue: 1) restricting the number of possible representations by sorting the nodes, or 2) using permutation-invariant/equivariant functions, specifically Graph Neural Networks (GNNs). In this paper, we introduce a new framework named Discrete Graph Auto-Encoder (DGAE), which leverages the strengths of both strategies and mitigate their respective limitations. In essence, we propose a strategy in 2 steps. We first use a permutation-equivariant auto-encoder to convert graphs into sets of discrete latent node representations, each node being represented by a sequence of quantized vectors. In the second step, we sort the sets of discrete latent representations and learn their distribution with a specifically designed auto-regressive model based on the Transformer architecture. Through multiple experimental evaluations, we demonstrate the competitive performances of our model in comparison to the existing state-of-the-art across various datasets. Various ablation studies support the interest of our method.
StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models

In this paper, we present StyleTTS 2, a text-to-speech (TTS) model that leverages style diffusion and adversarial training with large speech language models (SLMs) to achieve human-level TTS synthesis. StyleTTS 2 differs from its predecessor by modeling styles as a latent random variable through diffusion models to generate the most suitable style for the text without requiring reference speech, achieving efficient latent diffusion while benefiting from the diverse speech synthesis offered by diffusion models. Furthermore, we employ large pre-trained SLMs, such as WavLM, as discriminators with our novel differentiable duration modeling for end-to-end training, resulting in improved speech naturalness. StyleTTS 2 surpasses human recordings on the single-speaker LJSpeech dataset and matches it on the multispeaker VCTK dataset as judged by native English speakers. Moreover, when trained on the LibriTTS dataset, our model outperforms previous publicly available models for zero-shot speaker adaptation. This work achieves the first human-level TTS on both single and multispeaker datasets, showcasing the potential of style diffusion and adversarial training with large SLMs. The audio demos and source code are available at https://styletts2.github. io/.
Skill Disentanglement for Imitation Learning from Suboptimal Demonstrations

Imitation learning has achieved great success in many sequential decision-making tasks, in which a neural agent is learned by imitating collected human demonstrations. However, existing algorithms typically require a large number of high-quality demonstrations that are difficult and expensive to collect. Usually, a trade-off needs to be made between demonstration quality and quantity in practice. Targeting this problem, in this work we consider the imitation of sub-optimal demonstrations, with both a small clean demonstration set and a large noisy set. Some pioneering works have been proposed, but they suffer from many limitations, e.g., assuming a demonstration to be of the same optimality throughout time steps and failing to provide any interpretation w.r. t knowledge learned from the noisy set. Addressing these problems, we propose \method by evaluating and imitating at the sub-demonstration level, encoding action primitives of varying quality into different skills. Concretely, SDIL consists of a high-level controller to discover skills and a skill-conditioned module to capture action-taking policies, and is trained following a two-phase pipeline by first discovering skills with all demonstrations and then adapting the controller to only the clean set. A mutual-information-based regularization and a dynamic sub-demonstration optimality estimator are designed to promote disentanglement in the skill space. Extensive experiments are conducted over two gym environments and a real-world healthcare dataset to demonstrate the superiority of SDIL in learning from sub-optimal demonstrations and its improved interpretability by examining learned skills.
Accuracy-Time Tradeoffs in AI-Assisted Decision Making under Time Pressure

In settings where users both need high accuracy and are time-pressured, such as doctors working in emergency rooms, we want to provide AI assistance that both increases decision accuracy and reduces decision-making time. Current literature focusses on how users interact with AI assistance when there is no time pressure, finding that different AI assistances have different benefits: some can reduce time taken while increasing overreliance on AI, while others do the opposite. The precise benefit can depend on both the user and task. In time-pressured scenarios, adapting when we show AI assistance is especially important: relying on the AI assistance can save time, and can therefore be beneficial when the AI is likely to be right. We would ideally adapt what AI assistance we show depending on various properties (of the task and of the user) in order to best trade off accuracy and time. We introduce a study where users have to answer a series of logic puzzles. We find that time pressure affects how users use different AI assistances, making some assistances more beneficial than others when compared to no-time-pressure settings. We also find that a user's overreliance rate is a key predictor of their behaviour: overreliers and not-overreliers use different AI assistance types differently. We find marginal correlations between a user's overreliance rate (which is related to the user's trust in AI recommendations) and their personality traits (Big Five Personality traits). Overall, our work suggests that AI assistances have different accuracy-time tradeoffs when people are under time pressure compared to no time pressure, and we explore how we might adapt AI assistances in this setting.
Oracle-Efficient Pessimism: Offline Policy Optimization in Contextual Bandits

We consider offline policy optimization (OPO) in contextual bandits, where one is given a fixed dataset of logged interactions. While pessimistic regularizers are typically used to mitigate distribution shift, prior implementations thereof are either specialized or computationally inefficient. We present the first general oracle-efficient algorithm for pessimistic OPO: it reduces to supervised learning, leading to broad applicability. We obtain statistical guarantees analogous to those for prior pessimistic approaches. We instantiate our approach for both discrete and continuous actions and perform experiments in both settings, showing advantage over unregularized OPO across a wide range of configurations.
Excited Hydroxyl Outflow in the High-mass Star-forming Region G34.26 + 0.15

G34.26 + 0.15 is a region of high-mass star formation that contains a broad range of young stellar objects in different stages of evolution, including a hot molecular core, hypercompact H ii regions, and a prototypical cometary ultracompact H ii region. Previous high-sensitivity single-dish observations by our group resulted in the detection of broad 6035 MHz OH absorption in this region; the line showed a significant blueshifted asymmetry indicative of molecular gas expansion. We present high-sensitivity Karl G. Jansky Very Large Array (VLA) observations of the 6035 MHz OH line conducted to image the absorption and investigate its origin with respect to the different star formation sites in the region. In addition, we report detection of 6030 MHz OH absorption with the VLA and further observations of 4.7 GHz and 6.0 GHz OH lines obtained with the Arecibo Telescope. The 6030 MHz OH line shows a very similar absorption profile as the 6035 MHz OH line. We found that the 6035 MHz OH line absorption region is spatially unresolved at ∼2″ scales, and it is coincident with one of the bright ionized cores of the cometary H ii region that shows broad radio recombination line emission. We discuss a scenario where the OH absorption is tracing the remnants of a pole-on molecular outflow that is being ionized inside-out by the ultracompact H ii region.
EriBERTa: A Bilingual Pre-Trained Language Model for Clinical Natural Language Processing

The utilization of clinical reports for various secondary purposes, including health research and treatment monitoring, is crucial for enhancing patient care. Natural Language Processing (NLP) tools have emerged as valuable assets for extracting and processing relevant information from these reports. However, the availability of specialized language models for the clinical domain in Spanish has been limited. In this paper, we introduce EriBERTa, a bilingual domain-specific language model pre-trained on extensive medical and clinical corpora. We demonstrate that EriBERTa outperforms previous Spanish language models in the clinical domain, showcasing its superior capabilities in understanding medical texts and extracting meaningful information. Moreover, EriBERTa exhibits promising transfer learning abilities, allowing for knowledge transfer from one language to another. This aspect is particularly beneficial given the scarcity of Spanish clinical data.
Khovanskii bases for semimixed systems of polynomial equations -- a case of approximating stationary nonlinear Newtonian dynamics

We provide an approach to counting roots of polynomial systems, where each polynomial is a general linear combination of prescribed, fixed polynomials. Our tools rely on the theory of Khovanskii bases, combined with toric geometry, the Bernstein-Khovanskii-Kushnirenko (BKK) Theorem, and fiber products. As a direct application of this theory, we solve the problem of counting the number of approximate stationary states for coupled driven nonlinear resonators. We set up a system of polynomial equations that depends on three numbers $N, n$ and $M$ and whose solutions model the stationary states. The parameter $N$ is the number of coupled resonators, $2n - 1$ is the degree of nonlinearity of the underlying differential equation, and $M$ is the number of frequencies used in the approximation. We use our main theorems, that is, the generalized BKK Theorem and the Decoupling Theorem, to count the number of (complex) solutions of the polynomial system for an arbitrary degree of nonlinearity $2n - 1 \geq 3$, any number of resonators $N \geq 1$, and $M = 1$ harmonic. We also solve the case $N = 1, n = 2$ and $M = 2$ and give a computational way to check the number of solutions for $N = 1, n = 2$ and $M \geq 2$. This extends the results of arXiv:2208.08179.