Orionnnn/hub-stats huggingface dataset:
Multimodal Autoregressive Pre-training of Large Vision Encoders
Authors: name: Enrico Fini, name: Mustafa Shukor, name: Xiujun Li, name: Philipp Dufter, name: Michal Klein, name: David Haldimann, name: Sai Aitharaju, name: Victor Guilherme Turrisi da Costa, name: Louis Béthune, name: Zhe Gan, name: Alexander T Toshev, name: Marcin Eichner, name: Moin Nabi, name: Yinfei Yang, name: Joshua M. Susskind, name: Alaaeldin El-Nouby
Published: 2024-11-21 18:31:25

We introduce a novel method for pre-training of large-scale vision encoders.
Building on recent advancements in autoregressive pre-training of vision
models, we extend this framework to a multimodal setting, i.e., images and
text. In this paper, we present AIMV2, a family of generalist vision encoders
characterized by a straightforward pre-training process, scalability, and
remarkable performance across a range of downstream tasks. This is achieved by
pairing the vision encoder with a multimodal decoder that autoregressively
generates raw image patches and text tokens. Our encoders excel not only in
multimodal evaluations but also in vision benchmarks such as localization,
grounding, and classification. Notably, our AIMV2-3B encoder achieves 89.5%
accuracy on ImageNet-1k with a frozen trunk. Furthermore, AIMV2 consistently
outperforms state-of-the-art contrastive models (e.g., CLIP, SigLIP) in
multimodal image understanding across diverse settings.
Stable Flow: Vital Layers for Training-Free Image Editing
Authors: name: Omri Avrahami, name: Or Patashnik, name: Ohad Fried, name: Egor Nemchinov, name: Kfir Aberman, name: Dani Lischinski, name: Daniel Cohen-Or
Published: 2024-11-21 18:59:51

Diffusion models have revolutionized the field of content synthesis and
editing. Recent models have replaced the traditional UNet architecture with the
Diffusion Transformer (DiT), and employed flow-matching for improved training
and sampling. However, they exhibit limited generation diversity. In this work,
we leverage this limitation to perform consistent image edits via selective
injection of attention features. The main challenge is that, unlike the
UNet-based models, DiT lacks a coarse-to-fine synthesis structure, making it
unclear in which layers to perform the injection. Therefore, we propose an
automatic method to identify "vital layers" within DiT, crucial for image
formation, and demonstrate how these layers facilitate a range of controlled
stable edits, from non-rigid modifications to object addition, using the same
mechanism. Next, to enable real-image editing, we introduce an improved image
inversion method for flow models. Finally, we evaluate our approach through
qualitative and quantitative comparisons, along with a user study, and
demonstrate its effectiveness across multiple applications. The project page is
available at https://omriavrahami.com/stable-flow
Enhancing the Reasoning Ability of Multimodal Large Language Models via
  Mixed Preference Optimization
Authors: name: Weiyun Wang, name: Zhe Chen, name: Wenhai Wang, name: Yue Cao, name: Yangzhou Liu, name: Zhangwei Gao, name: Jinguo Zhu, name: Xizhou Zhu, name: Lewei Lu, name: Yu Qiao, name: Jifeng Dai
Published: 2024-11-15 18:59:27

Existing open-source multimodal large language models (MLLMs) generally
follow a training process involving pre-training and supervised fine-tuning.
However, these models suffer from distribution shifts, which limit their
multimodal reasoning, particularly in the Chain-of-Thought (CoT) performance.
To address this, we introduce a preference optimization (PO) process to enhance
the multimodal reasoning capabilities of MLLMs. Specifically, (1) on the data
side, we design an automated preference data construction pipeline to create
MMPR, a high-quality, large-scale multimodal reasoning preference dataset. and
(2) on the model side, we explore integrating PO with MLLMs, developing a
simple yet effective method, termed Mixed Preference Optimization (MPO), which
boosts multimodal CoT performance. Our approach demonstrates improved
performance across multiple benchmarks, particularly in multimodal reasoning
tasks. Notably, our model, InternVL2-8B-MPO, achieves an accuracy of 67.0 on
MathVista, outperforming InternVL2-8B by 8.7 points and achieving performance
comparable to the 10x larger InternVL2-76B. We hope this study could inspire
further advancements in MLLMs. Code, data, and model shall be publicly
released.
Hymba: A Hybrid-head Architecture for Small Language Models
Authors: name: Xin Dong, name: Yonggan Fu, name: Shizhe Diao, name: Wonmin Byeon, name: Zijia Chen, name: Ameya Sunil Mahabaleshwarkar, name: Shih-Yang Liu, name: Matthijs Van Keirsbilck, name: Min-Hung Chen, name: Yoshi Suhara, name: Yingyan Lin, name: Jan Kautz, name: Pavlo Molchanov
Published: 2024-11-20 19:51:25

We propose Hymba, a family of small language models featuring a hybrid-head
parallel architecture that integrates transformer attention mechanisms with
state space models (SSMs) for enhanced efficiency. Attention heads provide
high-resolution recall, while SSM heads enable efficient context summarization.
Additionally, we introduce learnable meta tokens that are prepended to prompts,
storing critical information and alleviating the "forced-to-attend" burden
associated with attention mechanisms. This model is further optimized by
incorporating cross-layer key-value (KV) sharing and partial sliding window
attention, resulting in a compact cache size. During development, we conducted
a controlled study comparing various architectures under identical settings and
observed significant advantages of our proposed architecture. Notably, Hymba
achieves state-of-the-art results for small LMs: Our Hymba-1.5B-Base model
surpasses all sub-2B public models in performance and even outperforms
Llama-3.2-3B with 1.32% higher average accuracy, an 11.67x cache size
reduction, and 3.49x throughput.
Ultra-Sparse Memory Network
Authors: name: Zihao Huang, name: Qiyang Min, name: Hongzhi Huang, name: Defa Zhu, name: Yutao Zeng, name: Ran Guo, name: Xun Zhou
Published: 2024-11-19 09:24:34

It is widely acknowledged that the performance of Transformer models is
exponentially related to their number of parameters and computational
complexity. While approaches like Mixture of Experts (MoE) decouple parameter
count from computational complexity, they still face challenges in inference
due to high memory access costs. This work introduces UltraMem, incorporating
large-scale, ultra-sparse memory layer to address these limitations. Our
approach significantly reduces inference latency while maintaining model
performance. We also investigate the scaling laws of this new architecture,
demonstrating that it not only exhibits favorable scaling properties but
outperforms traditional models. In our experiments, we train networks with up
to 20 million memory slots. The results show that our method achieves
state-of-the-art inference speed and model performance within a given
computational budget.
Marco-o1: Towards Open Reasoning Models for Open-Ended Solutions
Authors: name: Yu Zhao, name: Huifeng Yin, name: Bo Zeng, name: Hao Wang, name: Tianqi Shi, name: Chenyang Lyu, name: Longyue Wang, name: Weihua Luo, name: Kaifu Zhang
Published: 2024-11-21 18:37:33

Currently OpenAI o1 has sparked a surge of interest in the study of large
reasoning models (LRM). Building on this momentum, Marco-o1 not only focuses on
disciplines with standard answers, such as mathematics, physics, and coding --
which are well-suited for reinforcement learning (RL) -- but also places
greater emphasis on open-ended resolutions. We aim to address the question:
"Can the o1 model effectively generalize to broader domains where clear
standards are absent and rewards are challenging to quantify?" Marco-o1 is
powered by Chain-of-Thought (CoT) fine-tuning, Monte Carlo Tree Search (MCTS),
reflection mechanisms, and innovative reasoning strategies -- optimized for
complex real-world problem-solving tasks.
Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large
  Language Models
Authors: name: Yuhao Dong, name: Zuyan Liu, name: Hai-Long Sun, name: Jingkang Yang, name: Winston Hu, name: Yongming Rao, name: Ziwei Liu
Published: 2024-11-21 18:59:55

Large Language Models (LLMs) demonstrate enhanced capabilities and
reliability by reasoning more, evolving from Chain-of-Thought prompting to
product-level solutions like OpenAI o1. Despite various efforts to improve LLM
reasoning, high-quality long-chain reasoning data and optimized training
pipelines still remain inadequately explored in vision-language tasks. In this
paper, we present Insight-V, an early effort to 1) scalably produce long and
robust reasoning data for complex multi-modal tasks, and 2) an effective
training pipeline to enhance the reasoning capabilities of multi-modal large
language models (MLLMs). Specifically, to create long and structured reasoning
data without human labor, we design a two-step pipeline with a progressive
strategy to generate sufficiently long and diverse reasoning paths and a
multi-granularity assessment method to ensure data quality. We observe that
directly supervising MLLMs with such long and complex reasoning data will not
yield ideal reasoning ability. To tackle this problem, we design a multi-agent
system consisting of a reasoning agent dedicated to performing long-chain
reasoning and a summary agent trained to judge and summarize reasoning results.
We further incorporate an iterative DPO algorithm to enhance the reasoning
agent's generation stability and quality. Based on the popular LLaVA-NeXT model
and our stronger base MLLM, we demonstrate significant performance gains across
challenging multi-modal benchmarks requiring visual reasoning. Benefiting from
our multi-agent system, Insight-V can also easily maintain or improve
performance on perception-focused multi-modal tasks.
OpenScholar: Synthesizing Scientific Literature with Retrieval-augmented
  LMs
Authors: name: Akari Asai, name: Jacqueline He, name: Rulin Shao, name: Weijia Shi, name: Amanpreet Singh, name: Joseph Chee Chang, name: Kyle Lo, name: Luca Soldaini, name: Sergey Feldman, name: Mike D'arcy, name: David Wadden, name: Matt Latzke, name: Minyang Tian, name: Pan Ji, name: Shengyan Liu, name: Hao Tong, name: Bohao Wu, name: Yanyu Xiong, name: Luke Zettlemoyer, name: Graham Neubig, name: Dan Weld, name: Doug Downey, name: Wen-tau Yih, name: Pang Wei Koh, name: Hannaneh Hajishirzi
Published: 2024-11-21 15:07:42

Scientific progress depends on researchers' ability to synthesize the growing
body of literature. Can large language models (LMs) assist scientists in this
task? We introduce OpenScholar, a specialized retrieval-augmented LM that
answers scientific queries by identifying relevant passages from 45 million
open-access papers and synthesizing citation-backed responses. To evaluate
OpenScholar, we develop ScholarQABench, the first large-scale multi-domain
benchmark for literature search, comprising 2,967 expert-written queries and
208 long-form answers across computer science, physics, neuroscience, and
biomedicine. On ScholarQABench, OpenScholar-8B outperforms GPT-4o by 5% and
PaperQA2 by 7% in correctness, despite being a smaller, open model. While GPT4o
hallucinates citations 78 to 90% of the time, OpenScholar achieves citation
accuracy on par with human experts. OpenScholar's datastore, retriever, and
self-feedback inference loop also improves off-the-shelf LMs: for instance,
OpenScholar-GPT4o improves GPT-4o's correctness by 12%. In human evaluations,
experts preferred OpenScholar-8B and OpenScholar-GPT4o responses over
expert-written ones 51% and 70% of the time, respectively, compared to GPT4o's
32%. We open-source all of our code, models, datastore, data and a public demo.
Natural Language Reinforcement Learning
Authors: name: Xidong Feng, name: Ziyu Wan, name: Haotian Fu, name: Bo Liu, name: Mengyue Yang, name: Girish A. Koushik, name: Zhiyuan Hu, name: Ying Wen, name: Jun Wang
Published: 2024-11-21 15:57:02

Reinforcement Learning (RL) mathematically formulates decision-making with
Markov Decision Process (MDP). With MDPs, researchers have achieved remarkable
breakthroughs across various domains, including games, robotics, and language
models. This paper seeks a new possibility, Natural Language Reinforcement
Learning (NLRL), by extending traditional MDP to natural language-based
representation space. Specifically, NLRL innovatively redefines RL principles,
including task objectives, policy, value function, Bellman equation, and policy
iteration, into their language counterparts. With recent advancements in large
language models (LLMs), NLRL can be practically implemented to achieve RL-like
policy and value improvement by either pure prompting or gradient-based
training. Experiments over Maze, Breakthrough, and Tic-Tac-Toe games
demonstrate the effectiveness, efficiency, and interpretability of the NLRL
framework among diverse use cases. Our code will be released at
https://github.com/waterhorse1/Natural-language-RL.
Generating Compositional Scenes via Text-to-image RGBA Instance
  Generation
Authors: name: Alessandro Fontanella, name: Petru-Daniel Tudosiu, name: Yongxin Yang, name: Shifeng Zhang, name: Sarah Parisot
Published: 2024-11-16 23:44:14

Text-to-image diffusion generative models can generate high quality images at
the cost of tedious prompt engineering. Controllability can be improved by
introducing layout conditioning, however existing methods lack layout editing
ability and fine-grained control over object attributes. The concept of
multi-layer generation holds great potential to address these limitations,
however generating image instances concurrently to scene composition limits
control over fine-grained object attributes, relative positioning in 3D space
and scene manipulation abilities. In this work, we propose a novel multi-stage
generation paradigm that is designed for fine-grained control, flexibility and
interactivity. To ensure control over instance attributes, we devise a novel
training paradigm to adapt a diffusion model to generate isolated scene
components as RGBA images with transparency information. To build complex
images, we employ these pre-generated instances and introduce a multi-layer
composite generation process that smoothly assembles components in realistic
scenes. Our experiments show that our RGBA diffusion model is capable of
generating diverse and high quality instances with precise control over object
attributes. Through multi-layer composition, we demonstrate that our approach
allows to build and manipulate images from highly complex prompts with
fine-grained control over object appearance and location, granting a higher
degree of control than competing methods.
Loss-to-Loss Prediction: Scaling Laws for All Datasets
Authors: name: David Brandfonbrener, name: Nikhil Anand, name: Nikhil Vyas, name: Eran Malach, name: Sham Kakade
Published: 2024-11-19 23:23:16

While scaling laws provide a reliable methodology for predicting train loss
across compute scales for a single data distribution, less is known about how
these predictions should change as we change the distribution. In this paper,
we derive a strategy for predicting one loss from another and apply it to
predict across different pre-training datasets and from pre-training data to
downstream task data. Our predictions extrapolate well even at 20x the largest
FLOP budget used to fit the curves. More precisely, we find that there are
simple shifted power law relationships between (1) the train losses of two
models trained on two separate datasets when the models are paired by training
compute (train-to-train), (2) the train loss and the test loss on any
downstream distribution for a single model (train-to-test), and (3) the test
losses of two models trained on two separate train datasets (test-to-test). The
results hold up for pre-training datasets that differ substantially (some are
entirely code and others have no code at all) and across a variety of
downstream tasks. Finally, we find that in some settings these shifted power
law relationships can yield more accurate predictions than extrapolating
single-dataset scaling laws.
ViBe: A Text-to-Video Benchmark for Evaluating Hallucination in Large
  Multimodal Models
Authors: name: Vipula Rawte, name: Sarthak Jain, name: Aarush Sinha, name: Garv Kaushik, name: Aman Bansal, name: Prathiksha Rumale Vishwanath, name: Samyak Rajesh Jain, name: Aishwarya Naresh Reganti, name: Vinija Jain, name: Aman Chadha, name: Amit P. Sheth, name: Amitava Das
Published: 2024-11-16 19:23:12

Latest developments in Large Multimodal Models (LMMs) have broadened their
capabilities to include video understanding. Specifically, Text-to-video (T2V)
models have made significant progress in quality, comprehension, and duration,
excelling at creating videos from simple textual prompts. Yet, they still
frequently produce hallucinated content that clearly signals the video is
AI-generated. We introduce ViBe: a large-scale Text-to-Video Benchmark of
hallucinated videos from T2V models. We identify five major types of
hallucination: Vanishing Subject, Numeric Variability, Temporal Dysmorphia,
Omission Error, and Physical Incongruity. Using 10 open-source T2V models, we
developed the first large-scale dataset of hallucinated videos, comprising
3,782 videos annotated by humans into these five categories. ViBe offers a
unique resource for evaluating the reliability of T2V models and provides a
foundation for improving hallucination detection and mitigation in video
generation. We establish classification as a baseline and present various
ensemble classifier configurations, with the TimeSFormer + CNN combination
yielding the best performance, achieving 0.345 accuracy and 0.342 F1 score.
This benchmark aims to drive the development of robust T2V models that produce
videos more accurately aligned with input prompts.
Stylecodes: Encoding Stylistic Information For Image Generation
Authors: name: Ciara Rowles
Published: 2024-11-19 19:04:31

Diffusion models excel in image generation, but controlling them remains a
challenge. We focus on the problem of style-conditioned image generation.
Although example images work, they are cumbersome: srefs (style-reference
codes) from MidJourney solve this issue by expressing a specific image style in
a short numeric code. These have seen widespread adoption throughout social
media due to both their ease of sharing and the fact they allow using an image
for style control, without having to post the source images themselves.
However, users are not able to generate srefs from their own images, nor is the
underlying training procedure public. We propose StyleCodes: an open-source and
open-research style encoder architecture and training procedure to express
image style as a 20-symbol base64 code. Our experiments show that our encoding
results in minimal loss in quality compared to traditional image-to-style
techniques.
When Precision Meets Position: BFloat16 Breaks Down RoPE in Long-Context
  Training
Authors: name: Haonan Wang, name: Qian Liu, name: Chao Du, name: Tongyao Zhu, name: Cunxiao Du, name: Kenji Kawaguchi, name: Tianyu Pang
Published: 2024-11-20 17:22:31

Extending context window sizes allows large language models (LLMs) to process
longer sequences and handle more complex tasks. Rotary Positional Embedding
(RoPE) has become the de facto standard due to its relative positional encoding
properties that benefit long-context training. However, we observe that using
RoPE with BFloat16 format results in numerical issues, causing it to deviate
from its intended relative positional encoding, especially in long-context
scenarios. This issue arises from BFloat16's limited precision and accumulates
as context length increases, with the first token contributing significantly to
this problem. To address this, we develop AnchorAttention, a plug-and-play
attention method that alleviates numerical issues caused by BFloat16, improves
long-context capabilities, and speeds up training. AnchorAttention reduces
unnecessary attention computations, maintains semantic coherence, and boosts
computational efficiency by treating the first token as a shared anchor with a
consistent position ID, making it visible to all documents within the training
context. Experiments on three types of LLMs demonstrate that AnchorAttention
significantly improves long-context performance and reduces training time by
over 50\% compared to standard full attention mechanisms, while preserving the
original LLM's capabilities on general tasks. Our code is available at
https://github.com/haonan3/AnchorContext.
ORID: Organ-Regional Information Driven Framework for Radiology Report
  Generation
Authors: name: Tiancheng Gu, name: Kaicheng Yang, name: Xiang An, name: Ziyong Feng, name: Dongnan Liu, name: Weidong Cai
Published: 2024-11-20 04:13:43

The objective of Radiology Report Generation (RRG) is to automatically
generate coherent textual analyses of diseases based on radiological images,
thereby alleviating the workload of radiologists. Current AI-based methods for
RRG primarily focus on modifications to the encoder-decoder model architecture.
To advance these approaches, this paper introduces an Organ-Regional
Information Driven (ORID) framework which can effectively integrate multi-modal
information and reduce the influence of noise from unrelated organs.
Specifically, based on the LLaVA-Med, we first construct an RRG-related
instruction dataset to improve organ-regional diagnosis description ability and
get the LLaVA-Med-RRG. After that, we propose an organ-based cross-modal fusion
module to effectively combine the information from the organ-regional diagnosis
description and radiology image. To further reduce the influence of noise from
unrelated organs on the radiology report generation, we introduce an organ
importance coefficient analysis module, which leverages Graph Neural Network
(GNN) to examine the interconnections of the cross-modal information of each
organ region. Extensive experiments an1d comparisons with state-of-the-art
methods across various evaluation metrics demonstrate the superior performance
of our proposed method.
VideoAutoArena: An Automated Arena for Evaluating Large Multimodal
  Models in Video Analysis through User Simulation
Authors: name: Ziyang Luo, name: Haoning Wu, name: Dongxu Li, name: Jing Ma, name: Mohan Kankanhalli, name: Junnan Li
Published: 2024-11-20 12:48:34

Large multimodal models (LMMs) with advanced video analysis capabilities have
recently garnered significant attention. However, most evaluations rely on
traditional methods like multiple-choice questions in benchmarks such as
VideoMME and LongVideoBench, which are prone to lack the depth needed to
capture the complex demands of real-world users. To address this limitation-and
due to the prohibitive cost and slow pace of human annotation for video
tasks-we introduce VideoAutoArena, an arena-style benchmark inspired by LMSYS
Chatbot Arena's framework, designed to automatically assess LMMs' video
analysis abilities. VideoAutoArena utilizes user simulation to generate
open-ended, adaptive questions that rigorously assess model performance in
video understanding. The benchmark features an automated, scalable evaluation
framework, incorporating a modified ELO Rating System for fair and continuous
comparisons across multiple LMMs. To validate our automated judging system, we
construct a 'gold standard' using a carefully curated subset of human
annotations, demonstrating that our arena strongly aligns with human judgment
while maintaining scalability. Additionally, we introduce a fault-driven
evolution strategy, progressively increasing question complexity to push models
toward handling more challenging video analysis scenarios. Experimental results
demonstrate that VideoAutoArena effectively differentiates among
state-of-the-art LMMs, providing insights into model strengths and areas for
improvement. To further streamline our evaluation, we introduce VideoAutoBench
as an auxiliary benchmark, where human annotators label winners in a subset of
VideoAutoArena battles. We use GPT-4o as a judge to compare responses against
these human-validated answers. Together, VideoAutoArena and VideoAutoBench
offer a cost-effective, and scalable framework for evaluating LMMs in
user-centric video analysis.