Orionnnn/hub-stats huggingface dataset:
MV-Adapter: Multi-view Consistent Image Generation Made Easy
Authors: name: Zehuan Huang, name: Yuan-Chen Guo, name: Haoran Wang, name: Ran Yi, name: Lizhuang Ma, name: Yan-Pei Cao, name: Lu Sheng
Published: 2024-12-04 18:48:20

Existing multi-view image generation methods often make invasive
modifications to pre-trained text-to-image (T2I) models and require full
fine-tuning, leading to (1) high computational costs, especially with large
base models and high-resolution images, and (2) degradation in image quality
due to optimization difficulties and scarce high-quality 3D data. In this
paper, we propose the first adapter-based solution for multi-view image
generation, and introduce MV-Adapter, a versatile plug-and-play adapter that
enhances T2I models and their derivatives without altering the original network
structure or feature space. By updating fewer parameters, MV-Adapter enables
efficient training and preserves the prior knowledge embedded in pre-trained
models, mitigating overfitting risks. To efficiently model the 3D geometric
knowledge within the adapter, we introduce innovative designs that include
duplicated self-attention layers and parallel attention architecture, enabling
the adapter to inherit the powerful priors of the pre-trained models to model
the novel 3D knowledge. Moreover, we present a unified condition encoder that
seamlessly integrates camera parameters and geometric information, facilitating
applications such as text- and image-based 3D generation and texturing.
MV-Adapter achieves multi-view generation at 768 resolution on Stable Diffusion
XL (SDXL), and demonstrates adaptability and versatility. It can also be
extended to arbitrary view generation, enabling broader applications. We
demonstrate that MV-Adapter sets a new quality standard for multi-view image
generation, and opens up new possibilities due to its efficiency, adaptability
and versatility.
Densing Law of LLMs
Authors: name: Chaojun Xiao, name: Jie Cai, name: Weilin Zhao, name: Guoyang Zeng, name: Xu Han, name: Zhiyuan Liu, name: Maosong Sun
Published: 2024-12-05 16:31:13

Large Language Models (LLMs) have emerged as a milestone in artificial
intelligence, and their performance can improve as the model size increases.
However, this scaling brings great challenges to training and inference
efficiency, particularly for deploying LLMs in resource-constrained
environments, and the scaling trend is becoming increasingly unsustainable.
This paper introduces the concept of ``capacity density'' as a new
metric to evaluate the quality of the LLMs across different scales and
describes the trend of LLMs in terms of both effectiveness and efficiency. To
calculate the capacity density of a given target LLM, we first introduce a set
of reference models and develop a scaling law to predict the downstream
performance of these reference models based on their parameter sizes. We then
define the effective parameter size of the target LLM as the parameter
size required by a reference model to achieve equivalent performance, and
formalize the capacity density as the ratio of the effective parameter size to
the actual parameter size of the target LLM. Capacity density provides a
unified framework for assessing both model effectiveness and efficiency. Our
further analysis of recent open-source base LLMs reveals an empirical law (the
densing law)that the capacity density of LLMs grows exponentially over time.
More specifically, using some widely used benchmarks for evaluation, the
capacity density of LLMs doubles approximately every three months. The law
provides new perspectives to guide future LLM development, emphasizing the
importance of improving capacity density to achieve optimal results with
minimal computational overhead.
ZipAR: Accelerating Autoregressive Image Generation through Spatial
  Locality
Authors: name: Yefei He, name: Feng Chen, name: Yuanyu He, name: Shaoxuan He, name: Hong Zhou, name: Kaipeng Zhang, name: Bohan Zhuang
Published: 2024-12-05 10:57:08

In this paper, we propose ZipAR, a training-free, plug-and-play parallel
decoding framework for accelerating auto-regressive (AR) visual generation. The
motivation stems from the observation that images exhibit local structures, and
spatially distant regions tend to have minimal interdependence. Given a
partially decoded set of visual tokens, in addition to the original next-token
prediction scheme in the row dimension, the tokens corresponding to spatially
adjacent regions in the column dimension can be decoded in parallel, enabling
the ``next-set prediction'' paradigm. By decoding multiple tokens
simultaneously in a single forward pass, the number of forward passes required
to generate an image is significantly reduced, resulting in a substantial
improvement in generation efficiency. Experiments demonstrate that ZipAR can
reduce the number of model forward passes by up to 91% on the Emu3-Gen model
without requiring any additional retraining.
Florence-VL: Enhancing Vision-Language Models with Generative Vision
  Encoder and Depth-Breadth Fusion
Authors: name: Jiuhai Chen, name: Jianwei Yang, name: Haiping Wu, name: Dianqi Li, name: Jianfeng Gao, name: Tianyi Zhou, name: Bin Xiao
Published: 2024-12-05 18:50:39

We present Florence-VL, a new family of multimodal large language models
(MLLMs) with enriched visual representations produced by Florence-2, a
generative vision foundation model. Unlike the widely used CLIP-style vision
transformer trained by contrastive learning, Florence-2 can capture different
levels and aspects of visual features, which are more versatile to be adapted
to diverse downstream tasks. We propose a novel feature-fusion architecture and
an innovative training recipe that effectively integrates Florence-2's visual
features into pretrained LLMs, such as Phi 3.5 and LLama 3. In particular, we
propose "depth-breath fusion (DBFusion)" to fuse the visual features extracted
from different depths and under multiple prompts. Our model training is
composed of end-to-end pretraining of the whole model followed by finetuning of
the projection layer and the LLM, on a carefully designed recipe of diverse
open-source datasets that include high-quality image captions and
instruction-tuning pairs. Our quantitative analysis and visualization of
Florence-VL's visual features show its advantages over popular vision encoders
on vision-language alignment, where the enriched depth and breath play
important roles. Florence-VL achieves significant improvements over existing
state-of-the-art MLLMs across various multi-modal and vision-centric benchmarks
covering general VQA, perception, hallucination, OCR, Chart,
knowledge-intensive understanding, etc. To facilitate future research, our
models and the complete training recipe are open-sourced.
https://github.com/JiuhaiChen/Florence-VL
Infinity: Scaling Bitwise AutoRegressive Modeling for High-Resolution
  Image Synthesis
Authors: name: Jian Han, name: Jinlai Liu, name: Yi Jiang, name: Bin Yan, name: Yuqi Zhang, name: Zehuan Yuan, name: Bingyue Peng, name: Xiaobing Liu
Published: 2024-12-05 18:53:02

We present Infinity, a Bitwise Visual AutoRegressive Modeling capable of
generating high-resolution, photorealistic images following language
instruction. Infinity redefines visual autoregressive model under a bitwise
token prediction framework with an infinite-vocabulary tokenizer & classifier
and bitwise self-correction mechanism, remarkably improving the generation
capacity and details. By theoretically scaling the tokenizer vocabulary size to
infinity and concurrently scaling the transformer size, our method
significantly unleashes powerful scaling capabilities compared to vanilla VAR.
Infinity sets a new record for autoregressive text-to-image models,
outperforming top-tier diffusion models like SD3-Medium and SDXL. Notably,
Infinity surpasses SD3-Medium by improving the GenEval benchmark score from
0.62 to 0.73 and the ImageReward benchmark score from 0.87 to 0.96, achieving a
win rate of 66%. Without extra optimization, Infinity generates a high-quality
1024x1024 image in 0.8 seconds, making it 2.6x faster than SD3-Medium and
establishing it as the fastest text-to-image model. Models and codes will be
released to promote further exploration of Infinity for visual generation and
unified tokenizer modeling.
Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction
Authors: name: Yiheng Xu, name: Zekun Wang, name: Junli Wang, name: Dunjie Lu, name: Tianbao Xie, name: Amrita Saha, name: Doyen Sahoo, name: Tao Yu, name: Caiming Xiong
Published: 2024-12-05 18:58:26

Graphical User Interfaces (GUIs) are critical to human-computer interaction,
yet automating GUI tasks remains challenging due to the complexity and
variability of visual environments. Existing approaches often rely on textual
representations of GUIs, which introduce limitations in generalization,
efficiency, and scalability. In this paper, we introduce Aguvis, a unified pure
vision-based framework for autonomous GUI agents that operates across various
platforms. Our approach leverages image-based observations, and grounding
instructions in natural language to visual elements, and employs a consistent
action space to ensure cross-platform generalization. To address the
limitations of previous work, we integrate explicit planning and reasoning
within the model, enhancing its ability to autonomously navigate and interact
with complex digital environments. We construct a large-scale dataset of GUI
agent trajectories, incorporating multimodal reasoning and grounding, and
employ a two-stage training pipeline that first focuses on general GUI
grounding, followed by planning and reasoning. Through comprehensive
experiments, we demonstrate that Aguvis surpasses previous state-of-the-art
methods in both offline and real-world online scenarios, achieving, to our
knowledge, the first fully autonomous pure vision GUI agent capable of
performing tasks independently without collaboration with external
closed-source models. We open-sourced all datasets, models, and training
recipes to facilitate future research at https://aguvis-project.github.io/.
OmniFlow: Any-to-Any Generation with Multi-Modal Rectified Flows
Authors: name: Shufan Li, name: Konstantinos Kallidromitis, name: Akash Gokul, name: Zichun Liao, name: Yusuke Kato, name: Kazuki Kozuka, name: Aditya Grover
Published: 2024-12-02 06:13:01

We introduce OmniFlow, a novel generative model designed for any-to-any
generation tasks such as text-to-image, text-to-audio, and audio-to-image
synthesis. OmniFlow advances the rectified flow (RF) framework used in
text-to-image models to handle the joint distribution of multiple modalities.
It outperforms previous any-to-any models on a wide range of tasks, such as
text-to-image and text-to-audio synthesis. Our work offers three key
contributions: First, we extend RF to a multi-modal setting and introduce a
novel guidance mechanism, enabling users to flexibly control the alignment
between different modalities in the generated outputs. Second, we propose a
novel architecture that extends the text-to-image MMDiT architecture of Stable
Diffusion 3 and enables audio and text generation. The extended modules can be
efficiently pretrained individually and merged with the vanilla text-to-image
MMDiT for fine-tuning. Lastly, we conduct a comprehensive study on the design
choices of rectified flow transformers for large-scale audio and text
generation, providing valuable insights into optimizing performance across
diverse modalities. The Code will be available at
https://github.com/jacklishufan/OmniFlows.
Towards Universal Soccer Video Understanding
Authors: name: Jiayuan Rao, name: Haoning Wu, name: Hao Jiang, name: Ya Zhang, name: Yanfeng Wang, name: Weidi Xie
Published: 2024-12-02 18:58:04

As a globally celebrated sport, soccer has attracted widespread interest from
fans all over the world. This paper aims to develop a comprehensive multi-modal
framework for soccer video understanding. Specifically, we make the following
contributions in this paper: (i) we introduce SoccerReplay-1988, the largest
multi-modal soccer dataset to date, featuring videos and detailed annotations
from 1,988 complete matches, with an automated annotation pipeline; (ii) we
present the first visual-language foundation model in the soccer domain,
MatchVision, which leverages spatiotemporal information across soccer videos
and excels in various downstream tasks; (iii) we conduct extensive experiments
and ablation studies on event classification, commentary generation, and
multi-view foul recognition. MatchVision demonstrates state-of-the-art
performance on all of them, substantially outperforming existing models, which
highlights the superiority of our proposed data and model. We believe that this
work will offer a standard paradigm for sports understanding research.
Evaluating Language Models as Synthetic Data Generators
Authors: name: Seungone Kim, name: Juyoung Suk, name: Xiang Yue, name: Vijay Viswanathan, name: Seongyun Lee, name: Yizhong Wang, name: Kiril Gashteovski, name: Carolin Lawrence, name: Sean Welleck, name: Graham Neubig
Published: 2024-12-04 19:20:32

Given the increasing use of synthetic data in language model (LM)
post-training, an LM's ability to generate high-quality data has become nearly
as crucial as its ability to solve problems directly. While prior works have
focused on developing effective data generation methods, they lack systematic
comparison of different LMs as data generators in a unified setting. To address
this gap, we propose AgoraBench, a benchmark that provides standardized
settings and metrics to evaluate LMs' data generation abilities. Through
synthesizing 1.26 million training instances using 6 LMs and training 99
student models, we uncover key insights about LMs' data generation
capabilities. First, we observe that LMs exhibit distinct strengths. For
instance, GPT-4o excels at generating new problems, while Claude-3.5-Sonnet
performs better at enhancing existing ones. Furthermore, our analysis reveals
that an LM's data generation ability doesn't necessarily correlate with its
problem-solving ability. Instead, multiple intrinsic features of data
quality-including response quality, perplexity, and instruction
difficulty-collectively serve as better indicators. Finally, we demonstrate
that strategic choices in output format and cost-conscious model selection
significantly impact data generation effectiveness.
MRGen: Diffusion-based Controllable Data Engine for MRI Segmentation
  towards Unannotated Modalities
Authors: name: Haoning Wu, name: Ziheng Zhao, name: Ya Zhang, name: Weidi Xie, name: Yanfeng Wang
Published: 2024-12-04 16:34:22

Medical image segmentation has recently demonstrated impressive progress with
deep neural networks, yet the heterogeneous modalities and scarcity of mask
annotations limit the development of segmentation models on unannotated
modalities. This paper investigates a new paradigm for leveraging generative
models in medical applications: controllably synthesizing data for unannotated
modalities, without requiring registered data pairs. Specifically, we make the
following contributions in this paper: (i) we collect and curate a large-scale
radiology image-text dataset, MedGen-1M, comprising modality labels,
attributes, region, and organ information, along with a subset of organ mask
annotations, to support research in controllable medical image generation; (ii)
we propose a diffusion-based data engine, termed MRGen, which enables
generation conditioned on text prompts and masks, synthesizing MR images for
diverse modalities lacking mask annotations, to train segmentation models on
unannotated modalities; (iii) we conduct extensive experiments across various
modalities, illustrating that our data engine can effectively synthesize
training samples and extend MRI segmentation towards unannotated modalities.
VisionZip: Longer is Better but Not Necessary in Vision Language Models
Authors: name: Senqiao Yang, name: Yukang Chen, name: Zhuotao Tian, name: Chengyao Wang, name: Jingyao Li, name: Bei Yu, name: Jiaya Jia
Published: 2024-12-05 18:59:53

Recent advancements in vision-language models have enhanced performance by
increasing the length of visual tokens, making them much longer than text
tokens and significantly raising computational costs. However, we observe that
the visual tokens generated by popular vision encoders, such as CLIP and
SigLIP, contain significant redundancy. To address this, we introduce
VisionZip, a simple yet effective method that selects a set of informative
tokens for input to the language model, reducing visual token redundancy and
improving efficiency while maintaining model performance. The proposed
VisionZip can be widely applied to image and video understanding tasks and is
well-suited for multi-turn dialogues in real-world scenarios, where previous
methods tend to underperform. Experimental results show that VisionZip
outperforms the previous state-of-the-art method by at least 5% performance
gains across nearly all settings. Moreover, our method significantly enhances
model inference speed, improving the prefilling time by 8x and enabling the
LLaVA-Next 13B model to infer faster than the LLaVA-Next 7B model while
achieving better results. Furthermore, we analyze the causes of this redundancy
and encourage the community to focus on extracting better visual features
rather than merely increasing token length. Our code is available at
https://github.com/dvlab-research/VisionZip .
Structured 3D Latents for Scalable and Versatile 3D Generation
Authors: name: Jianfeng Xiang, name: Zelong Lv, name: Sicheng Xu, name: Yu Deng, name: Ruicheng Wang, name: Bowen Zhang, name: Dong Chen, name: Xin Tong, name: Jiaolong Yang
Published: 2024-12-02 13:58:38

We introduce a novel 3D generation method for versatile and high-quality 3D
asset creation. The cornerstone is a unified Structured LATent (SLAT)
representation which allows decoding to different output formats, such as
Radiance Fields, 3D Gaussians, and meshes. This is achieved by integrating a
sparsely-populated 3D grid with dense multiview visual features extracted from
a powerful vision foundation model, comprehensively capturing both structural
(geometry) and textural (appearance) information while maintaining flexibility
during decoding. We employ rectified flow transformers tailored for SLAT as our
3D generation models and train models with up to 2 billion parameters on a
large 3D asset dataset of 500K diverse objects. Our model generates
high-quality results with text or image conditions, significantly surpassing
existing methods, including recent ones at similar scales. We showcase flexible
output format selection and local 3D editing capabilities which were not
offered by previous models. Code, model, and data will be released.
HumanEdit: A High-Quality Human-Rewarded Dataset for Instruction-based
  Image Editing
Authors: name: Jinbin Bai, name: Wei Chow, name: Ling Yang, name: Xiangtai Li, name: Juncheng Li, name: Hanwang Zhang, name: Shuicheng Yan
Published: 2024-12-05 16:00:59

We present HumanEdit, a high-quality, human-rewarded dataset specifically
designed for instruction-guided image editing, enabling precise and diverse
image manipulations through open-form language instructions. Previous
large-scale editing datasets often incorporate minimal human feedback, leading
to challenges in aligning datasets with human preferences. HumanEdit bridges
this gap by employing human annotators to construct data pairs and
administrators to provide feedback. With meticulously curation, HumanEdit
comprises 5,751 images and requires more than 2,500 hours of human effort
across four stages, ensuring both accuracy and reliability for a wide range of
image editing tasks. The dataset includes six distinct types of editing
instructions: Action, Add, Counting, Relation, Remove, and Replace,
encompassing a broad spectrum of real-world scenarios. All images in the
dataset are accompanied by masks, and for a subset of the data, we ensure that
the instructions are sufficiently detailed to support mask-free editing.
Furthermore, HumanEdit offers comprehensive diversity and high-resolution 1024
times 1024 content sourced from various domains, setting a new versatile
benchmark for instructional image editing datasets. With the aim of advancing
future research and establishing evaluation benchmarks in the field of image
editing, we release HumanEdit at
https://huggingface.co/datasets/BryanW/HumanEdit.
VideoICL: Confidence-based Iterative In-context Learning for
  Out-of-Distribution Video Understanding
Authors: name: Kangsan Kim, name: Geon Park, name: Youngwan Lee, name: Woongyeong Yeo, name: Sung Ju Hwang
Published: 2024-12-03 05:54:43

Recent advancements in video large multimodal models (LMMs) have
significantly improved their video understanding and reasoning capabilities.
However, their performance drops on out-of-distribution (OOD) tasks that are
underrepresented in training data. Traditional methods like fine-tuning on OOD
datasets are impractical due to high computational costs. While In-context
learning (ICL) with demonstration examples has shown promising generalization
performance in language tasks and image-language tasks without fine-tuning,
applying ICL to video-language tasks faces challenges due to the limited
context length in Video LMMs, as videos require longer token lengths. To
address these issues, we propose VideoICL, a novel video in-context learning
framework for OOD tasks that introduces a similarity-based relevant example
selection strategy and a confidence-based iterative inference approach. This
allows to select the most relevant examples and rank them based on similarity,
to be used for inference. If the generated response has low confidence, our
framework selects new examples and performs inference again, iteratively
refining the results until a high-confidence response is obtained. This
approach improves OOD video understanding performance by extending effective
context length without incurring high costs. The experimental results on
multiple benchmarks demonstrate significant performance gains, especially in
domain-specific scenarios, laying the groundwork for broader video
comprehension applications. Code will be released at
https://github.com/KangsanKim07/VideoICL
LumiNet: Latent Intrinsics Meets Diffusion Models for Indoor Scene
  Relighting
Authors: name: Xiaoyan Xing, name: Konrad Groh, name: Sezer Karaoglu, name: Theo Gevers, name: Anand Bhattad
Published: 2024-11-29 18:59:11

We introduce LumiNet, a novel architecture that leverages generative models
and latent intrinsic representations for effective lighting transfer. Given a
source image and a target lighting image, LumiNet synthesizes a relit version
of the source scene that captures the target's lighting. Our approach makes two
key contributions: a data curation strategy from the StyleGAN-based relighting
model for our training, and a modified diffusion-based ControlNet that
processes both latent intrinsic properties from the source image and latent
extrinsic properties from the target image. We further improve lighting
transfer through a learned adaptor (MLP) that injects the target's latent
extrinsic properties via cross-attention and fine-tuning.
  Unlike traditional ControlNet, which generates images with conditional maps
from a single scene, LumiNet processes latent representations from two
different images - preserving geometry and albedo from the source while
transferring lighting characteristics from the target. Experiments demonstrate
that our method successfully transfers complex lighting phenomena including
specular highlights and indirect illumination across scenes with varying
spatial layouts and materials, outperforming existing approaches on challenging
indoor scenes using only images as input.