Orionnnn/hub-stats huggingface dataset:
VLsI: Verbalized Layers-to-Interactions from Large to Small Vision
  Language Models
Authors: name: Byung-Kwan Lee, name: Ryo Hachiuma, name: Yu-Chiang Frank Wang, name: Yong Man Ro, name: Yueh-Hua Wu
Published: 2024-12-02 18:58:25

The recent surge in high-quality visual instruction tuning samples from
closed-source vision-language models (VLMs) such as GPT-4V has accelerated the
release of open-source VLMs across various model sizes. However, scaling VLMs
to improve performance using larger models brings significant computational
challenges, especially for deployment on resource-constrained devices like
mobile platforms and robots. To address this, we propose VLsI: Verbalized
Layers-to-Interactions, a new VLM family in 2B and 7B model sizes, which
prioritizes efficiency without compromising accuracy. VLsI leverages a unique,
layer-wise distillation process, introducing intermediate "verbalizers" that
map features from each layer to natural language space, allowing smaller VLMs
to flexibly align with the reasoning processes of larger VLMs. This approach
mitigates the training instability often encountered in output imitation and
goes beyond typical final-layer tuning by aligning the small VLMs' layer-wise
progression with that of the large ones. We validate VLsI across ten
challenging vision-language benchmarks, achieving notable performance gains
(11.0% for 2B and 17.4% for 7B) over GPT-4V without the need for model scaling,
merging, or architectural changes.
Collaborative Instance Navigation: Leveraging Agent Self-Dialogue to
  Minimize User Input
Authors: name: Francesco Taioli, name: Edoardo Zorzi, name: Gianni Franchi, name: Alberto Castellini, name: Alessandro Farinelli, name: Marco Cristani, name: Yiming Wang
Published: 2024-12-02 08:16:38

Existing embodied instance goal navigation tasks, driven by natural language,
assume human users to provide complete and nuanced instance descriptions prior
to the navigation, which can be impractical in the real world as human
instructions might be brief and ambiguous. To bridge this gap, we propose a new
task, Collaborative Instance Navigation (CoIN), with dynamic agent-human
interaction during navigation to actively resolve uncertainties about the
target instance in natural, template-free, open-ended dialogues. To address
CoIN, we propose a novel method, Agent-user Interaction with UncerTainty
Awareness (AIUTA), leveraging the perception capability of Vision Language
Models (VLMs) and the capability of Large Language Models (LLMs). First, upon
object detection, a Self-Questioner model initiates a self-dialogue to obtain a
complete and accurate observation description, while a novel uncertainty
estimation technique mitigates inaccurate VLM perception. Then, an Interaction
Trigger module determines whether to ask a question to the user, continue or
halt navigation, minimizing user input. For evaluation, we introduce
CoIN-Bench, a benchmark supporting both real and simulated humans. AIUTA
achieves competitive performance in instance navigation against
state-of-the-art methods, demonstrating great flexibility in handling user
inputs.
Switti: Designing Scale-Wise Transformers for Text-to-Image Synthesis
Authors: name: Anton Voronov, name: Denis Kuznedelev, name: Mikhail Khoroshikh, name: Valentin Khrulkov, name: Dmitry Baranchuk
Published: 2024-12-02 18:57:41

This work presents Switti, a scale-wise transformer for text-to-image
generation. Starting from existing next-scale prediction AR models, we first
explore them for T2I generation and propose architectural modifications to
improve their convergence and overall performance. We then observe that
self-attention maps of our pretrained scale-wise AR model exhibit weak
dependence on preceding scales. Based on this insight, we propose a non-AR
counterpart facilitating {sim}11% faster sampling and lower memory usage
while also achieving slightly better generation quality.Furthermore, we reveal
that classifier-free guidance at high-resolution scales is often unnecessary
and can even degrade performance. %may be not only unnecessary but potentially
detrimental. By disabling guidance at these scales, we achieve an additional
sampling acceleration of {sim}20% and improve the generation of
fine-grained details. Extensive human preference studies and automated
evaluations show that Switti outperforms existing T2I AR models and competes
with state-of-the-art T2I diffusion models while being up to 7{times}
faster.
VLSBench: Unveiling Visual Leakage in Multimodal Safety
Authors: name: Xuhao Hu, name: Dongrui Liu, name: Hao Li, name: Xuanjing Huang, name: Jing Shao
Published: 2024-11-29 18:56:37

Safety concerns of Multimodal large language models (MLLMs) have gradually
become an important problem in various applications. Surprisingly, previous
works indicate a counter-intuitive phenomenon that using textual unlearning to
align MLLMs achieves comparable safety performances with MLLMs trained with
image-text pairs. To explain such a counter-intuitive phenomenon, we discover a
visual safety information leakage (VSIL) problem in existing multimodal safety
benchmarks, i.e., the potentially risky and sensitive content in the image has
been revealed in the textual query. In this way, MLLMs can easily refuse these
sensitive text-image queries according to textual queries. However, image-text
pairs without VSIL are common in real-world scenarios and are overlooked by
existing multimodal safety benchmarks. To this end, we construct multimodal
visual leakless safety benchmark (VLSBench) preventing visual safety leakage
from image to textual query with 2.4k image-text pairs. Experimental results
indicate that VLSBench poses a significant challenge to both open-source and
close-source MLLMs, including LLaVA, Qwen2-VL, Llama3.2-Vision, and GPT-4o.
This study demonstrates that textual alignment is enough for multimodal safety
scenarios with VSIL, while multimodal alignment is a more promising solution
for multimodal safety scenarios without VSIL. Please see our code and data at:
http://hxhcreate.github.io/VLSBench
INCLUDE: Evaluating Multilingual Language Understanding with Regional
  Knowledge
Authors: name: Angelika Romanou, name: Negar Foroutan, name: Anna Sotnikova, name: Zeming Chen, name: Sree Harsha Nelaturu, name: Shivalika Singh, name: Rishabh Maheshwary, name: Micol Altomare, name: Mohamed A. Haggag, name: Snegha A, name: Alfonso Amayuelas, name: Azril Hafizi Amirudin, name: Viraat Aryabumi, name: Danylo Boiko, name: Michael Chang, name: Jenny Chim, name: Gal Cohen, name: Aditya Kumar Dalmia, name: Abraham Diress, name: Sharad Duwal, name: Daniil Dzenhaliou, name: Daniel Fernando Erazo Florez, name: Fabian Farestam, name: Joseph Marvin Imperial, name: Shayekh Bin Islam, name: Perttu Isotalo, name: Maral Jabbarishiviari, name: Börje F. Karlsson, name: Eldar Khalilov, name: Christopher Klamm, name: Fajri Koto, name: Dominik Krzemiński, name: Gabriel Adriano de Melo, name: Syrielle Montariol, name: Yiyang Nan, name: Joel Niklaus, name: Jekaterina Novikova, name: Johan Samir Obando Ceron, name: Debjit Paul, name: Esther Ploeger, name: Jebish Purbey, name: Swati Rajwal, name: Selvan Sunitha Ravi, name: Sara Rydell, name: Roshan Santhosh, name: Drishti Sharma, name: Marjana Prifti Skenduli, name: Arshia Soltani Moakhar, name: Bardia Soltani Moakhar, name: Ran Tamir, name: Ayush Kumar Tarun, name: Azmine Toushik Wasi, name: Thenuka Ovin Weerasinghe, name: Serhan Yilmaz, name: Mike Zhang, name: Imanol Schlag, name: Marzieh Fadaee, name: Sara Hooker, name: Antoine Bosselut
Published: 2024-11-29 16:03:14

The performance differential of large language models (LLM) between languages
hinders their effective deployment in many regions, inhibiting the potential
economic and societal value of generative AI tools in many communities.
However, the development of functional LLMs in many languages (\ie,
multilingual LLMs) is bottlenecked by the lack of high-quality evaluation
resources in languages other than English. Moreover, current practices in
multilingual benchmark construction often translate English resources, ignoring
the regional and cultural knowledge of the environments in which multilingual
systems would be used. In this work, we construct an evaluation suite of
197,243 QA pairs from local exam sources to measure the capabilities of
multilingual LLMs in a variety of regional contexts. Our novel resource,
INCLUDE, is a comprehensive knowledge- and reasoning-centric benchmark across
44 written languages that evaluates multilingual LLMs for performance in the
actual language environments where they would be deployed.
SOLAMI: Social Vision-Language-Action Modeling for Immersive Interaction
  with 3D Autonomous Characters
Authors: name: Jianping Jiang, name: Weiye Xiao, name: Zhengyu Lin, name: Huaizhong Zhang, name: Tianxiang Ren, name: Yang Gao, name: Zhiqian Lin, name: Zhongang Cai, name: Lei Yang, name: Ziwei Liu
Published: 2024-11-29 18:53:40

Human beings are social animals. How to equip 3D autonomous characters with
similar social intelligence that can perceive, understand and interact with
humans remains an open yet foundamental problem. In this paper, we introduce
SOLAMI, the first end-to-end Social vision-Language-Action (VLA) Modeling
framework for Immersive interaction with 3D autonomous characters.
Specifically, SOLAMI builds 3D autonomous characters from three aspects: (1)
Social VLA Architecture: We propose a unified social VLA framework to generate
multimodal response (speech and motion) based on the user's multimodal input to
drive the character for social interaction. (2) Interactive Multimodal Data: We
present SynMSI, a synthetic multimodal social interaction dataset generated by
an automatic pipeline using only existing motion datasets to address the issue
of data scarcity. (3) Immersive VR Interface: We develop a VR interface that
enables users to immersively interact with these characters driven by various
architectures. Extensive quantitative experiments and user studies demonstrate
that our framework leads to more precise and natural character responses (in
both speech and motion) that align with user expectations with lower latency.
GATE OpenING: A Comprehensive Benchmark for Judging Open-ended
  Interleaved Image-Text Generation
Authors: name: Pengfei Zhou, name: Xiaopeng Peng, name: Jiajun Song, name: Chuanhao Li, name: Zhaopan Xu, name: Yue Yang, name: Ziyao Guo, name: Hao Zhang, name: Yuqi Lin, name: Yefei He, name: Lirui Zhao, name: Shuo Liu, name: Tianhua Li, name: Yuxuan Xie, name: Xiaojun Chang, name: Yu Qiao, name: Wenqi Shao, name: Kaipeng Zhang
Published: 2024-11-27 16:39:04

Multimodal Large Language Models (MLLMs) have made significant strides in
visual understanding and generation tasks. However, generating interleaved
image-text content remains a challenge, which requires integrated multimodal
understanding and generation abilities. While the progress in unified models
offers new solutions, existing benchmarks are insufficient for evaluating these
methods due to data size and diversity limitations. To bridge this gap, we
introduce GATE OpenING (OpenING), a comprehensive benchmark comprising 5,400
high-quality human-annotated instances across 56 real-world tasks. OpenING
covers diverse daily scenarios such as travel guide, design, and brainstorming,
offering a robust platform for challenging interleaved generation methods. In
addition, we present IntJudge, a judge model for evaluating open-ended
multimodal generation methods. Trained with a novel data pipeline, our IntJudge
achieves an agreement rate of 82. 42% with human judgments, outperforming
GPT-based evaluators by 11.34%. Extensive experiments on OpenING reveal that
current interleaved generation methods still have substantial room for
improvement. Key findings on interleaved image-text generation are further
presented to guide the development of next-generation models. The OpenING is
open-sourced at https://opening.github.io.
FLOAT: Generative Motion Latent Flow Matching for Audio-driven Talking
  Portrait
Authors: name: Taekyung Ki, name: Dongchan Min, name: Gyoungsu Chae
Published: 2024-12-02 02:50:07

With the rapid advancement of diffusion-based generative models, portrait
image animation has achieved remarkable results. However, it still faces
challenges in temporally consistent video generation and fast sampling due to
its iterative sampling nature. This paper presents FLOAT, an audio-driven
talking portrait video generation method based on flow matching generative
model. We shift the generative modeling from the pixel-based latent space to a
learned motion latent space, enabling efficient design of temporally consistent
motion. To achieve this, we introduce a transformer-based vector field
predictor with a simple yet effective frame-wise conditioning mechanism.
Additionally, our method supports speech-driven emotion enhancement, enabling a
natural incorporation of expressive motions. Extensive experiments demonstrate
that our method outperforms state-of-the-art audio-driven talking portrait
methods in terms of visual quality, motion fidelity, and efficiency.
A Simple and Provable Scaling Law for the Test-Time Compute of Large
  Language Models
Authors: name: Yanxi Chen, name: Xuchen Pan, name: Yaliang Li, name: Bolin Ding, name: Jingren Zhou
Published: 2024-11-29 05:29:47

We propose a general two-stage algorithm that enjoys a provable scaling law
for the test-time compute of large language models (LLMs). Given an input
problem, the proposed algorithm first generates N candidate solutions, and
then chooses the best one via a multiple-round knockout tournament where each
pair of candidates are compared for K times and only the winners move on to
the next round. In a minimalistic implementation, both stages can be executed
with a black-box LLM alone and nothing else (e.g., no external verifier or
reward model), and a total of N times (K + 1) highly parallelizable LLM
calls are needed for solving an input problem. Assuming that a generated
candidate solution is correct with probability p_{gen} > 0 and a
comparison between a pair of correct and incorrect solutions identifies the
right winner with probability p_{comp} > 0.5 (i.e., better than a
random guess), we prove theoretically that the failure probability of the
proposed algorithm decays to zero exponentially with respect to N and K:
$P(final output is incorrect) le (1 - p_{gen})^N +
lceil log_2 N rceil e^{-2 K (p_{comp} - 0.5)^2}.$ Our empirical
results with the challenging MMLU-Pro benchmark validate the technical
assumptions, as well as the efficacy of the proposed algorithm and the gains
from scaling up its test-time compute.
VISTA: Enhancing Long-Duration and High-Resolution Video Understanding
  by Video Spatiotemporal Augmentation
Authors: name: Weiming Ren, name: Huan Yang, name: Jie Min, name: Cong Wei, name: Wenhu Chen
Published: 2024-12-01 18:27:28

Current large multimodal models (LMMs) face significant challenges in
processing and comprehending long-duration or high-resolution videos, which is
mainly due to the lack of high-quality datasets. To address this issue from a
data-centric perspective, we propose VISTA, a simple yet effective Video
Spatiotemporal Augmentation framework that synthesizes long-duration and
high-resolution video instruction-following pairs from existing video-caption
datasets. VISTA spatially and temporally combines videos to create new
synthetic videos with extended durations and enhanced resolutions, and
subsequently produces question-answer pairs pertaining to these newly
synthesized videos. Based on this paradigm, we develop seven video augmentation
methods and curate VISTA-400K, a video instruction-following dataset aimed at
enhancing long-duration and high-resolution video understanding. Finetuning
various video LMMs on our data resulted in an average improvement of 3.3%
across four challenging benchmarks for long-video understanding. Furthermore,
we introduce the first comprehensive high-resolution video understanding
benchmark HRVideoBench, on which our finetuned models achieve a 6.5%
performance gain. These results highlight the effectiveness of our framework.
o1-Coder: an o1 Replication for Coding
Authors: name: Yuxiang Zhang, name: Shangxi Wu, name: Yuqi Yang, name: Jiangming Shu, name: Jinlin Xiao, name: Chao Kong, name: Jitao Sang
Published: 2024-11-29 07:19:56

The technical report introduces O1-CODER, an attempt to replicate OpenAI's o1
model with a focus on coding tasks. It integrates reinforcement learning (RL)
and Monte Carlo Tree Search (MCTS) to enhance the model's System-2 thinking
capabilities. The framework includes training a Test Case Generator (TCG) for
standardized code testing, using MCTS to generate code data with reasoning
processes, and iteratively fine-tuning the policy model to initially produce
pseudocode, followed by the generation of the full code. The report also
addresses the opportunities and challenges in deploying o1-like models in
real-world applications, suggesting transitioning to the System-2 paradigm and
highlighting the imperative for environment state updates. Updated model
progress and experimental results will be reported in subsequent versions. All
source code, curated datasets, as well as the derived models will be disclosed
at https://github.com/ADaM-BJTU/O1-CODER .
Efficient Track Anything
Authors: name: Yunyang Xiong, name: Chong Zhou, name: Xiaoyu Xiang, name: Lemeng Wu, name: Chenchen Zhu, name: Zechun Liu, name: Saksham Suri, name: Balakrishnan Varadarajan, name: Ramya Akula, name: Forrest Iandola, name: Raghuraman Krishnamoorthi, name: Bilge Soran, name: Vikas Chandra
Published: 2024-11-28 05:52:10

Segment Anything Model 2 (SAM 2) has emerged as a powerful tool for video
object segmentation and tracking anything. Key components of SAM 2 that drive
the impressive video object segmentation performance include a large multistage
image encoder for frame feature extraction and a memory mechanism that stores
memory contexts from past frames to help current frame segmentation. The high
computation complexity of multistage image encoder and memory module has
limited its applications in real-world tasks, e.g., video object segmentation
on mobile devices. To address this limitation, we propose EfficientTAMs,
lightweight track anything models that produce high-quality results with low
latency and model size. Our idea is based on revisiting the plain,
nonhierarchical Vision Transformer (ViT) as an image encoder for video object
segmentation, and introducing an efficient memory module, which reduces the
complexity for both frame feature extraction and memory computation for current
frame segmentation. We take vanilla lightweight ViTs and efficient memory
module to build EfficientTAMs, and train the models on SA-1B and SA-V datasets
for video object segmentation and track anything tasks. We evaluate on multiple
video segmentation benchmarks including semi-supervised VOS and promptable
video segmentation, and find that our proposed EfficientTAM with vanilla ViT
perform comparably to SAM 2 model (HieraB+SAM 2) with ~2x speedup on A100 and
~2.4x parameter reduction. On segment anything image tasks, our EfficientTAMs
also perform favorably over original SAM with ~20x speedup on A100 and ~20x
parameter reduction. On mobile devices such as iPhone 15 Pro Max, our
EfficientTAMs can run at ~10 FPS for performing video object segmentation with
reasonable quality, highlighting the capability of small models for on-device
video object segmentation applications.
TinyFusion: Diffusion Transformers Learned Shallow
Authors: name: Gongfan Fang, name: Kunjun Li, name: Xinyin Ma, name: Xinchao Wang
Published: 2024-12-02 07:05:39

Diffusion Transformers have demonstrated remarkable capabilities in image
generation but often come with excessive parameterization, resulting in
considerable inference overhead in real-world applications. In this work, we
present TinyFusion, a depth pruning method designed to remove redundant layers
from diffusion transformers via end-to-end learning. The core principle of our
approach is to create a pruned model with high recoverability, allowing it to
regain strong performance after fine-tuning. To accomplish this, we introduce a
differentiable sampling technique to make pruning learnable, paired with a
co-optimized parameter to simulate future fine-tuning. While prior works focus
on minimizing loss or error after pruning, our method explicitly models and
optimizes the post-fine-tuning performance of pruned models. Experimental
results indicate that this learnable paradigm offers substantial benefits for
layer pruning of diffusion transformers, surpassing existing importance-based
and error-based methods. Additionally, TinyFusion exhibits strong
generalization across diverse architectures, such as DiTs, MARs, and SiTs.
Experiments with DiT-XL show that TinyFusion can craft a shallow diffusion
transformer at less than 7% of the pre-training cost, achieving a 2times
speedup with an FID score of 2.86, outperforming competitors with comparable
efficiency. Code is available at https://github.com/VainF/TinyFusion.
X-Prompt: Towards Universal In-Context Image Generation in
  Auto-Regressive Vision Language Foundation Models
Authors: name: Zeyi Sun, name: Ziyang Chu, name: Pan Zhang, name: Tong Wu, name: Xiaoyi Dong, name: Yuhang Zang, name: Yuanjun Xiong, name: Dahua Lin, name: Jiaqi Wang
Published: 2024-12-02 18:59:26

In-context generation is a key component of large language models' (LLMs)
open-task generalization capability. By leveraging a few examples as context,
LLMs can perform both in-domain and out-of-domain tasks. Recent advancements in
auto-regressive vision-language models (VLMs) built upon LLMs have showcased
impressive performance in text-to-image generation. However, the potential of
in-context learning for general image generation tasks remains largely
unexplored. To address this, we introduce X-Prompt, a purely auto-regressive
large-vision language model designed to deliver competitive performance across
a wide range of both seen and unseen image generation tasks, all within a
unified in-context learning framework. X-Prompt incorporates a specialized
design that efficiently compresses valuable features from in-context examples,
supporting longer in-context token sequences and improving its ability to
generalize to unseen tasks. A unified training task for both text and image
prediction enables X-Prompt to handle general image generation with enhanced
task awareness from in-context examples. Extensive experiments validate the
model's performance across diverse seen image generation tasks and its capacity
to generalize to previously unseen tasks.
Steering Rectified Flow Models in the Vector Field for Controlled Image
  Generation
Authors: name: Maitreya Patel, name: Song Wen, name: Dimitris N. Metaxas, name: Yezhou Yang
Published: 2024-11-27 19:04:40

Diffusion models (DMs) excel in photorealism, image editing, and solving
inverse problems, aided by classifier-free guidance and image inversion
techniques. However, rectified flow models (RFMs) remain underexplored for
these tasks. Existing DM-based methods often require additional training, lack
generalization to pretrained latent models, underperform, and demand
significant computational resources due to extensive backpropagation through
ODE solvers and inversion processes. In this work, we first develop a
theoretical and empirical understanding of the vector field dynamics of RFMs in
efficiently guiding the denoising trajectory. Our findings reveal that we can
navigate the vector field in a deterministic and gradient-free manner.
Utilizing this property, we propose FlowChef, which leverages the vector field
to steer the denoising trajectory for controlled image generation tasks,
facilitated by gradient skipping. FlowChef is a unified framework for
controlled image generation that, for the first time, simultaneously addresses
classifier guidance, linear inverse problems, and image editing without the
need for extra training, inversion, or intensive backpropagation. Finally, we
perform extensive evaluations and show that FlowChef significantly outperforms
baselines in terms of performance, memory, and time requirements, achieving new
state-of-the-art results. Project Page: https://flowchef.github.io.