Orionnnn/hub-stats huggingface dataset:
VLRewardBench: A Challenging Benchmark for Vision-Language Generative
  Reward Models
Authors: name: Lei Li, name: Yuancheng Wei, name: Zhihui Xie, name: Xuqing Yang, name: Yifan Song, name: Peiyi Wang, name: Chenxin An, name: Tianyu Liu, name: Sujian Li, name: Bill Yuchen Lin, name: Lingpeng Kong, name: Qi Liu
Published: 2024-11-26 14:08:34

Vision-language generative reward models (VL-GenRMs) play a crucial role in
aligning and evaluating multimodal AI systems, yet their own evaluation remains
under-explored. Current assessment methods primarily rely on AI-annotated
preference labels from traditional VL tasks, which can introduce biases and
often fail to effectively challenge state-of-the-art models. To address these
limitations, we introduce VL-RewardBench, a comprehensive benchmark spanning
general multimodal queries, visual hallucination detection, and complex
reasoning tasks. Through our AI-assisted annotation pipeline combining sample
selection with human verification, we curate 1,250 high-quality examples
specifically designed to probe model limitations. Comprehensive evaluation
across 16 leading large vision-language models, demonstrates VL-RewardBench's
effectiveness as a challenging testbed, where even GPT-4o achieves only 65.4%
accuracy, and state-of-the-art open-source models such as Qwen2-VL-72B,
struggle to surpass random-guessing. Importantly, performance on VL-RewardBench
strongly correlates (Pearson's r > 0.9) with MMMU-Pro accuracy using Best-of-N
sampling with VL-GenRMs. Analysis experiments uncover three critical insights
for improving VL-GenRMs: (i) models predominantly fail at basic visual
perception tasks rather than reasoning tasks; (ii) inference-time scaling
benefits vary dramatically by model capacity; and (iii) training VL-GenRMs to
learn to judge substantially boosts judgment capability (+14.7% accuracy for a
7B VL-GenRM). We believe VL-RewardBench along with the experimental insights
will become a valuable resource for advancing VL-GenRMs.
MolReFlect: Towards In-Context Fine-grained Alignments between Molecules
  and Texts
Authors: name: Jiatong Li, name: Yunqing Liu, name: Wei Liu, name: Jingdi Le, name: Di Zhang, name: Wenqi Fan, name: Dongzhan Zhou, name: Yuqiang Li, name: Qing Li
Published: 2024-11-22 04:28:56

Molecule discovery is a pivotal research field, impacting everything from the
medicines we take to the materials we use. Recently, Large Language Models
(LLMs) have been widely adopted in molecule understanding and generation, yet
the alignments between molecules and their corresponding captions remain a
significant challenge. Previous endeavours often treat the molecule as a
general SMILES string or molecular graph, neglecting the fine-grained
alignments between the molecular sub-structures and the descriptive textual
phrases, which are crucial for accurate and explainable predictions. In this
case, we introduce MolReFlect, a novel teacher-student framework designed to
contextually perform the molecule-caption alignments in a fine-grained way. Our
approach initially leverages a larger teacher LLM to label the detailed
alignments by directly extracting critical phrases from molecule captions or
SMILES strings and implying them to corresponding sub-structures or
characteristics. To refine these alignments, we propose In-Context Selective
Reflection, which retrieves previous extraction results as context examples for
teacher LLM to reflect and lets a smaller student LLM select from in-context
reflection and previous extraction results. Finally, we enhance the learning
process of the student LLM through Chain-of-Thought In-Context Molecule Tuning,
integrating the fine-grained alignments and the reasoning processes within the
Chain-of-Thought format. Our experimental results demonstrate that MolReFlect
enables LLMs like Mistral-7B to significantly outperform the previous
baselines, achieving SOTA performance on the ChEBI-20 dataset. This advancement
not only enhances the generative capabilities of LLMs in the molecule-caption
translation task, but also contributes to a more explainable framework.
Pathways on the Image Manifold: Image Editing via Video Generation
Authors: name: Noam Rotstein, name: Gal Yona, name: Daniel Silver, name: Roy Velich, name: David Bensaïd, name: Ron Kimmel
Published: 2024-11-25 16:41:45

Recent advances in image editing, driven by image diffusion models, have
shown remarkable progress. However, significant challenges remain, as these
models often struggle to follow complex edit instructions accurately and
frequently compromise fidelity by altering key elements of the original image.
Simultaneously, video generation has made remarkable strides, with models that
effectively function as consistent and continuous world simulators. In this
paper, we propose merging these two fields by utilizing image-to-video models
for image editing. We reformulate image editing as a temporal process, using
pretrained video models to create smooth transitions from the original image to
the desired edit. This approach traverses the image manifold continuously,
ensuring consistent edits while preserving the original image's key aspects.
Our approach achieves state-of-the-art results on text-based image editing,
demonstrating significant improvements in both edit accuracy and image
preservation.
Visual Counter Turing Test (VCT^2): Discovering the Challenges for
  AI-Generated Image Detection and Introducing Visual AI Index (V_AI)
Authors: name: Nasrin Imanpour, name: Shashwat Bajpai, name: Subhankar Ghosh, name: Sainath Reddy Sankepally, name: Abhilekh Borah, name: Hasnat Md Abdullah, name: Nishoak Kosaraju, name: Shreyas Dixit, name: Ashhar Aziz, name: Shwetangshu Biswas, name: Vinija Jain, name: Aman Chadha, name: Amit Sheth, name: Amitava Das
Published: 2024-11-24 06:03:49

The proliferation of AI techniques for image generation, coupled with their
increasing accessibility, has raised significant concerns about the potential
misuse of these images to spread misinformation. Recent AI-generated image
detection (AGID) methods include CNNDetection, NPR, DM Image Detection, Fake
Image Detection, DIRE, LASTED, GAN Image Detection, AIDE, SSP, DRCT, RINE,
OCC-CLIP, De-Fake, and Deep Fake Detection. However, we argue that the current
state-of-the-art AGID techniques are inadequate for effectively detecting
contemporary AI-generated images and advocate for a comprehensive reevaluation
of these methods. We introduce the Visual Counter Turing Test (VCT^2), a
benchmark comprising ~130K images generated by contemporary text-to-image
models (Stable Diffusion 2.1, Stable Diffusion XL, Stable Diffusion 3, DALL-E
3, and Midjourney 6). VCT^2 includes two sets of prompts sourced from tweets by
the New York Times Twitter account and captions from the MS COCO dataset. We
also evaluate the performance of the aforementioned AGID techniques on the
VCT^2 benchmark, highlighting their ineffectiveness in detecting AI-generated
images. As image-generative AI models continue to evolve, the need for a
quantifiable framework to evaluate these models becomes increasingly critical.
To meet this need, we propose the Visual AI Index (V_AI), which assesses
generated images from various visual perspectives, including texture complexity
and object coherence, setting a new standard for evaluating image-generative AI
models. To foster research in this domain, we make our
https://huggingface.co/datasets/anonymous1233/COCO_AI and
https://huggingface.co/datasets/anonymous1233/twitter_AI datasets publicly
available.
SALOVA: Segment-Augmented Long Video Assistant for Targeted Retrieval
  and Routing in Long-Form Video Analysis
Authors: name: Junho Kim, name: Hyunjun Kim, name: Hosu Lee, name: Yong Man Ro
Published: 2024-11-25 08:04:47

Despite advances in Large Multi-modal Models, applying them to long and
untrimmed video content remains challenging due to limitations in context
length and substantial memory overhead. These constraints often lead to
significant information loss and reduced relevance in the model responses. With
the exponential growth of video data across web platforms, understanding
long-form video is crucial for advancing generalized intelligence. In this
paper, we introduce SALOVA: Segment-Augmented LOng Video Assistant, a novel
video-LLM framework designed to enhance the comprehension of lengthy video
content through targeted retrieval process. We address two main challenges to
achieve it: (i) We present the SceneWalk dataset, a high-quality collection of
87.8K long videos, each densely captioned at the segment level to enable models
to capture scene continuity and maintain rich descriptive context. (ii) We
develop robust architectural designs integrating dynamic routing mechanism and
spatio-temporal projector to efficiently retrieve and process relevant video
segments based on user queries. Our framework mitigates the limitations of
current video-LMMs by allowing for precise identification and retrieval of
relevant video segments in response to queries, thereby improving the
contextual relevance of the generated responses. Through extensive experiments,
SALOVA demonstrates enhanced capability in processing complex long-form videos,
showing significant capability to maintain contextual integrity across extended
sequences.
Rethinking Token Reduction in MLLMs: Towards a Unified Paradigm for
  Training-Free Acceleration
Authors: name: Yuhang Han, name: Xuyang Liu, name: Pengxiang Ding, name: Donglin Wang, name: Honggang Chen, name: Qingsen Yan, name: Siteng Huang
Published: 2024-11-26 18:53:51

To accelerate the inference of heavy Multimodal Large Language Models
(MLLMs), this study rethinks the current landscape of training-free token
reduction research. We regret to find that the critical components of existing
methods are tightly intertwined, with their interconnections and effects
remaining unclear for comparison, transfer, and expansion. Therefore, we
propose a unified ''filter-correlate-compress'' paradigm that decomposes the
token reduction into three distinct stages within a pipeline, maintaining
consistent design objectives and elements while allowing for unique
implementations. We additionally demystify the popular works and subsume them
into our paradigm to showcase its universality. Finally, we offer a suite of
methods grounded in the paradigm, striking a balance between speed and accuracy
throughout different phases of the inference. Experimental results across 10
benchmarks indicate that our methods can achieve up to an 82.4% reduction in
FLOPs with a minimal impact on performance, simultaneously surpassing
state-of-the-art training-free methods. Our project page is at
https://ficoco-accelerate.github.io/.
SAR3D: Autoregressive 3D Object Generation and Understanding via
  Multi-scale 3D VQVAE
Authors: name: Yongwei Chen, name: Yushi Lan, name: Shangchen Zhou, name: Tengfei Wang, name: XIngang Pan
Published: 2024-11-25 19:00:05

Autoregressive models have demonstrated remarkable success across various
fields, from large language models (LLMs) to large multimodal models (LMMs) and
2D content generation, moving closer to artificial general intelligence (AGI).
Despite these advances, applying autoregressive approaches to 3D object
generation and understanding remains largely unexplored. This paper introduces
Scale AutoRegressive 3D (SAR3D), a novel framework that leverages a multi-scale
3D vector-quantized variational autoencoder (VQVAE) to tokenize 3D objects for
efficient autoregressive generation and detailed understanding. By predicting
the next scale in a multi-scale latent representation instead of the next
single token, SAR3D reduces generation time significantly, achieving fast 3D
object generation in just 0.82 seconds on an A6000 GPU. Additionally, given the
tokens enriched with hierarchical 3D-aware information, we finetune a
pretrained LLM on them, enabling multimodal comprehension of 3D content. Our
experiments show that SAR3D surpasses current 3D generation methods in both
speed and quality and allows LLMs to interpret and caption 3D models
comprehensively.
MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMs
Authors: name: Chaoyou Fu, name: Yi-Fan Zhang, name: Shukang Yin, name: Bo Li, name: Xinyu Fang, name: Sirui Zhao, name: Haodong Duan, name: Xing Sun, name: Ziwei Liu, name: Liang Wang, name: Caifeng Shan, name: Ran He
Published: 2024-11-22 18:59:54

As a prominent direction of Artificial General Intelligence (AGI), Multimodal
Large Language Models (MLLMs) have garnered increased attention from both
industry and academia. Building upon pre-trained LLMs, this family of models
further develops multimodal perception and reasoning capabilities that are
impressive, such as writing code given a flow chart or creating stories based
on an image. In the development process, evaluation is critical since it
provides intuitive feedback and guidance on improving models. Distinct from the
traditional train-eval-test paradigm that only favors a single task like image
classification, the versatility of MLLMs has spurred the rise of various new
benchmarks and evaluation methods. In this paper, we aim to present a
comprehensive survey of MLLM evaluation, discussing four key aspects: 1) the
summarised benchmarks types divided by the evaluation capabilities, including
foundation capabilities, model self-analysis, and extented applications; 2) the
typical process of benchmark counstruction, consisting of data collection,
annotation, and precautions; 3) the systematic evaluation manner composed of
judge, metric, and toolkit; 4) the outlook for the next benchmark. This work
aims to offer researchers an easy grasp of how to effectively evaluate MLLMs
according to different needs and to inspire better evaluation methods, thereby
driving the progress of MLLM research.
Low-Bit Quantization Favors Undertrained LLMs: Scaling Laws for
  Quantized LLMs with 100T Training Tokens
Authors: name: Xu Ouyang, name: Tao Ge, name: Thomas Hartvigsen, name: Zhisong Zhang, name: Haitao Mi, name: Dong Yu
Published: 2024-11-26 18:57:58

We reveal that low-bit quantization favors undertrained large language models
(LLMs) by observing that models with larger sizes or fewer training tokens
experience less quantization-induced degradation (QiD) when applying low-bit
quantization, whereas smaller models with extensive training tokens suffer
significant QiD. To gain deeper insights into this trend, we study over 1500
quantized LLM checkpoints of various sizes and at different training levels
(undertrained or fully trained) in a controlled setting, deriving scaling laws
for understanding the relationship between QiD and factors such as the number
of training tokens, model size and bit width.
  With the derived scaling laws, we propose a novel perspective that we can use
QiD to measure an LLM's training levels and determine the number of training
tokens required for fully training LLMs of various sizes. Moreover, we use the
scaling laws to predict the quantization performance of different-sized LLMs
trained with 100 trillion tokens. Our projection shows that the low-bit
quantization performance of future models, which are expected to be trained
with over 100 trillion tokens, may NOT be desirable. This poses a potential
challenge for low-bit quantization in the future and highlights the need for
awareness of a model's training level when evaluating low-bit quantization
research. To facilitate future research on this problem, we release all the
1500+ quantized checkpoints used in this work at
https://huggingface.co/Xu-Ouyang.
SketchAgent: Language-Driven Sequential Sketch Generation
Authors: name: Yael Vinker, name: Tamar Rott Shaham, name: Kristine Zheng, name: Alex Zhao, name: Judith E Fan, name: Antonio Torralba
Published: 2024-11-26 18:32:06

Sketching serves as a versatile tool for externalizing ideas, enabling rapid
exploration and visual communication that spans various disciplines. While
artificial systems have driven substantial advances in content creation and
human-computer interaction, capturing the dynamic and abstract nature of human
sketching remains challenging. In this work, we introduce SketchAgent, a
language-driven, sequential sketch generation method that enables users to
create, modify, and refine sketches through dynamic, conversational
interactions. Our approach requires no training or fine-tuning. Instead, we
leverage the sequential nature and rich prior knowledge of off-the-shelf
multimodal large language models (LLMs). We present an intuitive sketching
language, introduced to the model through in-context examples, enabling it to
"draw" using string-based actions. These are processed into vector graphics and
then rendered to create a sketch on a pixel canvas, which can be accessed again
for further tasks. By drawing stroke by stroke, our agent captures the
evolving, dynamic qualities intrinsic to sketching. We demonstrate that
SketchAgent can generate sketches from diverse prompts, engage in
dialogue-driven drawing, and collaborate meaningfully with human users.
ShowUI: One Vision-Language-Action Model for GUI Visual Agent
Authors: name: Kevin Qinghong Lin, name: Linjie Li, name: Difei Gao, name: Zhengyuan Yang, name: Shiwei Wu, name: Zechen Bai, name: Weixian Lei, name: Lijuan Wang, name: Mike Zheng Shou
Published: 2024-11-26 14:29:47

Building Graphical User Interface (GUI) assistants holds significant promise
for enhancing human workflow productivity. While most agents are
language-based, relying on closed-source API with text-rich meta-information
(e.g., HTML or accessibility tree), they show limitations in perceiving UI
visuals as humans do, highlighting the need for GUI visual agents. In this
work, we develop a vision-language-action model in digital world, namely
ShowUI, which features the following innovations: (i) UI-Guided Visual Token
Selection to reduce computational costs by formulating screenshots as an UI
connected graph, adaptively identifying their redundant relationship and serve
as the criteria for token selection during self-attention blocks; (ii)
Interleaved Vision-Language-Action Streaming that flexibly unifies diverse
needs within GUI tasks, enabling effective management of visual-action history
in navigation or pairing multi-turn query-action sequences per screenshot to
enhance training efficiency; (iii) Small-scale High-quality GUI
Instruction-following Datasets by careful data curation and employing a
resampling strategy to address significant data type imbalances. With above
components, ShowUI, a lightweight 2B model using 256K data, achieves a strong
75.1% accuracy in zero-shot screenshot grounding. Its UI-guided token selection
further reduces 33% of redundant visual tokens during training and speeds up
the performance by 1.4x. Navigation experiments across web Mind2Web, mobile
AITW, and online MiniWob environments further underscore the effectiveness and
potential of our model in advancing GUI visual agents. The models are available
at https://github.com/showlab/ShowUI.
AnchorCrafter: Animate CyberAnchors Saling Your Products via
  Human-Object Interacting Video Generation
Authors: name: Ziyi Xu, name: Ziyao Huang, name: Juan Cao, name: Yong Zhang, name: Xiaodong Cun, name: Qing Shuai, name: Yuchen Wang, name: Linchao Bao, name: Jintao Li, name: Fan Tang
Published: 2024-11-26 12:42:13

The automatic generation of anchor-style product promotion videos presents
promising opportunities in online commerce, advertising, and consumer
engagement. However, this remains a challenging task despite significant
advancements in pose-guided human video generation. In addressing this
challenge, we identify the integration of human-object interactions (HOI) into
pose-guided human video generation as a core issue. To this end, we introduce
AnchorCrafter, a novel diffusion-based system designed to generate 2D videos
featuring a target human and a customized object, achieving high visual
fidelity and controllable interactions. Specifically, we propose two key
innovations: the HOI-appearance perception, which enhances object appearance
recognition from arbitrary multi-view perspectives and disentangles object and
human appearance, and the HOI-motion injection, which enables complex
human-object interactions by overcoming challenges in object trajectory
conditioning and inter-occlusion management. Additionally, we introduce the
HOI-region reweighting loss, a training objective that enhances the learning of
object details. Extensive experiments demonstrate that our proposed system
outperforms existing methods in preserving object appearance and shape
awareness, while simultaneously maintaining consistency in human appearance and
motion. Project page: https://cangcz.github.io/Anchor-Crafter/
FINECAPTION: Compositional Image Captioning Focusing on Wherever You
  Want at Any Granularity
Authors: name: Hang Hua, name: Qing Liu, name: Lingzhi Zhang, name: Jing Shi, name: Zhifei Zhang, name: Yilin Wang, name: Jianming Zhang, name: Jiebo Luo
Published: 2024-11-23 02:20:32

The advent of large Vision-Language Models (VLMs) has significantly advanced
multimodal tasks, enabling more sophisticated and accurate reasoning across
various applications, including image and video captioning, visual question
answering, and cross-modal retrieval. Despite their superior capabilities, VLMs
struggle with fine-grained image regional composition information perception.
Specifically, they have difficulty accurately aligning the segmentation masks
with the corresponding semantics and precisely describing the compositional
aspects of the referred regions.
  However, compositionality - the ability to understand and generate novel
combinations of known visual and textual components - is critical for
facilitating coherent reasoning and understanding across modalities by VLMs. To
address this issue, we propose FINECAPTION, a novel VLM that can recognize
arbitrary masks as referential inputs and process high-resolution images for
compositional image captioning at different granularity levels. To support this
endeavor, we introduce COMPOSITIONCAP, a new dataset for multi-grained region
compositional image captioning, which introduces the task of compositional
attribute-aware regional image captioning.
  Empirical results demonstrate the effectiveness of our proposed model
compared to other state-of-the-art VLMs. Additionally, we analyze the
capabilities of current VLMs in recognizing various visual prompts for
compositional region image captioning, highlighting areas for improvement in
VLM design and training.
Learning 3D Representations from Procedural 3D Programs
Authors: name: Xuweiyi Chen, name: Zezhou Cheng
Published: 2024-11-25 18:59:57

Self-supervised learning has emerged as a promising approach for acquiring
transferable 3D representations from unlabeled 3D point clouds. Unlike 2D
images, which are widely accessible, acquiring 3D assets requires specialized
expertise or professional 3D scanning equipment, making it difficult to scale
and raising copyright concerns. To address these challenges, we propose
learning 3D representations from procedural 3D programs that automatically
generate 3D shapes using simple primitives and augmentations.
  Remarkably, despite lacking semantic content, the 3D representations learned
from this synthesized dataset perform on par with state-of-the-art
representations learned from semantically recognizable 3D models (e.g.,
airplanes) across various downstream 3D tasks, including shape classification,
part segmentation, and masked point cloud completion. Our analysis further
suggests that current self-supervised learning methods primarily capture
geometric structures rather than high-level semantics.