Orionnnn/hub-stats huggingface dataset:
AV-Odyssey Bench: Can Your Multimodal LLMs Really Understand
  Audio-Visual Information?
Authors: name: Kaixiong Gong, name: Kaituo Feng, name: Bohao Li, name: Yibing Wang, name: Mofan Cheng, name: Shijia Yang, name: Jiaming Han, name: Benyou Wang, name: Yutong Bai, name: Zhuoran Yang, name: Xiangyu Yue
Published: 2024-12-03 17:41:23

Recently, multimodal large language models (MLLMs), such as GPT-4o, Gemini
1.5 Pro, and Reka Core, have expanded their capabilities to include vision and
audio modalities. While these models demonstrate impressive performance across
a wide range of audio-visual applications, our proposed DeafTest reveals that
MLLMs often struggle with simple tasks humans find trivial: 1) determining
which of two sounds is louder, and 2) determining which of two sounds has a
higher pitch. Motivated by these observations, we introduce AV-Odyssey Bench, a
comprehensive audio-visual benchmark designed to assess whether those MLLMs can
truly understand the audio-visual information. This benchmark encompasses 4,555
carefully crafted problems, each incorporating text, visual, and audio
components. To successfully infer answers, models must effectively leverage
clues from both visual and audio inputs. To ensure precise and objective
evaluation of MLLM responses, we have structured the questions as
multiple-choice, eliminating the need for human evaluation or LLM-assisted
assessment. We benchmark a series of closed-source and open-source models and
summarize the observations. By revealing the limitations of current models, we
aim to provide useful insight for future dataset collection and model
development.
Scaling Image Tokenizers with Grouped Spherical Quantization
Authors: name: Jiangtao Wang, name: Zhen Qin, name: Yifan Zhang, name: Vincent Tao Hu, name: Björn Ommer, name: Rania Briq, name: Stefan Kesselheim
Published: 2024-12-03 18:01:45

Vision tokenizers have gained a lot of attraction due to their scalability
and compactness; previous works depend on old-school GAN-based hyperparameters,
biased comparisons, and a lack of comprehensive analysis of the scaling
behaviours. To tackle those issues, we introduce Grouped Spherical Quantization
(GSQ), featuring spherical codebook initialization and lookup regularization to
constrain codebook latent to a spherical surface. Our empirical analysis of
image tokenizer training strategies demonstrates that GSQ-GAN achieves superior
reconstruction quality over state-of-the-art methods with fewer training
iterations, providing a solid foundation for scaling studies. Building on this,
we systematically examine the scaling behaviours of GSQ, specifically in latent
dimensionality, codebook size, and compression ratios, and their impact on
model performance. Our findings reveal distinct behaviours at high and low
spatial compression levels, underscoring challenges in representing
high-dimensional latent spaces. We show that GSQ can restructure
high-dimensional latent into compact, low-dimensional spaces, thus enabling
efficient scaling with improved quality. As a result, GSQ-GAN achieves a 16x
down-sampling with a reconstruction FID (rFID) of 0.50.
LSceneLLM: Enhancing Large 3D Scene Understanding Using Adaptive Visual
  Preferences
Authors: name: Hongyan Zhi, name: Peihao Chen, name: Junyan Li, name: Shuailei Ma, name: Xinyu Sun, name: Tianhang Xiang, name: Yinjie Lei, name: Mingkui Tan, name: Chuang Gan
Published: 2024-12-02 09:07:57

Research on 3D Vision-Language Models (3D-VLMs) is gaining increasing
attention, which is crucial for developing embodied AI within 3D scenes, such
as visual navigation and embodied question answering. Due to the high density
of visual features, especially in large 3D scenes, accurately locating
task-relevant visual information is challenging. Existing works attempt to
segment all objects and consider their features as scene representations.
However, these task-agnostic object features include much redundant information
and missing details for the task-relevant area. To tackle these problems, we
propose LSceneLLM, an adaptive framework that automatically identifies
task-relevant areas by leveraging LLM's visual preference for different tasks,
followed by a plug-and-play scene magnifier module to capture fine-grained
details in focused areas. Specifically, a dense token selector examines the
attention map of LLM to identify visual preferences for the instruction input.
It then magnifies fine-grained details of the focusing area. An adaptive
self-attention module is leveraged to fuse the coarse-grained and selected
fine-grained visual information. To comprehensively evaluate the large scene
understanding ability of 3D-VLMs, we further introduce a cross-room
understanding benchmark, XR-Scene, which contains a series of large scene
understanding tasks including XR-QA, XR-EmbodiedPlanning, and XR-SceneCaption.
Experiments show that our method surpasses existing methods on both large scene
understanding and existing scene understanding benchmarks. Plunging our scene
magnifier module into the existing 3D-VLMs also brings significant improvement.
VideoGen-of-Thought: A Collaborative Framework for Multi-Shot Video
  Generation
Authors: name: Mingzhe Zheng, name: Yongqi Xu, name: Haojian Huang, name: Xuran Ma, name: Yexin Liu, name: Wenjie Shu, name: Yatian Pang, name: Feilong Tang, name: Qifeng Chen, name: Harry Yang, name: Ser-Nam Lim
Published: 2024-12-03 08:33:50

Current video generation models excel at generating short clips but still
struggle with creating multi-shot, movie-like videos. Existing models trained
on large-scale data on the back of rich computational resources are
unsurprisingly inadequate for maintaining a logical storyline and visual
consistency across multiple shots of a cohesive script since they are often
trained with a single-shot objective. To this end, we propose
VideoGen-of-Thought (VGoT), a collaborative and training-free architecture
designed specifically for multi-shot video generation. VGoT is designed with
three goals in mind as follows. Multi-Shot Video Generation: We divide the
video generation process into a structured, modular sequence, including (1)
Script Generation, which translates a curt story into detailed prompts for each
shot; (2) Keyframe Generation, responsible for creating visually consistent
keyframes faithful to character portrayals; and (3) Shot-Level Video
Generation, which transforms information from scripts and keyframes into shots;
(4) Smoothing Mechanism that ensures a consistent multi-shot output. Reasonable
Narrative Design: Inspired by cinematic scriptwriting, our prompt generation
approach spans five key domains, ensuring logical consistency, character
development, and narrative flow across the entire video. Cross-Shot
Consistency: We ensure temporal and identity consistency by leveraging
identity-preserving (IP) embeddings across shots, which are automatically
created from the narrative. Additionally, we incorporate a cross-shot smoothing
mechanism, which integrates a reset boundary that effectively combines latent
features from adjacent shots, resulting in smooth transitions and maintaining
visual coherence throughout the video. Our experiments demonstrate that VGoT
surpasses existing video generation methods in producing high-quality,
coherent, multi-shot videos.
Critical Tokens Matter: Token-Level Contrastive Estimation Enhence LLM's
  Reasoning Capability
Authors: name: Zicheng Lin, name: Tian Liang, name: Jiahao Xu, name: Xing Wang, name: Ruilin Luo, name: Chufan Shi, name: Siheng Li, name: Yujiu Yang, name: Zhaopeng Tu
Published: 2024-11-29 18:58:22

Large Language Models (LLMs) have exhibited remarkable performance on
reasoning tasks. They utilize autoregressive token generation to construct
reasoning trajectories, enabling the development of a coherent chain of
thought. In this work, we explore the impact of individual tokens on the final
outcomes of reasoning tasks. We identify the existence of ``critical tokens''
that lead to incorrect reasoning trajectories in LLMs. Specifically, we find
that LLMs tend to produce positive outcomes when forced to decode other tokens
instead of critical tokens. Motivated by this observation, we propose a novel
approach - cDPO - designed to automatically recognize and conduct token-level
rewards for the critical tokens during the alignment process. Specifically, we
develop a contrastive estimation approach to automatically identify critical
tokens. It is achieved by comparing the generation likelihood of positive and
negative models. To achieve this, we separately fine-tune the positive and
negative models on various reasoning trajectories, consequently, they are
capable of identifying identify critical tokens within incorrect trajectories
that contribute to erroneous outcomes. Moreover, to further align the model
with the critical token information during the alignment process, we extend the
conventional DPO algorithms to token-level DPO and utilize the differential
likelihood from the aforementioned positive and negative model as important
weight for token-level DPO learning.Experimental results on GSM8K and MATH500
benchmarks with two-widely used models Llama-3 (8B and 70B) and deepseek-math
(7B) demonstrate the effectiveness of the propsoed approach cDPO.
HUGSIM: A Real-Time, Photo-Realistic and Closed-Loop Simulator for
  Autonomous Driving
Authors: name: Hongyu Zhou, name: Longzhong Lin, name: Jiabao Wang, name: Yichong Lu, name: Dongfeng Bai, name: Bingbing Liu, name: Yue Wang, name: Andreas Geiger, name: Yiyi Liao
Published: 2024-12-02 17:07:59

In the past few decades, autonomous driving algorithms have made significant
progress in perception, planning, and control. However, evaluating individual
components does not fully reflect the performance of entire systems,
highlighting the need for more holistic assessment methods. This motivates the
development of HUGSIM, a closed-loop, photo-realistic, and real-time simulator
for evaluating autonomous driving algorithms. We achieve this by lifting
captured 2D RGB images into the 3D space via 3D Gaussian Splatting, improving
the rendering quality for closed-loop scenarios, and building the closed-loop
environment. In terms of rendering, We tackle challenges of novel view
synthesis in closed-loop scenarios, including viewpoint extrapolation and
360-degree vehicle rendering. Beyond novel view synthesis, HUGSIM further
enables the full closed simulation loop, dynamically updating the ego and actor
states and observations based on control commands. Moreover, HUGSIM offers a
comprehensive benchmark across more than 70 sequences from KITTI-360, Waymo,
nuScenes, and PandaSet, along with over 400 varying scenarios, providing a fair
and realistic evaluation platform for existing autonomous driving algorithms.
HUGSIM not only serves as an intuitive evaluation benchmark but also unlocks
the potential for fine-tuning autonomous driving algorithms in a photorealistic
closed-loop setting.
AMO Sampler: Enhancing Text Rendering with Overshooting
Authors: name: Xixi Hu, name: Keyang Xu, name: Bo Liu, name: Qiang Liu, name: Hongliang Fei
Published: 2024-11-28 23:45:45

Achieving precise alignment between textual instructions and generated images
in text-to-image generation is a significant challenge, particularly in
rendering written text within images. Sate-of-the-art models like Stable
Diffusion 3 (SD3), Flux, and AuraFlow still struggle with accurate text
depiction, resulting in misspelled or inconsistent text. We introduce a
training-free method with minimal computational overhead that significantly
enhances text rendering quality. Specifically, we introduce an overshooting
sampler for pretrained rectified flow (RF) models, by alternating between
over-simulating the learned ordinary differential equation (ODE) and
reintroducing noise. Compared to the Euler sampler, the overshooting sampler
effectively introduces an extra Langevin dynamics term that can help correct
the compounding error from successive Euler steps and therefore improve the
text rendering. However, when the overshooting strength is high, we observe
over-smoothing artifacts on the generated images. To address this issue, we
propose an Attention Modulated Overshooting sampler (AMO), which adaptively
controls the strength of overshooting for each image patch according to their
attention score with the text content. AMO demonstrates a 32.3% and 35.9%
improvement in text rendering accuracy on SD3 and Flux without compromising
overall image quality or increasing inference cost.
Improving speaker verification robustness with synthetic emotional
  utterances
Authors: name: Nikhil Kumar Koditala, name: Chelsea Jui-Ting Ju, name: Ruirui Li, name: Minho Jin, name: Aman Chadha, name: Andreas Stolcke
Published: 2024-11-30 02:18:26

A speaker verification (SV) system offers an authentication service designed
to confirm whether a given speech sample originates from a specific speaker.
This technology has paved the way for various personalized applications that
cater to individual preferences. A noteworthy challenge faced by SV systems is
their ability to perform consistently across a range of emotional spectra. Most
existing models exhibit high error rates when dealing with emotional utterances
compared to neutral ones. Consequently, this phenomenon often leads to missing
out on speech of interest. This issue primarily stems from the limited
availability of labeled emotional speech data, impeding the development of
robust speaker representations that encompass diverse emotional states.
  To address this concern, we propose a novel approach employing the CycleGAN
framework to serve as a data augmentation method. This technique synthesizes
emotional speech segments for each specific speaker while preserving the unique
vocal identity. Our experimental findings underscore the effectiveness of
incorporating synthetic emotional data into the training process. The models
trained using this augmented dataset consistently outperform the baseline
models on the task of verifying speakers in emotional speech scenarios,
reducing equal error rate by as much as 3.64% relative.
Exploring the Abilities of Large Language Models to Solve Proportional
  Analogies via Knowledge-Enhanced Prompting
Authors: name: Thilini Wijesiriwardene, name: Ruwan Wickramarachchi, name: Sreeram Vennam, name: Vinija Jain, name: Aman Chadha, name: Amitava Das, name: Ponnurangam Kumaraguru, name: Amit Sheth
Published: 2024-12-01 16:15:14

Making analogies is fundamental to cognition. Proportional analogies, which
consist of four terms, are often used to assess linguistic and cognitive
abilities. For instance, completing analogies like "Oxygen is to Gas as <blank>
is to <blank>" requires identifying the semantic relationship (e.g., "type of")
between the first pair of terms ("Oxygen" and "Gas") and finding a second pair
that shares the same relationship (e.g., "Aluminum" and "Metal"). In this work,
we introduce a 15K Multiple-Choice Question Answering (MCQA) dataset for
proportional analogy completion and evaluate the performance of contemporary
Large Language Models (LLMs) in various knowledge-enhanced prompt settings.
Specifically, we augment prompts with three types of knowledge: exemplar,
structured, and targeted. Our results show that despite extensive training
data, solving proportional analogies remains challenging for current LLMs, with
the best model achieving an accuracy of 55%. Notably, we find that providing
targeted knowledge can better assist models in completing proportional
analogies compared to providing exemplars or collections of structured
knowledge.
The Well: a Large-Scale Collection of Diverse Physics Simulations for
  Machine Learning
Authors: name: Ruben Ohana, name: Michael McCabe, name: Lucas Meyer, name: Rudy Morel, name: Fruzsina J. Agocs, name: Miguel Beneitez, name: Marsha Berger, name: Blakesley Burkhart, name: Stuart B. Dalziel, name: Drummond B. Fielding, name: Daniel Fortunato, name: Jared A. Goldberg, name: Keiya Hirashima, name: Yan-Fei Jiang, name: Rich R. Kerswell, name: Suryanarayana Maddu, name: Jonah Miller, name: Payel Mukhopadhyay, name: Stefan S. Nixon, name: Jeff Shen, name: Romain Watteaux, name: Bruno Régaldo-Saint Blancard, name: François Rozet, name: Liam H. Parker, name: Miles Cranmer, name: Shirley Ho
Published: 2024-11-30 19:42:14

Machine learning based surrogate models offer researchers powerful tools for
accelerating simulation-based workflows. However, as standard datasets in this
space often cover small classes of physical behavior, it can be difficult to
evaluate the efficacy of new approaches. To address this gap, we introduce the
Well: a large-scale collection of datasets containing numerical simulations of
a wide variety of spatiotemporal physical systems. The Well draws from domain
experts and numerical software developers to provide 15TB of data across 16
datasets covering diverse domains such as biological systems, fluid dynamics,
acoustic scattering, as well as magneto-hydrodynamic simulations of
extra-galactic fluids or supernova explosions. These datasets can be used
individually or as part of a broader benchmark suite. To facilitate usage of
the Well, we provide a unified PyTorch interface for training and evaluating
models. We demonstrate the function of this library by introducing example
baselines that highlight the new challenges posed by the complex dynamics of
the Well. The code and data is available at
https://github.com/PolymathicAI/the_well.
World-consistent Video Diffusion with Explicit 3D Modeling
Authors: name: Qihang Zhang, name: Shuangfei Zhai, name: Miguel Angel Bautista, name: Kevin Miao, name: Alexander Toshev, name: Joshua Susskind, name: Jiatao Gu
Published: 2024-12-02 18:58:23

Recent advancements in diffusion models have set new benchmarks in image and
video generation, enabling realistic visual synthesis across single- and
multi-frame contexts. However, these models still struggle with efficiently and
explicitly generating 3D-consistent content. To address this, we propose
World-consistent Video Diffusion (WVD), a novel framework that incorporates
explicit 3D supervision using XYZ images, which encode global 3D coordinates
for each image pixel. More specifically, we train a diffusion transformer to
learn the joint distribution of RGB and XYZ frames. This approach supports
multi-task adaptability via a flexible inpainting strategy. For example, WVD
can estimate XYZ frames from ground-truth RGB or generate novel RGB frames
using XYZ projections along a specified camera trajectory. In doing so, WVD
unifies tasks like single-image-to-3D generation, multi-view stereo, and
camera-controlled video generation. Our approach demonstrates competitive
performance across multiple benchmarks, providing a scalable solution for
3D-consistent video and image generation with a single pretrained model.
Art-Free Generative Models: Art Creation Without Graphic Art Knowledge
Authors: name: Hui Ren, name: Joanna Materzynska, name: Rohit Gandikota, name: David Bau, name: Antonio Torralba
Published: 2024-11-29 18:59:01

We explore the question: "How much prior art knowledge is needed to create
art?" To investigate this, we propose a text-to-image generation model trained
without access to art-related content. We then introduce a simple yet effective
method to learn an art adapter using only a few examples of selected artistic
styles. Our experiments show that art generated using our method is perceived
by users as comparable to art produced by models trained on large, art-rich
datasets. Finally, through data attribution techniques, we illustrate how
examples from both artistic and non-artistic datasets contributed to the
creation of new artistic styles.
Towards Cross-Lingual Audio Abuse Detection in Low-Resource Settings
  with Few-Shot Learning
Authors: name: Aditya Narayan Sankaran, name: Reza Farahbaksh, name: Noel Crespi
Published: 2024-12-02 11:51:19

Online abusive content detection, particularly in low-resource settings and
within the audio modality, remains underexplored. We investigate the potential
of pre-trained audio representations for detecting abusive language in
low-resource languages, in this case, in Indian languages using Few Shot
Learning (FSL). Leveraging powerful representations from models such as Wav2Vec
and Whisper, we explore cross-lingual abuse detection using the ADIMA dataset
with FSL. Our approach integrates these representations within the
Model-Agnostic Meta-Learning (MAML) framework to classify abusive language in
10 languages. We experiment with various shot sizes (50-200) evaluating the
impact of limited data on performance. Additionally, a feature visualization
study was conducted to better understand model behaviour. This study highlights
the generalization ability of pre-trained models in low-resource scenarios and
offers valuable insights into detecting abusive language in multilingual
contexts.
PhysGame: Uncovering Physical Commonsense Violations in Gameplay Videos
Authors: name: Meng Cao, name: Haoran Tang, name: Haoze Zhao, name: Hangyu Guo, name: Jiaheng Liu, name: Ge Zhang, name: Ruyang Liu, name: Qiang Sun, name: Ian Reid, name: Xiaodan Liang
Published: 2024-12-02 18:47:25

Recent advancements in video-based large language models (Video LLMs) have
witnessed the emergence of diverse capabilities to reason and interpret dynamic
visual content. Among them, gameplay videos stand out as a distinctive data
source, often containing glitches that defy physics commonsense. This
characteristic renders them an effective benchmark for assessing the
under-explored capability of physical commonsense understanding in video LLMs.
In this paper, we propose PhysGame as a pioneering benchmark to evaluate
physical commonsense violations in gameplay videos. PhysGame comprises 880
videos associated with glitches spanning four fundamental domains (i.e.,
mechanics, kinematics, optics, and material properties) and across 12 distinct
physical commonsense. Through extensively evaluating various state-ofthe-art
video LLMs, our findings reveal that the performance of current open-source
video LLMs significantly lags behind that of proprietary counterparts. To
bridge this gap, we curate an instruction tuning dataset PhysInstruct with
140,057 question-answering pairs to facilitate physical commonsense learning.
In addition, we also propose a preference optimization dataset PhysDPO with
34,358 training pairs, where the dis-preferred responses are generated
conditioned on misleading titles (i.e., meta information hacking), fewer frames
(i.e., temporal hacking) and lower spatial resolutions (i.e., spatial hacking).
Based on the suite of datasets, we propose PhysVLM as a physical
knowledge-enhanced video LLM. Extensive experiments on both physical-oriented
benchmark PhysGame and general video understanding benchmarks demonstrate the
state-ofthe-art performance of PhysVLM.