Orionnnn/hub-stats huggingface dataset:
Guiding Vision-Language Model Selection for Visual Question-Answering
  Across Tasks, Domains, and Knowledge Types
Authors: name: Neelabh Sinha, name: Vinija Jain, name: Aman Chadha
Published: 2024-09-14 02:29:36

Visual Question-Answering (VQA) has become a key use-case in several
applications to aid user experience, particularly after Vision-Language Models
(VLMs) achieving good results in zero-shot inference. But evaluating different
VLMs for an application requirement using a standardized framework in practical
settings is still challenging. This paper introduces a comprehensive framework
for evaluating VLMs tailored to VQA tasks in practical settings. We present a
novel dataset derived from established VQA benchmarks, annotated with task
types, application domains, and knowledge types, three key practical aspects on
which tasks can vary. We also introduce GoEval, a multimodal evaluation metric
developed using GPT-4o, achieving a correlation factor of 56.71% with human
judgments. Our experiments with ten state-of-the-art VLMs reveals that no
single model excelling universally, making appropriate selection a key design
decision. Proprietary models such as Gemini-1.5-Pro and GPT-4o-mini generally
outperform others, though open-source models like InternVL-2-8B and
CogVLM-2-Llama-3-19B demonstrate competitive strengths in specific contexts,
while providing additional advantages. This study guides the selection of VLMs
based on specific task requirements and resource constraints, and can also be
extended to other vision-language tasks.
ReCLAP: Improving Zero Shot Audio Classification by Describing Sounds
Authors: name: Sreyan Ghosh, name: Sonal Kumar, name: Chandra Kiran Reddy Evuru, name: Oriol Nieto, name: Ramani Duraiswami, name: Dinesh Manocha
Published: 2024-09-13 21:58:20

Open-vocabulary audio-language models, like CLAP, offer a promising approach
for zero-shot audio classification (ZSAC) by enabling classification with any
arbitrary set of categories specified with natural language prompts. In this
paper, we propose a simple but effective method to improve ZSAC with CLAP.
Specifically, we shift from the conventional method of using prompts with
abstract category labels (e.g., Sound of an organ) to prompts that describe
sounds using their inherent descriptive features in a diverse context (e.g.,The
organ's deep and resonant tones filled the cathedral.). To achieve this, we
first propose ReCLAP, a CLAP model trained with rewritten audio captions for
improved understanding of sounds in the wild. These rewritten captions describe
each sound event in the original caption using their unique discriminative
characteristics. ReCLAP outperforms all baselines on both multi-modal
audio-text retrieval and ZSAC. Next, to improve zero-shot audio classification
with ReCLAP, we propose prompt augmentation. In contrast to the traditional
method of employing hand-written template prompts, we generate custom prompts
for each unique label in the dataset. These custom prompts first describe the
sound event in the label and then employ them in diverse scenes. Our proposed
method improves ReCLAP's performance on ZSAC by 1%-18% and outperforms all
baselines by 1% - 55%.
RetrievalAttention: Accelerating Long-Context LLM Inference via Vector
  Retrieval
Authors: name: Di Liu, name: Meng Chen, name: Baotong Lu, name: Huiqiang Jiang, name: Zhenhua Han, name: Qianxi Zhang, name: Qi Chen, name: Chengruidong Zhang, name: Bailu Ding, name: Kai Zhang, name: Chen Chen, name: Fan Yang, name: Yuqing Yang, name: Lili Qiu
Published: 2024-09-16 17:59:52

Transformer-based large Language Models (LLMs) become increasingly important
in various domains. However, the quadratic time complexity of attention
operation poses a significant challenge for scaling to longer contexts due to
the extremely high inference latency and GPU memory consumption for caching
key-value (KV) vectors. This paper proposes RetrievalAttention, a training-free
approach to accelerate attention computation. To leverage the dynamic sparse
property of attention, RetrievalAttention builds approximate nearest neighbor
search (ANNS) indexes upon KV vectors in CPU memory and retrieves the most
relevant ones via vector search during generation. Due to the
out-of-distribution (OOD) between query vectors and key vectors, off-the-shelf
ANNS indexes still need to scan O(N) (usually 30% of all keys) data for
accurate retrieval, which fails to exploit the high sparsity.
RetrievalAttention first identifies the OOD challenge of ANNS-based attention,
and addresses it via an attention-aware vector search algorithm that can adapt
to queries and only access 1--3% of data, thus achieving a sub-linear time
complexity. RetrievalAttention greatly reduces the inference cost of
long-context LLM with much lower GPU memory requirements while maintaining the
model accuracy. Especially, RetrievalAttention only needs 16GB GPU memory for
serving 128K tokens in LLMs with 8B parameters, which is capable of generating
one token in 0.188 seconds on a single NVIDIA RTX4090 (24GB).
Seed-Music: A Unified Framework for High Quality and Controlled Music
  Generation
Authors: name: Ye Bai, name: Haonan Chen, name: Jitong Chen, name: Zhuo Chen, name: Yi Deng, name: Xiaohong Dong, name: Lamtharn Hantrakul, name: Weituo Hao, name: Qingqing Huang, name: Zhongyi Huang, name: Dongya Jia, name: Feihu La, name: Duc Le, name: Bochen Li, name: Chumin Li, name: Hui Li, name: Xingxing Li, name: Shouda Liu, name: Wei-Tsung Lu, name: Yiqing Lu, name: Andrew Shaw, name: Janne Spijkervet, name: Yakun Sun, name: Bo Wang, name: Ju-Chiang Wang, name: Yuping Wang, name: Yuxuan Wang, name: Ling Xu, name: Yifeng Yang, name: Chao Yao, name: Shuo Zhang, name: Yang Zhang, name: Yilin Zhang, name: Hang Zhao, name: Ziyi Zhao, name: Dejian Zhong, name: Shicen Zhou, name: Pei Zou
Published: 2024-09-13 22:06:18

We introduce Seed-Music, a suite of music generation systems capable of
producing high-quality music with fine-grained style control. Our unified
framework leverages both auto-regressive language modeling and diffusion
approaches to support two key music creation workflows: controlled
music generation and post-production editing. For controlled music
generation, our system enables vocal music generation with performance controls
from multi-modal inputs, including style descriptions, audio references,
musical scores, and voice prompts. For post-production editing, it offers
interactive tools for editing lyrics and vocal melodies directly in the
generated audio.
  We encourage readers to listen to demo audio examples at
https://team.doubao.com/seed-music .
A Diffusion Approach to Radiance Field Relighting using
  Multi-Illumination Synthesis
Authors: name: Yohan Poirier-Ginter, name: Alban Gauthier, name: Julien Phillip, name: Jean-Francois Lalonde, name: George Drettakis
Published: 2024-09-13 16:07:25

Relighting radiance fields is severely underconstrained for multi-view data,
which is most often captured under a single illumination condition; It is
especially hard for full scenes containing multiple objects. We introduce a
method to create relightable radiance fields using such single-illumination
data by exploiting priors extracted from 2D image diffusion models. We first
fine-tune a 2D diffusion model on a multi-illumination dataset conditioned by
light direction, allowing us to augment a single-illumination capture into a
realistic -- but possibly inconsistent -- multi-illumination dataset from
directly defined light directions. We use this augmented data to create a
relightable radiance field represented by 3D Gaussian splats. To allow direct
control of light direction for low-frequency lighting, we represent appearance
with a multi-layer perceptron parameterized on light direction. To enforce
multi-view consistency and overcome inaccuracies we optimize a per-image
auxiliary feature vector. We show results on synthetic and real multi-view data
under single illumination, demonstrating that our method successfully exploits
2D diffusion model priors to allow realistic 3D relighting for complete scenes.
Project site
https://repo-sam.inria.fr/fungraph/generative-radiance-field-relighting/
InstantDrag: Improving Interactivity in Drag-based Image Editing
Authors: name: Joonghyuk Shin, name: Daehyeon Choi, name: Jaesik Park
Published: 2024-09-13 14:19:27

Drag-based image editing has recently gained popularity for its interactivity
and precision. However, despite the ability of text-to-image models to generate
samples within a second, drag editing still lags behind due to the challenge of
accurately reflecting user interaction while maintaining image content. Some
existing approaches rely on computationally intensive per-image optimization or
intricate guidance-based methods, requiring additional inputs such as masks for
movable regions and text prompts, thereby compromising the interactivity of the
editing process. We introduce InstantDrag, an optimization-free pipeline that
enhances interactivity and speed, requiring only an image and a drag
instruction as input. InstantDrag consists of two carefully designed networks:
a drag-conditioned optical flow generator (FlowGen) and an optical
flow-conditioned diffusion model (FlowDiffusion). InstantDrag learns motion
dynamics for drag-based image editing in real-world video datasets by
decomposing the task into motion generation and motion-conditioned image
generation. We demonstrate InstantDrag's capability to perform fast,
photo-realistic edits without masks or text prompts through experiments on
facial video datasets and general scenes. These results highlight the
efficiency of our approach in handling drag-based image editing, making it a
promising solution for interactive, real-time applications.
Click2Mask: Local Editing with Dynamic Mask Generation
Authors: name: Omer Regev, name: Omri Avrahami, name: Dani Lischinski
Published: 2024-09-12 17:59:04

Recent advancements in generative models have revolutionized image generation
and editing, making these tasks accessible to non-experts. This paper focuses
on local image editing, particularly the task of adding new content to a
loosely specified area. Existing methods often require a precise mask or a
detailed description of the location, which can be cumbersome and prone to
errors. We propose Click2Mask, a novel approach that simplifies the local
editing process by requiring only a single point of reference (in addition to
the content description). A mask is dynamically grown around this point during
a Blended Latent Diffusion (BLD) process, guided by a masked CLIP-based
semantic loss. Click2Mask surpasses the limitations of segmentation-based and
fine-tuning dependent methods, offering a more user-friendly and contextually
accurate solution. Our experiments demonstrate that Click2Mask not only
minimizes user effort but also delivers competitive or superior local image
manipulation results compared to SoTA methods, according to both human
judgement and automatic metrics. Key contributions include the simplification
of user input, the ability to freely add objects unconstrained by existing
segments, and the integration potential of our dynamic mask approach within
other editing methods.
DrawingSpinUp: 3D Animation from Single Character Drawings
Authors: name: Jie Zhou, name: Chufeng Xiao, name: Miu-Ling Lam, name: Hongbo Fu
Published: 2024-09-13 08:05:20

Animating various character drawings is an engaging visual content creation
task. Given a single character drawing, existing animation methods are limited
to flat 2D motions and thus lack 3D effects. An alternative solution is to
reconstruct a 3D model from a character drawing as a proxy and then retarget 3D
motion data onto it. However, the existing image-to-3D methods could not work
well for amateur character drawings in terms of appearance and geometry. We
observe the contour lines, commonly existing in character drawings, would
introduce significant ambiguity in texture synthesis due to their
view-dependence. Additionally, thin regions represented by single-line contours
are difficult to reconstruct (e.g., slim limbs of a stick figure) due to their
delicate structures. To address these issues, we propose a novel system,
DrawingSpinUp, to produce plausible 3D animations and breathe life into
character drawings, allowing them to freely spin up, leap, and even perform a
hip-hop dance. For appearance improvement, we adopt a removal-then-restoration
strategy to first remove the view-dependent contour lines and then render them
back after retargeting the reconstructed character. For geometry refinement, we
develop a skeleton-based thinning deformation algorithm to refine the slim
structures represented by the single-line contours. The experimental
evaluations and a perceptual user study show that our proposed method
outperforms the existing 2D and 3D animation methods and generates high-quality
3D animations from a single character drawing. Please refer to our project page
(https://lordliang.github.io/DrawingSpinUp) for the code and generated
animations.
Apollo: Band-sequence Modeling for High-Quality Audio Restoration
Authors: name: Kai Li, name: Yi Luo
Published: 2024-09-13 03:25:34

Audio restoration has become increasingly significant in modern society, not
only due to the demand for high-quality auditory experiences enabled by
advanced playback devices, but also because the growing capabilities of
generative audio models necessitate high-fidelity audio. Typically, audio
restoration is defined as a task of predicting undistorted audio from damaged
input, often trained using a GAN framework to balance perception and
distortion. Since audio degradation is primarily concentrated in mid- and
high-frequency ranges, especially due to codecs, a key challenge lies in
designing a generator capable of preserving low-frequency information while
accurately reconstructing high-quality mid- and high-frequency content.
Inspired by recent advancements in high-sample-rate music separation, speech
enhancement, and audio codec models, we propose Apollo, a generative model
designed for high-sample-rate audio restoration. Apollo employs an explicit
frequency band split module to model the relationships between different
frequency bands, allowing for more coherent and higher-quality restored audio.
Evaluated on the MUSDB18-HQ and MoisesDB datasets, Apollo consistently
outperforms existing SR-GAN models across various bit rates and music genres,
particularly excelling in complex scenarios involving mixtures of multiple
instruments and vocals. Apollo significantly improves music restoration quality
while maintaining computational efficiency. The source code for Apollo is
publicly available at https://github.com/JusperLee/Apollo.
Mamba-YOLO-World: Marrying YOLO-World with Mamba for Open-Vocabulary
  Detection
Authors: name: Haoxuan Wang, name: Qingdong He, name: Jinlong Peng, name: Hao Yang, name: Mingmin Chi, name: Yabiao Wang
Published: 2024-09-13 03:23:52

Open-vocabulary detection (OVD) aims to detect objects beyond a predefined
set of categories. As a pioneering model incorporating the YOLO series into
OVD, YOLO-World is well-suited for scenarios prioritizing speed and
efficiency.However, its performance is hindered by its neck feature fusion
mechanism, which causes the quadratic complexity and the limited guided
receptive fields.To address these limitations, we present Mamba-YOLO-World, a
novel YOLO-based OVD model employing the proposed MambaFusion Path Aggregation
Network (MambaFusion-PAN) as its neck architecture. Specifically, we introduce
an innovative State Space Model-based feature fusion mechanism consisting of a
Parallel-Guided Selective Scan algorithm and a Serial-Guided Selective Scan
algorithm with linear complexity and globally guided receptive fields. It
leverages multi-modal input sequences and mamba hidden states to guide the
selective scanning process.Experiments demonstrate that our model outperforms
the original YOLO-World on the COCO and LVIS benchmarks in both zero-shot and
fine-tuning settings while maintaining comparable parameters and FLOPs.
Additionally, it surpasses existing state-of-the-art OVD methods with fewer
parameters and FLOPs.
Robust Dual Gaussian Splatting for Immersive Human-centric Volumetric
  Videos
Authors: name: Yuheng Jiang, name: Zhehao Shen, name: Yu Hong, name: Chengcheng Guo, name: Yize Wu, name: Yingliang Zhang, name: Jingyi Yu, name: Lan Xu
Published: 2024-09-12 18:33:13

Volumetric video represents a transformative advancement in visual media,
enabling users to freely navigate immersive virtual experiences and narrowing
the gap between digital and real worlds. However, the need for extensive manual
intervention to stabilize mesh sequences and the generation of excessively
large assets in existing workflows impedes broader adoption. In this paper, we
present a novel Gaussian-based approach, dubbed DualGS, for real-time
and high-fidelity playback of complex human performance with excellent
compression ratios. Our key idea in DualGS is to separately represent motion
and appearance using the corresponding skin and joint Gaussians. Such an
explicit disentanglement can significantly reduce motion redundancy and enhance
temporal coherence. We begin by initializing the DualGS and anchoring skin
Gaussians to joint Gaussians at the first frame. Subsequently, we employ a
coarse-to-fine training strategy for frame-by-frame human performance modeling.
It includes a coarse alignment phase for overall motion prediction as well as a
fine-grained optimization for robust tracking and high-fidelity rendering. To
integrate volumetric video seamlessly into VR environments, we efficiently
compress motion using entropy encoding and appearance using codec compression
coupled with a persistent codebook. Our approach achieves a compression ratio
of up to 120 times, only requiring approximately 350KB of storage per frame. We
demonstrate the efficacy of our representation through photo-realistic,
free-view experiences on VR headsets, enabling users to immersively watch
musicians in performance and feel the rhythm of the notes at the performers'
fingertips.
DSBench: How Far Are Data Science Agents to Becoming Data Science
  Experts?
Authors: name: Liqiang Jing, name: Zhehui Huang, name: Xiaoyang Wang, name: Wenlin Yao, name: Wenhao Yu, name: Kaixin Ma, name: Hongming Zhang, name: Xinya Du, name: Dong Yu
Published: 2024-09-12 02:08:00

Large Language Models (LLMs) and Large Vision-Language Models (LVLMs) have
demonstrated impressive language/vision reasoning abilities, igniting the
recent trend of building agents for targeted applications such as shopping
assistants or AI software engineers. Recently, many data science benchmarks
have been proposed to investigate their performance in the data science domain.
However, existing data science benchmarks still fall short when compared to
real-world data science applications due to their simplified settings. To
bridge this gap, we introduce DSBench, a comprehensive benchmark designed to
evaluate data science agents with realistic tasks. This benchmark includes 466
data analysis tasks and 74 data modeling tasks, sourced from Eloquence and
Kaggle competitions. DSBench offers a realistic setting by encompassing long
contexts, multimodal task backgrounds, reasoning with large data files and
multi-table structures, and performing end-to-end data modeling tasks. Our
evaluation of state-of-the-art LLMs, LVLMs, and agents shows that they struggle
with most tasks, with the best agent solving only 34.12% of data analysis tasks
and achieving a 34.74% Relative Performance Gap (RPG). These findings
underscore the need for further advancements in developing more practical,
intelligent, and autonomous data science agents.
Can OOD Object Detectors Learn from Foundation Models?
Authors: name: Jiahui Liu, name: Xin Wen, name: Shizhen Zhao, name: Yingxian Chen, name: Xiaojuan Qi
Published: 2024-09-08 17:28:22

Out-of-distribution (OOD) object detection is a challenging task due to the
absence of open-set OOD data. Inspired by recent advancements in text-to-image
generative models, such as Stable Diffusion, we study the potential of
generative models trained on large-scale open-set data to synthesize OOD
samples, thereby enhancing OOD object detection. We introduce SyncOOD, a simple
data curation method that capitalizes on the capabilities of large foundation
models to automatically extract meaningful OOD data from text-to-image
generative models. This offers the model access to open-world knowledge
encapsulated within off-the-shelf foundation models. The synthetic OOD samples
are then employed to augment the training of a lightweight, plug-and-play OOD
detector, thus effectively optimizing the in-distribution (ID)/OOD decision
boundaries. Extensive experiments across multiple benchmarks demonstrate that
SyncOOD significantly outperforms existing methods, establishing new
state-of-the-art performance with minimal synthetic data usage.
PiTe: Pixel-Temporal Alignment for Large Video-Language Model
Authors: name: Yang Liu, name: Pengxiang Ding, name: Siteng Huang, name: Min Zhang, name: Han Zhao, name: Donglin Wang
Published: 2024-09-11 12:53:07

Fueled by the Large Language Models (LLMs) wave, Large Visual-Language Models
(LVLMs) have emerged as a pivotal advancement, bridging the gap between image
and text. However, video making it challenging for LVLMs to perform adequately
due to the complexity of the relationship between language and spatial-temporal
data structure. Recent Large Video-Language Models (LVidLMs) align feature of
static visual data like image into latent space of language feature, by general
multi-modal tasks to leverage abilities of LLMs sufficiently. In this paper, we
explore fine-grained alignment approach via object trajectory for different
modalities across both spatial and temporal dimensions simultaneously. Thus, we
propose a novel LVidLM by trajectory-guided Pixel-Temporal Alignment, dubbed
PiTe, that exhibits promising applicable model property. To achieve
fine-grained video-language alignment, we curate a multi-modal pre-training
dataset PiTe-143k, the dataset provision of moving trajectories in pixel level
for all individual objects, that appear and mention in the video and caption
both, by our automatic annotation pipeline. Meanwhile, PiTe demonstrates
astounding capabilities on myriad video-related multi-modal tasks through beat
the state-of-the-art methods by a large margin.
IFAdapter: Instance Feature Control for Grounded Text-to-Image
  Generation
Authors: name: Yinwei Wu, name: Xianpan Zhou, name: Bing Ma, name: Xuefeng Su, name: Kai Ma, name: Xinchao Wang
Published: 2024-09-12 17:39:23

While Text-to-Image (T2I) diffusion models excel at generating visually
appealing images of individual instances, they struggle to accurately position
and control the features generation of multiple instances. The Layout-to-Image
(L2I) task was introduced to address the positioning challenges by
incorporating bounding boxes as spatial control signals, but it still falls
short in generating precise instance features. In response, we propose the
Instance Feature Generation (IFG) task, which aims to ensure both positional
accuracy and feature fidelity in generated instances. To address the IFG task,
we introduce the Instance Feature Adapter (IFAdapter). The IFAdapter enhances
feature depiction by incorporating additional appearance tokens and utilizing
an Instance Semantic Map to align instance-level features with spatial
locations. The IFAdapter guides the diffusion process as a plug-and-play
module, making it adaptable to various community models. For evaluation, we
contribute an IFG benchmark and develop a verification pipeline to objectively
compare models' abilities to generate instances with accurate positioning and
features. Experimental results demonstrate that IFAdapter outperforms other
models in both quantitative and qualitative evaluations.