Orionnnn/hub-stats huggingface dataset:
Surveying the Effects of Quality, Diversity, and Complexity in Synthetic
  Data From Large Language Models
Authors: name: Alex Havrilla, name: Andrew Dai, name: Laura O'Mahony, name: Koen Oostermeijer, name: Vera Zisler, name: Alon Albalak, name: Fabrizio Milo, name: Sharath Chandra Raparthy, name: Kanishk Gandhi, name: Baber Abbasi, name: Duy Phung, name: Maia Iyer, name: Dakota Mahan, name: Chase Blagden, name: Srishti Gureja, name: Mohammed Hamdy, name: Wen-Ding Li, name: Giovanni Paolini, name: Pawan Sasanka Ammanamanchi, name: Elliot Meyerson
Published: 2024-12-04 02:47:45

Synthetic data generation with Large Language Models is a promising paradigm
for augmenting natural data over a nearly infinite range of tasks. Given this
variety, direct comparisons among synthetic data generation algorithms are
scarce, making it difficult to understand where improvement comes from and what
bottlenecks exist. We propose to evaluate algorithms via the makeup of
synthetic data generated by each algorithm in terms of data quality, diversity,
and complexity. We choose these three characteristics for their significance in
open-ended processes and the impact each has on the capabilities of downstream
models. We find quality to be essential for in-distribution model
generalization, diversity to be essential for out-of-distribution
generalization, and complexity to be beneficial for both. Further, we emphasize
the existence of Quality-Diversity trade-offs in training data and the
downstream effects on model performance. We then examine the effect of various
components in the synthetic data pipeline on each data characteristic. This
examination allows us to taxonomize and compare synthetic data generation
algorithms through the components they utilize and the resulting effects on
data QDC composition. This analysis extends into a discussion on the importance
of balancing QDC in synthetic data for efficient reinforcement learning and
self-improvement algorithms. Analogous to the QD trade-offs in training data,
often there exist trade-offs between model output quality and output diversity
which impact the composition of synthetic data. We observe that many models are
currently evaluated and optimized only for output quality, thereby limiting
output diversity and the potential for self-improvement. We argue that
balancing these trade-offs is essential to the development of future
self-improvement algorithms and highlight a number of works making progress in
this direction.
U-MATH: A University-Level Benchmark for Evaluating Mathematical Skills
  in LLMs
Authors: name: Konstantin Chernyshev, name: Vitaliy Polshkov, name: Ekaterina Artemova, name: Alex Myasnikov, name: Vlad Stepanov, name: Alexei Miasnikov, name: Sergei Tilga
Published: 2024-12-04 10:44:50

The current evaluation of mathematical skills in LLMs is limited, as existing
benchmarks are either relatively small, primarily focus on elementary and
high-school problems, or lack diversity in topics. Additionally, the inclusion
of visual elements in tasks remains largely under-explored.
  To address these gaps, we introduce U-MATH, a novel benchmark of 1,100
unpublished open-ended university-level problems sourced from teaching
materials. It is balanced across six core subjects, with 20% of multimodal
problems. Given the open-ended nature of U-MATH problems, we employ an LLM to
judge the correctness of generated solutions. To this end, we release
mu-MATH, a dataset to evaluate the LLMs' capabilities in judging solutions.
  The evaluation of general domain, math-specific, and multimodal LLMs
highlights the challenges presented by U-MATH. Our findings reveal that LLMs
achieve a maximum accuracy of only 63% on text-based tasks, with even lower 45%
on visual problems. The solution assessment proves challenging for LLMs, with
the best LLM judge having an F1-score of 80% on mu-MATH.
Inst-IT: Boosting Multimodal Instance Understanding via Explicit Visual
  Prompt Instruction Tuning
Authors: name: Wujian Peng, name: Lingchen Meng, name: Yitong Chen, name: Yiweng Xie, name: Yang Liu, name: Tao Gui, name: Hang Xu, name: Xipeng Qiu, name: Zuxuan Wu, name: Yu-Gang Jiang
Published: 2024-12-04 18:58:10

Large Multimodal Models (LMMs) have made significant breakthroughs with the
advancement of instruction tuning. However, while existing models can
understand images and videos at a holistic level, they still struggle with
instance-level understanding that requires a more nuanced comprehension and
alignment. Instance-level understanding is crucial, as it focuses on the
specific elements that we are most interested in. Excitingly, existing works
find that the state-of-the-art LMMs exhibit strong instance understanding
capabilities when provided with explicit visual cues. Motivated by this, we
introduce an automated annotation pipeline assisted by GPT-4o to extract
instance-level information from images and videos through explicit visual
prompting for instance guidance. Building upon this pipeline, we proposed
Inst-IT, a solution to enhance LMMs in Instance understanding via explicit
visual prompt Instruction Tuning. Inst-IT consists of a benchmark to diagnose
multimodal instance-level understanding, a large-scale instruction-tuning
dataset, and a continuous instruction-tuning training paradigm to effectively
enhance spatial-temporal instance understanding capabilities of existing LMMs.
Experimental results show that, with the boost of Inst-IT, our models not only
achieve outstanding performance on Inst-IT Bench but also demonstrate
significant improvements across various generic image and video understanding
benchmarks. This highlights that our dataset not only boosts instance-level
understanding but also strengthens the overall capabilities of generic image
and video comprehension.
NitroFusion: High-Fidelity Single-Step Diffusion through Dynamic
  Adversarial Training
Authors: name: Dar-Yen Chen, name: Hmrishav Bandyopadhyay, name: Kai Zou, name: Yi-Zhe Song
Published: 2024-12-02 23:20:35

We introduce NitroFusion, a fundamentally different approach to single-step
diffusion that achieves high-quality generation through a dynamic adversarial
framework. While one-step methods offer dramatic speed advantages, they
typically suffer from quality degradation compared to their multi-step
counterparts. Just as a panel of art critics provides comprehensive feedback by
specializing in different aspects like composition, color, and technique, our
approach maintains a large pool of specialized discriminator heads that
collectively guide the generation process. Each discriminator group develops
expertise in specific quality aspects at different noise levels, providing
diverse feedback that enables high-fidelity one-step generation. Our framework
combines: (i) a dynamic discriminator pool with specialized discriminator
groups to improve generation quality, (ii) strategic refresh mechanisms to
prevent discriminator overfitting, and (iii) global-local discriminator heads
for multi-scale quality assessment, and unconditional/conditional training for
balanced generation. Additionally, our framework uniquely supports flexible
deployment through bottom-up refinement, allowing users to dynamically choose
between 1-4 denoising steps with the same model for direct quality-speed
trade-offs. Through comprehensive experiments, we demonstrate that NitroFusion
significantly outperforms existing single-step methods across multiple
evaluation metrics, particularly excelling in preserving fine details and
global consistency.
VARCO-VISION: Expanding Frontiers in Korean Vision-Language Models
Authors: name: Jeongho Ju, name: Daeyoung Kim, name: SunYoung Park, name: Youngjune Kim
Published: 2024-11-28 12:38:42

In this paper, we introduce an open-source Korean-English vision-language
model (VLM), VARCO-VISION. We incorporate a step-by-step training strategy that
allows a model learn both linguistic and visual information while preserving
the backbone model's knowledge. Our model demonstrates outstanding performance
in diverse settings requiring bilingual image-text understanding and generation
abilities compared to models of similar size. VARCO-VISION is also capable of
grounding, referring, and OCR, expanding its usage and potential applications
for real-world scenarios. In addition to the model, we release five Korean
evaluation datasets, including four closed-set and one openset benchmarks. We
anticipate that our milestone will broaden the opportunities for AI researchers
aiming to train VLMs. VARCO-VISION is available at
https://huggingface.co/NCSOFT/VARCO-VISION-14B.
PaliGemma 2: A Family of Versatile VLMs for Transfer
Authors: name: Andreas Steiner, name: André Susano Pinto, name: Michael Tschannen, name: Daniel Keysers, name: Xiao Wang, name: Yonatan Bitton, name: Alexey Gritsenko, name: Matthias Minderer, name: Anthony Sherbondy, name: Shangbang Long, name: Siyang Qin, name: Reeve Ingle, name: Emanuele Bugliarello, name: Sahar Kazemzadeh, name: Thomas Mesnard, name: Ibrahim Alabdulmohsin, name: Lucas Beyer, name: Xiaohua Zhai
Published: 2024-12-04 18:50:42

PaliGemma 2 is an upgrade of the PaliGemma open Vision-Language Model (VLM)
based on the Gemma 2 family of language models. We combine the SigLIP-So400m
vision encoder that was also used by PaliGemma with the whole range of Gemma 2
models, from the 2B one all the way up to the 27B model. We train these models
at three resolutions (224px, 448px, and 896px) in multiple stages to equip them
with broad knowledge for transfer via fine-tuning. The resulting family of base
models covering different model sizes and resolutions allows us to investigate
factors impacting transfer performance (such as learning rate) and to analyze
the interplay between the type of task, model size, and resolution. We further
increase the number and breadth of transfer tasks beyond the scope of PaliGemma
including different OCR-related tasks such as table structure recognition,
molecular structure recognition, music score recognition, as well as long
fine-grained captioning and radiography report generation, on which PaliGemma 2
obtains state-of-the-art results.
Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene
  Understanding
Authors: name: Duo Zheng, name: Shijia Huang, name: Liwei Wang
Published: 2024-11-30 14:28:53

The rapid advancement of Multimodal Large Language Models (MLLMs) has
significantly impacted various multimodal tasks. However, these models face
challenges in tasks that require spatial understanding within 3D environments.
Efforts to enhance MLLMs, such as incorporating point cloud features, have been
made, yet a considerable gap remains between the models' learned
representations and the inherent complexity of 3D scenes. This discrepancy
largely stems from the training of MLLMs on predominantly 2D data, which
restricts their effectiveness in comprehending 3D spaces. To address this
issue, in this paper, we propose a novel generalist model, i.e., Video-3D LLM,
for 3D scene understanding. By treating 3D scenes as dynamic videos and
incorporating 3D position encoding into these representations, our Video-3D LLM
aligns video representations with real-world spatial contexts more accurately.
Additionally, we have implemented a maximum coverage sampling technique to
optimize the balance between computational costs and performance efficiency.
Extensive experiments demonstrate that our model achieves state-of-the-art
performance on several 3D scene understanding benchmarks, including ScanRefer,
Multi3DRefer, Scan2Cap, ScanQA, and SQA3D.
CleanDIFT: Diffusion Features without Noise
Authors: name: Nick Stracke, name: Stefan Andreas Baumann, name: Kolja Bauer, name: Frank Fundel, name: Björn Ommer
Published: 2024-12-04 16:29:04

Internal features from large-scale pre-trained diffusion models have recently
been established as powerful semantic descriptors for a wide range of
downstream tasks. Works that use these features generally need to add noise to
images before passing them through the model to obtain the semantic features,
as the models do not offer the most useful features when given images with
little to no noise. We show that this noise has a critical impact on the
usefulness of these features that cannot be remedied by ensembling with
different random noises. We address this issue by introducing a lightweight,
unsupervised fine-tuning method that enables diffusion backbones to provide
high-quality, noise-free semantic features. We show that these features readily
outperform previous diffusion features by a wide margin in a wide variety of
extraction setups and downstream tasks, offering better performance than even
ensemble-based methods at a fraction of the cost.
SNOOPI: Supercharged One-step Diffusion Distillation with Proper
  Guidance
Authors: name: Viet Nguyen, name: Anh Aengus Nguyen, name: Trung Dao, name: Khoi Nguyen, name: Cuong Pham, name: Toan Tran, name: Anh Tran
Published: 2024-12-03 18:56:32

Recent approaches have yielded promising results in distilling multi-step
text-to-image diffusion models into one-step ones. The state-of-the-art
efficient distillation technique, i.e., SwiftBrushv2 (SBv2), even surpasses the
teacher model's performance with limited resources. However, our study reveals
its instability when handling different diffusion model backbones due to using
a fixed guidance scale within the Variational Score Distillation (VSD) loss.
Another weakness of the existing one-step diffusion models is the missing
support for negative prompt guidance, which is crucial in practical image
generation. This paper presents SNOOPI, a novel framework designed to address
these limitations by enhancing the guidance in one-step diffusion models during
both training and inference. First, we effectively enhance training stability
through Proper Guidance-SwiftBrush (PG-SB), which employs a random-scale
classifier-free guidance approach. By varying the guidance scale of both
teacher models, we broaden their output distributions, resulting in a more
robust VSD loss that enables SB to perform effectively across diverse backbones
while maintaining competitive performance. Second, we propose a training-free
method called Negative-Away Steer Attention (NASA), which integrates negative
prompts into one-step diffusion models via cross-attention to suppress
undesired elements in generated images. Our experimental results show that our
proposed methods significantly improve baseline models across various metrics.
Remarkably, we achieve an HPSv2 score of 31.08, setting a new state-of-the-art
benchmark for one-step diffusion models.
Weighted-Reward Preference Optimization for Implicit Model Fusion
Authors: name: Ziyi Yang, name: Fanqi Wan, name: Longguang Zhong, name: Tianyuan Shi, name: Xiaojun Quan
Published: 2024-12-04 10:15:12

While fusing heterogeneous open-source LLMs with varying architectures and
sizes can potentially integrate the strengths of different models, existing
fusion methods face significant challenges, such as vocabulary alignment and
merging distribution matrices. These procedures are not only complex but also
prone to introducing noise and errors. In this paper, we propose an implicit
fusion method, Weighted-Reward Preference Optimization (WRPO), which leverages
preference optimization between the source LLMs and the target LLM to transfer
their capabilities effectively. WRPO eliminates the need for vocabulary
alignment and matrix fusion and can be efficiently scaled to accommodate
various LLMs. To address distributional deviations between the source and
target LLMs, WRPO introduces a progressive adaptation strategy that gradually
shifts reliance on preferred examples from the target LLM to the source LLMs.
Extensive experiments on the MT-Bench, AlpacaEval-2, and Arena-Hard benchmarks
demonstrate that WRPO consistently outperforms existing knowledge fusion
methods and various fine-tuning baselines. When applied to LLaMA3-8B-Instruct
as the target model, WRPO achieves a length-controlled win rate of 55.9%
against GPT-4-Preview-1106 on AlpacaEval-2 and a win rate of 46.2% against
GPT-4-0314 on Arena-Hard. Our code is available at
https://github.com/SLIT-AI/WRPO.
NVComposer: Boosting Generative Novel View Synthesis with Multiple
  Sparse and Unposed Images
Authors: name: Lingen Li, name: Zhaoyang Zhang, name: Yaowei Li, name: Jiale Xu, name: Xiaoyu Li, name: Wenbo Hu, name: Weihao Cheng, name: Jinwei Gu, name: Tianfan Xue, name: Ying Shan
Published: 2024-12-04 17:58:03

Recent advancements in generative models have significantly improved novel
view synthesis (NVS) from multi-view data. However, existing methods depend on
external multi-view alignment processes, such as explicit pose estimation or
pre-reconstruction, which limits their flexibility and accessibility,
especially when alignment is unstable due to insufficient overlap or occlusions
between views. In this paper, we propose NVComposer, a novel approach that
eliminates the need for explicit external alignment. NVComposer enables the
generative model to implicitly infer spatial and geometric relationships
between multiple conditional views by introducing two key components: 1) an
image-pose dual-stream diffusion model that simultaneously generates target
novel views and condition camera poses, and 2) a geometry-aware feature
alignment module that distills geometric priors from dense stereo models during
training. Extensive experiments demonstrate that NVComposer achieves
state-of-the-art performance in generative multi-view NVS tasks, removing the
reliance on external alignment and thus improving model accessibility. Our
approach shows substantial improvements in synthesis quality as the number of
unposed input views increases, highlighting its potential for more flexible and
accessible generative NVS systems.
Imagine360: Immersive 360 Video Generation from Perspective Anchor
Authors: name: Jing Tan, name: Shuai Yang, name: Tong Wu, name: Jingwen He, name: Yuwei Guo, name: Ziwei Liu, name: Dahua Lin
Published: 2024-12-04 18:50:08

360^circ videos offer a hyper-immersive experience that allows the viewers
to explore a dynamic scene from full 360 degrees. To achieve more user-friendly
and personalized content creation in 360^circ video format, we seek to lift
standard perspective videos into 360^circ equirectangular videos. To this
end, we introduce Imagine360, the first perspective-to-360^circ video
generation framework that creates high-quality 360^circ videos with rich and
diverse motion patterns from video anchors. Imagine360 learns fine-grained
spherical visual and motion patterns from limited 360^circ video data with
several key designs. 1) Firstly we adopt the dual-branch design, including a
perspective and a panorama video denoising branch to provide local and global
constraints for 360^circ video generation, with motion module and spatial
LoRA layers fine-tuned on extended web 360^circ videos. 2) Additionally, an
antipodal mask is devised to capture long-range motion dependencies, enhancing
the reversed camera motion between antipodal pixels across hemispheres. 3) To
handle diverse perspective video inputs, we propose elevation-aware designs
that adapt to varying video masking due to changing elevations across frames.
Extensive experiments show Imagine360 achieves superior graphics quality and
motion coherence among state-of-the-art 360^circ video generation methods.
We believe Imagine360 holds promise for advancing personalized, immersive
360^circ video creation.
Mimir: Improving Video Diffusion Models for Precise Text Understanding
Authors: name: Shuai Tan, name: Biao Gong, name: Yutong Feng, name: Kecheng Zheng, name: Dandan Zheng, name: Shuwei Shi, name: Yujun Shen, name: Jingdong Chen, name: Ming Yang
Published: 2024-12-04 07:26:44

Text serves as the key control signal in video generation due to its
narrative nature. To render text descriptions into video clips, current video
diffusion models borrow features from text encoders yet struggle with limited
text comprehension. The recent success of large language models (LLMs)
showcases the power of decoder-only transformers, which offers three clear
benefits for text-to-video (T2V) generation, namely, precise text understanding
resulting from the superior scalability, imagination beyond the input text
enabled by next token prediction, and flexibility to prioritize user interests
through instruction tuning. Nevertheless, the feature distribution gap emerging
from the two different text modeling paradigms hinders the direct use of LLMs
in established T2V models. This work addresses this challenge with Mimir, an
end-to-end training framework featuring a carefully tailored token fuser to
harmonize the outputs from text encoders and LLMs. Such a design allows the T2V
model to fully leverage learned video priors while capitalizing on the
text-related capability of LLMs. Extensive quantitative and qualitative results
demonstrate the effectiveness of Mimir in generating high-quality videos with
excellent text comprehension, especially when processing short captions and
managing shifting motions. Project page:
https://lucaria-academy.github.io/Mimir/
MIDI: Multi-Instance Diffusion for Single Image to 3D Scene Generation
Authors: name: Zehuan Huang, name: Yuan-Chen Guo, name: Xingqiao An, name: Yunhan Yang, name: Yangguang Li, name: Zi-Xin Zou, name: Ding Liang, name: Xihui Liu, name: Yan-Pei Cao, name: Lu Sheng
Published: 2024-12-04 18:52:40

This paper introduces MIDI, a novel paradigm for compositional 3D scene
generation from a single image. Unlike existing methods that rely on
reconstruction or retrieval techniques or recent approaches that employ
multi-stage object-by-object generation, MIDI extends pre-trained image-to-3D
object generation models to multi-instance diffusion models, enabling the
simultaneous generation of multiple 3D instances with accurate spatial
relationships and high generalizability. At its core, MIDI incorporates a novel
multi-instance attention mechanism, that effectively captures inter-object
interactions and spatial coherence directly within the generation process,
without the need for complex multi-step processes. The method utilizes partial
object images and global scene context as inputs, directly modeling object
completion during 3D generation. During training, we effectively supervise the
interactions between 3D instances using a limited amount of scene-level data,
while incorporating single-object data for regularization, thereby maintaining
the pre-trained generalization ability. MIDI demonstrates state-of-the-art
performance in image-to-scene generation, validated through evaluations on
synthetic data, real-world scene data, and stylized scene images generated by
text-to-image diffusion models.