Orionnnn/hub-stats huggingface dataset:
AV-Link: Temporally-Aligned Diffusion Features for Cross-Modal
  Audio-Video Generation
Authors: name: Moayed Haji-Ali, name: Willi Menapace, name: Aliaksandr Siarohin, name: Ivan Skorokhodov, name: Alper Canberk, name: Kwot Sin Lee, name: Vicente Ordonez, name: Sergey Tulyakov
Published: 2024-12-19 18:57:21

We propose AV-Link, a unified framework for Video-to-Audio and Audio-to-Video
generation that leverages the activations of frozen video and audio diffusion
models for temporally-aligned cross-modal conditioning. The key to our
framework is a Fusion Block that enables bidirectional information exchange
between our backbone video and audio diffusion models through a
temporally-aligned self attention operation. Unlike prior work that uses
feature extractors pretrained for other tasks for the conditioning signal,
AV-Link can directly leverage features obtained by the complementary modality
in a single framework i.e. video features to generate audio, or audio features
to generate video. We extensively evaluate our design choices and demonstrate
the ability of our method to achieve synchronized and high-quality audiovisual
content, showcasing its potential for applications in immersive media
generation. Project Page: snap-research.github.io/AVLink/
PixelMan: Consistent Object Editing with Diffusion Models via Pixel
  Manipulation and Generation
Authors: name: Liyao Jiang, name: Negar Hassanpour, name: Mohammad Salameh, name: Mohammadreza Samadi, name: Jiao He, name: Fengyu Sun, name: Di Niu
Published: 2024-12-18 19:24:15

Recent research explores the potential of Diffusion Models (DMs) for
consistent object editing, which aims to modify object position, size, and
composition, etc., while preserving the consistency of objects and background
without changing their texture and attributes. Current inference-time methods
often rely on DDIM inversion, which inherently compromises efficiency and the
achievable consistency of edited images. Recent methods also utilize energy
guidance which iteratively updates the predicted noise and can drive the
latents away from the original image, resulting in distortions. In this paper,
we propose PixelMan, an inversion-free and training-free method for achieving
consistent object editing via Pixel Manipulation and generation, where we
directly create a duplicate copy of the source object at target location in the
pixel space, and introduce an efficient sampling approach to iteratively
harmonize the manipulated object into the target location and inpaint its
original location, while ensuring image consistency by anchoring the edited
image to be generated to the pixel-manipulated image as well as by introducing
various consistency-preserving optimization techniques during inference.
Experimental evaluations based on benchmark datasets as well as extensive
visual comparisons show that in as few as 16 inference steps, PixelMan
outperforms a range of state-of-the-art training-based and training-free
methods (usually requiring 50 steps) on multiple consistent object editing
tasks.
DateLogicQA: Benchmarking Temporal Biases in Large Language Models
Authors: name: Gagan Bhatia, name: MingZe Tang, name: Cristina Mahanta, name: Madiha Kazi
Published: 2024-12-17 23:25:47

This paper introduces DateLogicQA, a benchmark with 190 questions covering
diverse date formats, temporal contexts, and reasoning types. We propose the
Semantic Integrity Metric to assess tokenization quality and analyse two
biases: Representation-Level Bias, affecting embeddings, and Logical-Level
Bias, influencing reasoning outputs. Our findings provide a comprehensive
evaluation of LLMs' capabilities and limitations in temporal reasoning,
highlighting key challenges in handling temporal data accurately. The GitHub
repository for our work is available at
https://github.com/gagan3012/EAIS-Temporal-Bias
Move-in-2D: 2D-Conditioned Human Motion Generation
Authors: name: Hsin-Ping Huang, name: Yang Zhou, name: Jui-Hsien Wang, name: Difan Liu, name: Feng Liu, name: Ming-Hsuan Yang, name: Zhan Xu
Published: 2024-12-17 18:58:07

Generating realistic human videos remains a challenging task, with the most
effective methods currently relying on a human motion sequence as a control
signal. Existing approaches often use existing motion extracted from other
videos, which restricts applications to specific motion types and global scene
matching. We propose Move-in-2D, a novel approach to generate human motion
sequences conditioned on a scene image, allowing for diverse motion that adapts
to different scenes. Our approach utilizes a diffusion model that accepts both
a scene image and text prompt as inputs, producing a motion sequence tailored
to the scene. To train this model, we collect a large-scale video dataset
featuring single-human activities, annotating each video with the corresponding
human motion as the target output. Experiments demonstrate that our method
effectively predicts human motion that aligns with the scene image after
projection. Furthermore, we show that the generated motion sequence improves
human motion quality in video synthesis tasks.
LeviTor: 3D Trajectory Oriented Image-to-Video Synthesis
Authors: name: Hanlin Wang, name: Hao Ouyang, name: Qiuyu Wang, name: Wen Wang, name: Ka Leong Cheng, name: Qifeng Chen, name: Yujun Shen, name: Limin Wang
Published: 2024-12-19 18:59:56

The intuitive nature of drag-based interaction has led to its growing
adoption for controlling object trajectories in image-to-video synthesis.
Still, existing methods that perform dragging in the 2D space usually face
ambiguity when handling out-of-plane movements. In this work, we augment the
interaction with a new dimension, i.e., the depth dimension, such that users
are allowed to assign a relative depth for each point on the trajectory. That
way, our new interaction paradigm not only inherits the convenience from 2D
dragging, but facilitates trajectory control in the 3D space, broadening the
scope of creativity. We propose a pioneering method for 3D trajectory control
in image-to-video synthesis by abstracting object masks into a few cluster
points. These points, accompanied by the depth information and the instance
information, are finally fed into a video diffusion model as the control
signal. Extensive experiments validate the effectiveness of our approach,
dubbed LeviTor, in precisely manipulating the object movements when producing
photo-realistic videos from static images. Project page:
https://ppetrichor.github.io/levitor.github.io/
AceMath: Advancing Frontier Math Reasoning with Post-Training and Reward
  Modeling
Authors: name: Zihan Liu, name: Yang Chen, name: Mohammad Shoeybi, name: Bryan Catanzaro, name: Wei Ping
Published: 2024-12-19 17:29:44

In this paper, we introduce AceMath, a suite of frontier math models that
excel in solving complex math problems, along with highly effective reward
models capable of evaluating generated solutions and reliably identifying the
correct ones. To develop the instruction-tuned math models, we propose a
supervised fine-tuning (SFT) process that first achieves competitive
performance across general domains, followed by targeted fine-tuning for the
math domain using a carefully curated set of prompts and synthetically
generated responses. The resulting model, AceMath-72B-Instruct greatly
outperforms Qwen2.5-Math-72B-Instruct, GPT-4o and Claude-3.5 Sonnet. To develop
math-specialized reward model, we first construct AceMath-RewardBench, a
comprehensive and robust benchmark for evaluating math reward models across
diverse problems and difficulty levels. After that, we present a systematic
approach to build our math reward models. The resulting model, AceMath-72B-RM,
consistently outperforms state-of-the-art reward models. Furthermore, when
combining AceMath-72B-Instruct with AceMath-72B-RM, we achieve the highest
average rm@8 score across the math reasoning benchmarks. We will release model
weights, training data, and evaluation benchmarks at:
https://research.nvidia.com/labs/adlr/acemath
Descriptive Caption Enhancement with Visual Specialists for Multimodal
  Perception
Authors: name: Yanpeng Sun, name: Jing Hao, name: Ke Zhu, name: Jiang-Jiang Liu, name: Yuxiang Zhao, name: Xiaofan Li, name: Gang Zhang, name: Zechao Li, name: Jingdong Wang
Published: 2024-12-18 18:45:43

Training Large Multimodality Models (LMMs) relies on descriptive image
caption that connects image and language. Existing methods either distill the
caption from the LMM models or construct the captions from the internet images
or by human. We propose to leverage off-the-shelf visual specialists, which
were trained from annotated images initially not for image captioning, for
enhancing the image caption.
  Our approach, named DCE, explores object low-level and fine-grained
attributes (e.g., depth, emotion and fine-grained categories) and object
relations (e.g., relative location and human-object-interaction (HOI)), and
combine the attributes into the descriptive caption. Experiments demonstrate
that such visual specialists are able to improve the performance for visual
understanding tasks as well as reasoning that benefits from more accurate
visual understanding. We will release the source code and the pipeline so that
other visual specialists are easily combined into the pipeline. The complete
source code of DCE pipeline and datasets will be available at
https://github.com/syp2ysy/DCE.
DI-PCG: Diffusion-based Efficient Inverse Procedural Content Generation
  for High-quality 3D Asset Creation
Authors: name: Wang Zhao, name: Yan-Pei Cao, name: Jiale Xu, name: Yuejiang Dong, name: Ying Shan
Published: 2024-12-19 18:58:46

Procedural Content Generation (PCG) is powerful in creating high-quality 3D
contents, yet controlling it to produce desired shapes is difficult and often
requires extensive parameter tuning. Inverse Procedural Content Generation aims
to automatically find the best parameters under the input condition. However,
existing sampling-based and neural network-based methods still suffer from
numerous sample iterations or limited controllability. In this work, we present
DI-PCG, a novel and efficient method for Inverse PCG from general image
conditions. At its core is a lightweight diffusion transformer model, where PCG
parameters are directly treated as the denoising target and the observed images
as conditions to control parameter generation. DI-PCG is efficient and
effective. With only 7.6M network parameters and 30 GPU hours to train, it
demonstrates superior performance in recovering parameters accurately, and
generalizing well to in-the-wild images. Quantitative and qualitative
experiment results validate the effectiveness of DI-PCG in inverse PCG and
image-to-3D generation tasks. DI-PCG offers a promising approach for efficient
inverse PCG and represents a valuable exploration step towards a 3D generation
path that models how to construct a 3D asset using parametric models.
How to Synthesize Text Data without Model Collapse?
Authors: name: Xuekai Zhu, name: Daixuan Cheng, name: Hengli Li, name: Kaiyan Zhang, name: Ermo Hua, name: Xingtai Lv, name: Ning Ding, name: Zhouhan Lin, name: Zilong Zheng, name: Bowen Zhou
Published: 2024-12-19 09:43:39

Model collapse in synthetic data indicates that iterative training on
self-generated data leads to a gradual decline in performance. With the
proliferation of AI models, synthetic data will fundamentally reshape the web
data ecosystem. Future GPT-{n} models will inevitably be trained on a blend
of synthetic and human-produced data. In this paper, we focus on two questions:
what is the impact of synthetic data on language model training, and how to
synthesize data without model collapse? We first pre-train language models
across different proportions of synthetic data, revealing a negative
correlation between the proportion of synthetic data and model performance. We
further conduct statistical analysis on synthetic data to uncover
distributional shift phenomenon and over-concentration of n-gram features.
Inspired by the above findings, we propose token editing on human-produced data
to obtain semi-synthetic data. As a proof of concept, we theoretically
demonstrate that token-level editing can prevent model collapse, as the test
error is constrained by a finite upper bound. We conduct extensive experiments
on pre-training from scratch, continual pre-training, and supervised
fine-tuning. The results validate our theoretical proof that token-level
editing improves data quality and enhances model performance.
Affordance-Aware Object Insertion via Mask-Aware Dual Diffusion
Authors: name: Jixuan He, name: Wanhua Li, name: Ye Liu, name: Junsik Kim, name: Donglai Wei, name: Hanspeter Pfister
Published: 2024-12-19 02:23:13

As a common image editing operation, image composition involves integrating
foreground objects into background scenes. In this paper, we expand the
application of the concept of Affordance from human-centered image composition
tasks to a more general object-scene composition framework, addressing the
complex interplay between foreground objects and background scenes. Following
the principle of Affordance, we define the affordance-aware object insertion
task, which aims to seamlessly insert any object into any scene with various
position prompts. To address the limited data issue and incorporate this task,
we constructed the SAM-FB dataset, which contains over 3 million examples
across more than 3,000 object categories. Furthermore, we propose the
Mask-Aware Dual Diffusion (MADD) model, which utilizes a dual-stream
architecture to simultaneously denoise the RGB image and the insertion mask. By
explicitly modeling the insertion mask in the diffusion process, MADD
effectively facilitates the notion of affordance. Extensive experimental
results show that our method outperforms the state-of-the-art methods and
exhibits strong generalization performance on in-the-wild images. Please refer
to our code on https://github.com/KaKituken/affordance-aware-any.
UIP2P: Unsupervised Instruction-based Image Editing via Cycle Edit
  Consistency
Authors: name: Enis Simsar, name: Alessio Tonioni, name: Yongqin Xian, name: Thomas Hofmann, name: Federico Tombari
Published: 2024-12-19 18:59:58

We propose an unsupervised model for instruction-based image editing that
eliminates the need for ground-truth edited images during training. Existing
supervised methods depend on datasets containing triplets of input image,
edited image, and edit instruction. These are generated by either existing
editing methods or human-annotations, which introduce biases and limit their
generalization ability. Our method addresses these challenges by introducing a
novel editing mechanism called Cycle Edit Consistency (CEC), which applies
forward and backward edits in one training step and enforces consistency in
image and attention spaces. This allows us to bypass the need for ground-truth
edited images and unlock training for the first time on datasets comprising
either real image-caption pairs or image-caption-edit triplets. We empirically
show that our unsupervised technique performs better across a broader range of
edits with high fidelity and precision. By eliminating the need for
pre-existing datasets of triplets, reducing biases associated with supervised
methods, and proposing CEC, our work represents a significant advancement in
unblocking scaling of instruction-based image editing.
TOMG-Bench: Evaluating LLMs on Text-based Open Molecule Generation
Authors: name: Jiatong Li, name: Junxian Li, name: Yunqing Liu, name: Dongzhan Zhou, name: Qing Li
Published: 2024-12-19 08:51:16

In this paper, we propose Text-based Open Molecule Generation Benchmark
(TOMG-Bench), the first benchmark to evaluate the open-domain molecule
generation capability of LLMs. TOMG-Bench encompasses a dataset of three major
tasks: molecule editing (MolEdit), molecule optimization (MolOpt), and
customized molecule generation (MolCustom). Each task further contains three
subtasks, with each subtask comprising 5,000 test samples. Given the inherent
complexity of open molecule generation, we have also developed an automated
evaluation system that helps measure both the quality and the accuracy of the
generated molecules. Our comprehensive benchmarking of 25 LLMs reveals the
current limitations and potential areas for improvement in text-guided molecule
discovery. Furthermore, with the assistance of OpenMolIns, a specialized
instruction tuning dataset proposed for solving challenges raised by
TOMG-Bench, Llama3.1-8B could outperform all the open-source general LLMs, even
surpassing GPT-3.5-turbo by 46.5\% on TOMG-Bench. Our codes and datasets are
available through https://github.com/phenixace/TOMG-Bench.
Flowing from Words to Pixels: A Framework for Cross-Modality Evolution
Authors: name: Qihao Liu, name: Xi Yin, name: Alan Yuille, name: Andrew Brown, name: Mannat Singh
Published: 2024-12-19 18:59:56

Diffusion models, and their generalization, flow matching, have had a
remarkable impact on the field of media generation. Here, the conventional
approach is to learn the complex mapping from a simple source distribution of
Gaussian noise to the target media distribution. For cross-modal tasks such as
text-to-image generation, this same mapping from noise to image is learnt
whilst including a conditioning mechanism in the model. One key and thus far
relatively unexplored feature of flow matching is that, unlike Diffusion
models, they are not constrained for the source distribution to be noise.
Hence, in this paper, we propose a paradigm shift, and ask the question of
whether we can instead train flow matching models to learn a direct mapping
from the distribution of one modality to the distribution of another, thus
obviating the need for both the noise distribution and conditioning mechanism.
We present a general and simple framework, CrossFlow, for cross-modal flow
matching. We show the importance of applying Variational Encoders to the input
data, and introduce a method to enable Classifier-free guidance. Surprisingly,
for text-to-image, CrossFlow with a vanilla transformer without cross attention
slightly outperforms standard flow matching, and we show that it scales better
with training steps and model size, while also allowing for interesting latent
arithmetic which results in semantically meaningful edits in the output space.
To demonstrate the generalizability of our approach, we also show that
CrossFlow is on par with or outperforms the state-of-the-art for various
cross-modal / intra-modal mapping tasks, viz. image captioning, depth
estimation, and image super-resolution. We hope this paper contributes to
accelerating progress in cross-modal media generation.
Qwen2.5 Technical Report
Authors: name: Qwen, name: An Yang, name: Baosong Yang, name: Beichen Zhang, name: Binyuan Hui, name: Bo Zheng, name: Bowen Yu, name: Chengyuan Li, name: Dayiheng Liu, name: Fei Huang, name: Haoran Wei, name: Huan Lin, name: Jian Yang, name: Jianhong Tu, name: Jianwei Zhang, name: Jianxin Yang, name: Jiaxi Yang, name: Jingren Zhou, name: Junyang Lin, name: Kai Dang, name: Keming Lu, name: Keqin Bao, name: Kexin Yang, name: Le Yu, name: Mei Li, name: Mingfeng Xue, name: Pei Zhang, name: Qin Zhu, name: Rui Men, name: Runji Lin, name: Tianhao Li, name: Tingyu Xia, name: Xingzhang Ren, name: Xuancheng Ren, name: Yang Fan, name: Yang Su, name: Yichang Zhang, name: Yu Wan, name: Yuqiong Liu, name: Zeyu Cui, name: Zhenru Zhang, name: Zihan Qiu
Published: 2024-12-19 17:56:09

In this report, we introduce Qwen2.5, a comprehensive series of large
language models (LLMs) designed to meet diverse needs. Compared to previous
iterations, Qwen 2.5 has been significantly improved during both the
pre-training and post-training stages. In terms of pre-training, we have scaled
the high-quality pre-training datasets from the previous 7 trillion tokens to
18 trillion tokens. This provides a strong foundation for common sense, expert
knowledge, and reasoning capabilities. In terms of post-training, we implement
intricate supervised finetuning with over 1 million samples, as well as
multistage reinforcement learning. Post-training techniques enhance human
preference, and notably improve long text generation, structural data analysis,
and instruction following. To handle diverse and varied use cases effectively,
we present Qwen2.5 LLM series in rich sizes. Open-weight offerings include base
and instruction-tuned models, with quantized versions available. In addition,
for hosted solutions, the proprietary models currently include two
mixture-of-experts (MoE) variants: Qwen2.5-Turbo and Qwen2.5-Plus, both
available from Alibaba Cloud Model Studio. Qwen2.5 has demonstrated top-tier
performance on a wide range of benchmarks evaluating language understanding,
reasoning, mathematics, coding, human preference alignment, etc. Specifically,
the open-weight flagship Qwen2.5-72B-Instruct outperforms a number of open and
proprietary models and demonstrates competitive performance to the
state-of-the-art open-weight model, Llama-3-405B-Instruct, which is around 5
times larger. Qwen2.5-Turbo and Qwen2.5-Plus offer superior cost-effectiveness
while performing competitively against GPT-4o-mini and GPT-4o respectively.
Additionally, as the foundation, Qwen2.5 models have been instrumental in
training specialized models such as Qwen2.5-Math, Qwen2.5-Coder, QwQ, and
multimodal models.
MegaPairs: Massive Data Synthesis For Universal Multimodal Retrieval
Authors: name: Junjie Zhou, name: Zheng Liu, name: Ze Liu, name: Shitao Xiao, name: Yueze Wang, name: Bo Zhao, name: Chen Jason Zhang, name: Defu Lian, name: Yongping Xiong
Published: 2024-12-19 02:49:55

Despite the rapidly growing demand for multimodal retrieval, progress in this
field remains severely constrained by a lack of training data. In this paper,
we introduce MegaPairs, a novel data synthesis method that leverages vision
language models (VLMs) and open-domain images, together with a massive
synthetic dataset generated from this method. Our empirical analysis shows that
MegaPairs generates high-quality data, enabling the multimodal retriever to
significantly outperform the baseline model trained on 70times more data
from existing datasets. Moreover, since MegaPairs solely relies on general
image corpora and open-source VLMs, it can be easily scaled up, enabling
continuous improvements in retrieval performance. In this stage, we produced
more than 26 million training instances and trained several models of varying
sizes using this data. These new models achieve state-of-the-art zero-shot
performance across 4 popular composed image retrieval (CIR) benchmarks and the
highest overall performance on the 36 datasets provided by MMEB. They also
demonstrate notable performance improvements with additional downstream
fine-tuning. Our produced dataset, well-trained models, and data synthesis
pipeline will be made publicly available to facilitate the future development
of this field.