Orionnnn/hub-stats huggingface dataset:
ObjCtrl-2.5D: Training-free Object Control with Camera Poses
Authors: name: Zhouxia Wang, name: Yushi Lan, name: Shangchen Zhou, name: Chen Change Loy
Published: 2024-12-10 18:14:30

This study aims to achieve more precise and versatile object control in
image-to-video (I2V) generation. Current methods typically represent the
spatial movement of target objects with 2D trajectories, which often fail to
capture user intention and frequently produce unnatural results. To enhance
control, we present ObjCtrl-2.5D, a training-free object control approach that
uses a 3D trajectory, extended from a 2D trajectory with depth information, as
a control signal. By modeling object movement as camera movement, ObjCtrl-2.5D
represents the 3D trajectory as a sequence of camera poses, enabling object
motion control using an existing camera motion control I2V generation model
(CMC-I2V) without training. To adapt the CMC-I2V model originally designed for
global motion control to handle local object motion, we introduce a module to
isolate the target object from the background, enabling independent local
control. In addition, we devise an effective way to achieve more accurate
object control by sharing low-frequency warped latent within the object's
region across frames. Extensive experiments demonstrate that ObjCtrl-2.5D
significantly improves object control accuracy compared to training-free
methods and offers more diverse control capabilities than training-based
approaches using 2D trajectories, enabling complex effects like object
rotation. Code and results are available at
https://wzhouxiff.github.io/projects/ObjCtrl-2.5D/.
Perception Tokens Enhance Visual Reasoning in Multimodal Language Models
Authors: name: Mahtab Bigverdi, name: Zelun Luo, name: Cheng-Yu Hsieh, name: Ethan Shen, name: Dongping Chen, name: Linda G. Shapiro, name: Ranjay Krishna
Published: 2024-12-04 18:45:35

Multimodal language models (MLMs) still face challenges in fundamental visual
perception tasks where specialized models excel. Tasks requiring reasoning
about 3D structures benefit from depth estimation, and reasoning about 2D
object instances benefits from object detection. Yet, MLMs can not produce
intermediate depth or boxes to reason over. Finetuning MLMs on relevant data
doesn't generalize well and outsourcing computation to specialized vision tools
is too compute-intensive and memory-inefficient. To address this, we introduce
Perception Tokens, intrinsic image representations designed to assist reasoning
tasks where language is insufficient. Perception tokens act as auxiliary
reasoning tokens, akin to chain-of-thought prompts in language models. For
example, in a depth-related task, an MLM augmented with perception tokens can
reason by generating a depth map as tokens, enabling it to solve the problem
effectively. We propose AURORA, a training method that augments MLMs with
perception tokens for improved reasoning over visual inputs. AURORA leverages a
VQVAE to transform intermediate image representations, such as depth maps into
a tokenized format and bounding box tokens, which is then used in a multi-task
training framework. AURORA achieves notable improvements across counting
benchmarks: +10.8% on BLINK, +11.3% on CVBench, and +8.3% on SEED-Bench,
outperforming finetuning approaches in generalization across datasets. It also
improves on relative depth: over +6% on BLINK. With perception tokens, AURORA
expands the scope of MLMs beyond language-based reasoning, paving the way for
more effective visual reasoning capabilities.
Evaluating and Aligning CodeLLMs on Human Preference
Authors: name: Jian Yang, name: Jiaxi Yang, name: Ke Jin, name: Yibo Miao, name: Lei Zhang, name: Liqun Yang, name: Zeyu Cui, name: Yichang Zhang, name: Binyuan Hui, name: Junyang Lin
Published: 2024-12-06 17:40:38

Code large language models (codeLLMs) have made significant strides in code
generation. Most previous code-related benchmarks, which consist of various
programming exercises along with the corresponding test cases, are used as a
common measure to evaluate the performance and capabilities of code LLMs.
However, the current code LLMs focus on synthesizing the correct code snippet,
ignoring the alignment with human preferences, where the query should be
sampled from the practical application scenarios and the model-generated
responses should satisfy the human preference. To bridge the gap between the
model-generated response and human preference, we present a rigorous
human-curated benchmark CodeArena to emulate the complexity and diversity of
real-world coding tasks, where 397 high-quality samples spanning 40 categories
and 44 programming languages, carefully curated from user queries. Further, we
propose a diverse synthetic instruction corpus SynCode-Instruct (nearly 20B
tokens) by scaling instructions from the website to verify the effectiveness of
the large-scale synthetic instruction fine-tuning, where Qwen2.5-SynCoder
totally trained on synthetic instruction data can achieve top-tier performance
of open-source code LLMs. The results find performance differences between
execution-based benchmarks and CodeArena. Our systematic experiments of
CodeArena on 40+ LLMs reveal a notable performance gap between open SOTA code
LLMs (e.g. Qwen2.5-Coder) and proprietary LLMs (e.g., OpenAI o1), underscoring
the importance of the human preference
alignment.\url{https://codearenaeval.github.io/ }
Hidden in the Noise: Two-Stage Robust Watermarking for Images
Authors: name: Kasra Arabi, name: Benjamin Feuer, name: R. Teal Witter, name: Chinmay Hegde, name: Niv Cohen
Published: 2024-12-05 22:50:42

As the quality of image generators continues to improve, deepfakes become a
topic of considerable societal debate. Image watermarking allows responsible
model owners to detect and label their AI-generated content, which can mitigate
the harm. Yet, current state-of-the-art methods in image watermarking remain
vulnerable to forgery and removal attacks. This vulnerability occurs in part
because watermarks distort the distribution of generated images,
unintentionally revealing information about the watermarking techniques.
  In this work, we first demonstrate a distortion-free watermarking method for
images, based on a diffusion model's initial noise. However, detecting the
watermark requires comparing the initial noise reconstructed for an image to
all previously used initial noises. To mitigate these issues, we propose a
two-stage watermarking framework for efficient detection. During generation, we
augment the initial noise with generated Fourier patterns to embed information
about the group of initial noises we used. For detection, we (i) retrieve the
relevant group of noises, and (ii) search within the given group for an initial
noise that might match our image. This watermarking approach achieves
state-of-the-art robustness to forgery and removal against a large battery of
attacks.
UniReal: Universal Image Generation and Editing via Learning Real-world
  Dynamics
Authors: name: Xi Chen, name: Zhifei Zhang, name: He Zhang, name: Yuqian Zhou, name: Soo Ye Kim, name: Qing Liu, name: Yijun Li, name: Jianming Zhang, name: Nanxuan Zhao, name: Yilin Wang, name: Hui Ding, name: Zhe Lin, name: Hengshuang Zhao
Published: 2024-12-10 18:59:55

We introduce UniReal, a unified framework designed to address various image
generation and editing tasks. Existing solutions often vary by tasks, yet share
fundamental principles: preserving consistency between inputs and outputs while
capturing visual variations. Inspired by recent video generation models that
effectively balance consistency and variation across frames, we propose a
unifying approach that treats image-level tasks as discontinuous video
generation. Specifically, we treat varying numbers of input and output images
as frames, enabling seamless support for tasks such as image generation,
editing, customization, composition, etc. Although designed for image-level
tasks, we leverage videos as a scalable source for universal supervision.
UniReal learns world dynamics from large-scale videos, demonstrating advanced
capability in handling shadows, reflections, pose variation, and object
interaction, while also exhibiting emergent capability for novel applications.
Granite Guardian
Authors: name: Inkit Padhi, name: Manish Nagireddy, name: Giandomenico Cornacchia, name: Subhajit Chaudhury, name: Tejaswini Pedapati, name: Pierre Dognin, name: Keerthiram Murugesan, name: Erik Miehling, name: Martín Santillán Cooper, name: Kieran Fraser, name: Giulio Zizzo, name: Muhammad Zaid Hameed, name: Mark Purcell, name: Michael Desmond, name: Qian Pan, name: Inge Vejsbjerg, name: Elizabeth M. Daly, name: Michael Hind, name: Werner Geyer, name: Ambrish Rawat, name: Kush R. Varshney, name: Prasanna Sattigeri
Published: 2024-12-10 18:17:02

We introduce the Granite Guardian models, a suite of safeguards designed to
provide risk detection for prompts and responses, enabling safe and responsible
use in combination with any large language model (LLM). These models offer
comprehensive coverage across multiple risk dimensions, including social bias,
profanity, violence, sexual content, unethical behavior, jailbreaking, and
hallucination-related risks such as context relevance, groundedness, and answer
relevance for retrieval-augmented generation (RAG). Trained on a unique dataset
combining human annotations from diverse sources and synthetic data, Granite
Guardian models address risks typically overlooked by traditional risk
detection models, such as jailbreaks and RAG-specific issues. With AUC scores
of 0.871 and 0.854 on harmful content and RAG-hallucination-related benchmarks
respectively, Granite Guardian is the most generalizable and competitive model
available in the space. Released as open-source, Granite Guardian aims to
promote responsible AI development across the community.
  https://github.com/ibm-granite/granite-guardian
Fully Open Source Moxin-7B Technical Report
Authors: name: Pu Zhao, name: Xuan Shen, name: Zhenglun Kong, name: Yixin Shen, name: Sung-En Chang, name: Timothy Rupprecht, name: Lei Lu, name: Enfu Nan, name: Changdi Yang, name: Yumei He, name: Xingchen Xu, name: Yu Huang, name: Wei Wang, name: Yue Chen, name: Yong He, name: Yanzhi Wang
Published: 2024-12-08 02:01:46

Recently, Large Language Models (LLMs) have undergone a significant
transformation, marked by a rapid rise in both their popularity and
capabilities. Leading this evolution are proprietary LLMs like GPT-4 and
GPT-o1, which have captured widespread attention in the AI community due to
their remarkable performance and versatility. Simultaneously, open-source LLMs,
such as LLaMA and Mistral, have made great contributions to the ever-increasing
popularity of LLMs due to the ease to customize and deploy the models across
diverse applications. Although open-source LLMs present unprecedented
opportunities for innovation and research, the commercialization of LLMs has
raised concerns about transparency, reproducibility, and safety. Many
open-source LLMs fail to meet fundamental transparency requirements by
withholding essential components like training code and data, and some use
restrictive licenses whilst claiming to be "open-source," which may hinder
further innovations on LLMs. To mitigate this issue, we introduce Moxin 7B, a
fully open-source LLM developed in accordance with the Model Openness Framework
(MOF), a ranked classification system that evaluates AI models based on model
completeness and openness, adhering to principles of open science, open source,
open data, and open access. Our model achieves the highest MOF classification
level of "open science" through the comprehensive release of pre-training code
and configurations, training and fine-tuning datasets, and intermediate and
final checkpoints. Experiments show that our model achieves superior
performance in zero-shot evaluation compared with popular 7B models and
performs competitively in few-shot evaluation.
ILLUME: Illuminating Your LLMs to See, Draw, and Self-Enhance
Authors: name: Chunwei Wang, name: Guansong Lu, name: Junwei Yang, name: Runhui Huang, name: Jianhua Han, name: Lu Hou, name: Wei Zhang, name: Hang Xu
Published: 2024-12-09 17:11:50

In this paper, we introduce ILLUME, a unified multimodal large language model
(MLLM) that seamlessly integrates multimodal understanding and generation
capabilities within a single large language model through a unified next-token
prediction formulation. To address the large dataset size typically required
for image-text alignment, we propose to enhance data efficiency through the
design of a vision tokenizer that incorporates semantic information and a
progressive multi-stage training procedure. This approach reduces the dataset
size to just 15M for pretraining -- over four times fewer than what is
typically needed -- while achieving competitive or even superior performance
with existing unified MLLMs, such as Janus. Additionally, to promote
synergistic enhancement between understanding and generation capabilities,
which is under-explored in previous works, we introduce a novel self-enhancing
multimodal alignment scheme. This scheme supervises the MLLM to self-assess the
consistency between text descriptions and self-generated images, facilitating
the model to interpret images more accurately and avoid unrealistic and
incorrect predictions caused by misalignment in image generation. Based on
extensive experiments, our proposed ILLUME stands out and competes with
state-of-the-art unified MLLMs and specialized models across various benchmarks
for multimodal understanding, generation, and editing.
Maya: An Instruction Finetuned Multilingual Multimodal Model
Authors: name: Nahid Alam, name: Karthik Reddy Kanjula, name: Surya Guthikonda, name: Timothy Chung, name: Bala Krishna S Vegesna, name: Abhipsha Das, name: Anthony Susevski, name: Ryan Sze-Yin Chan, name: S M Iftekhar Uddin, name: Shayekh Bin Islam, name: Roshan Santhosh, name: Snegha A, name: Drishti Sharma, name: Chen Liu, name: Isha Chaturvedi, name: Genta Indra Winata, name: Ashvanth. S, name: Snehanshu Mukherjee, name: Alham Fikri Aji
Published: 2024-12-10 01:57:17

The rapid development of large Vision-Language Models (VLMs) has led to
impressive results on academic benchmarks, primarily in widely spoken
languages. However, significant gaps remain in the ability of current VLMs to
handle low-resource languages and varied cultural contexts, largely due to a
lack of high-quality, diverse, and safety-vetted data. Consequently, these
models often struggle to understand low-resource languages and cultural nuances
in a manner free from toxicity. To address these limitations, we introduce
Maya, an open-source Multimodal Multilingual model. Our contributions are
threefold: 1) a multilingual image-text pretraining dataset in eight languages,
based on the LLaVA pretraining dataset; 2) a thorough analysis of toxicity
within the LLaVA dataset, followed by the creation of a novel toxicity-free
version across eight languages; and 3) a multilingual image-text model
supporting these languages, enhancing cultural and linguistic comprehension in
vision-language tasks. Code available at https://github.com/nahidalam/maya.
Gated Delta Networks: Improving Mamba2 with Delta Rule
Authors: name: Songlin Yang, name: Jan Kautz, name: Ali Hatamizadeh
Published: 2024-12-09 13:09:04

Linear Transformers have gained attention as efficient alternatives to
standard Transformers, but their performance in retrieval and long-context
tasks has been limited. To address these limitations, recent work has explored
two distinct mechanisms: gating for adaptive memory control and the delta
update rule for precise memory modifications. We observe that these mechanisms
are complementary: gating enables rapid memory erasure while the delta rule
facilitates targeted updates. Building on this insight, we introduce the gated
delta rule and develop a parallel training algorithm optimized for modern
hardware. Our proposed architecture, Gated DeltaNet, consistently surpasses
existing models like Mamba2 and DeltaNet across multiple benchmarks, including
language modeling, common-sense reasoning, in-context retrieval, length
extrapolation, and long-context understanding. We further enhance performance
by developing hybrid architectures that combine Gated DeltaNet layers with
sliding window attention or Mamba2 layers, achieving both improved training
efficiency and superior task performance.
Turbo3D: Ultra-fast Text-to-3D Generation
Authors: name: Hanzhe Hu, name: Tianwei Yin, name: Fujun Luan, name: Yiwei Hu, name: Hao Tan, name: Zexiang Xu, name: Sai Bi, name: Shubham Tulsiani, name: Kai Zhang
Published: 2024-12-05 18:59:56

We present Turbo3D, an ultra-fast text-to-3D system capable of generating
high-quality Gaussian splatting assets in under one second. Turbo3D employs a
rapid 4-step, 4-view diffusion generator and an efficient feed-forward Gaussian
reconstructor, both operating in latent space. The 4-step, 4-view generator is
a student model distilled through a novel Dual-Teacher approach, which
encourages the student to learn view consistency from a multi-view teacher and
photo-realism from a single-view teacher. By shifting the Gaussian
reconstructor's inputs from pixel space to latent space, we eliminate the extra
image decoding time and halve the transformer sequence length for maximum
efficiency. Our method demonstrates superior 3D generation results compared to
previous baselines, while operating in a fraction of their runtime.
MAtCha Gaussians: Atlas of Charts for High-Quality Geometry and
  Photorealism From Sparse Views
Authors: name: Antoine Guédon, name: Tomoki Ichikawa, name: Kohei Yamashita, name: Ko Nishino
Published: 2024-12-09 18:55:23

We present a novel appearance model that simultaneously realizes explicit
high-quality 3D surface mesh recovery and photorealistic novel view synthesis
from sparse view samples. Our key idea is to model the underlying scene
geometry Mesh as an Atlas of Charts which we render with 2D Gaussian surfels
(MAtCha Gaussians). MAtCha distills high-frequency scene surface details from
an off-the-shelf monocular depth estimator and refines it through Gaussian
surfel rendering. The Gaussian surfels are attached to the charts on the fly,
satisfying photorealism of neural volumetric rendering and crisp geometry of a
mesh model, i.e., two seemingly contradicting goals in a single model. At the
core of MAtCha lies a novel neural deformation model and a structure loss that
preserve the fine surface details distilled from learned monocular depths while
addressing their fundamental scale ambiguities. Results of extensive
experimental validation demonstrate MAtCha's state-of-the-art quality of
surface reconstruction and photorealism on-par with top contenders but with
dramatic reduction in the number of input views and computational time. We
believe MAtCha will serve as a foundational tool for any visual application in
vision, graphics, and robotics that require explicit geometry in addition to
photorealism. Our project page is the following:
https://anttwo.github.io/matcha/
If You Can't Use Them, Recycle Them: Optimizing Merging at Scale
  Mitigates Performance Tradeoffs
Authors: name: Muhammad Khalifa, name: Yi-Chern Tan, name: Arash Ahmadian, name: Tom Hosking, name: Honglak Lee, name: Lu Wang, name: Ahmet Üstün, name: Tom Sherborne, name: Matthias Gallé
Published: 2024-12-05 13:12:51

Model merging has shown great promise at combining expert models, but the
benefit of merging is unclear when merging ``generalist'' models trained on
many tasks. We explore merging in the context of large (sim100B) models, by
recycling checkpoints that exhibit tradeoffs among different tasks.
Such checkpoints are often created in the process of developing a frontier
model, and many suboptimal ones are usually discarded. Given a pool of model
checkpoints obtained from different training runs (e.g., different stages,
objectives, hyperparameters, and data mixtures), which naturally show tradeoffs
across different language capabilities (e.g., instruction following vs. code
generation), we investigate whether merging can recycle such suboptimal models
into a Pareto-optimal one. Our optimization algorithm tunes the weight of each
checkpoint in a linear combination, resulting in a Pareto-optimal models that
outperforms both individual models and merge-based baselines. Further analysis
shows that good merges tend to include almost all checkpoints with with
non-zero weights, indicating that even seemingly bad initial checkpoints can
contribute to good final merges.
Around the World in 80 Timesteps: A Generative Approach to Global Visual
  Geolocation
Authors: name: Nicolas Dufour, name: David Picard, name: Vicky Kalogeiton, name: Loic Landrieu
Published: 2024-12-09 18:59:04

Global visual geolocation predicts where an image was captured on Earth.
Since images vary in how precisely they can be localized, this task inherently
involves a significant degree of ambiguity. However, existing approaches are
deterministic and overlook this aspect. In this paper, we aim to close the gap
between traditional geolocalization and modern generative methods. We propose
the first generative geolocation approach based on diffusion and Riemannian
flow matching, where the denoising process operates directly on the Earth's
surface. Our model achieves state-of-the-art performance on three visual
geolocation benchmarks: OpenStreetView-5M, YFCC-100M, and iNat21. In addition,
we introduce the task of probabilistic visual geolocation, where the model
predicts a probability distribution over all possible locations instead of a
single point. We introduce new metrics and baselines for this task,
demonstrating the advantages of our diffusion-based approach. Codes and models
will be made available.
MotionShop: Zero-Shot Motion Transfer in Video Diffusion Models with
  Mixture of Score Guidance
Authors: name: Hidir Yesiltepe, name: Tuna Han Salih Meral, name: Connor Dunlop, name: Pinar Yanardag
Published: 2024-12-06 18:59:17

In this work, we propose the first motion transfer approach in diffusion
transformer through Mixture of Score Guidance (MSG), a theoretically-grounded
framework for motion transfer in diffusion models. Our key theoretical
contribution lies in reformulating conditional score to decompose motion score
and content score in diffusion models. By formulating motion transfer as a
mixture of potential energies, MSG naturally preserves scene composition and
enables creative scene transformations while maintaining the integrity of
transferred motion patterns. This novel sampling operates directly on
pre-trained video diffusion models without additional training or fine-tuning.
Through extensive experiments, MSG demonstrates successful handling of diverse
scenarios including single object, multiple objects, and cross-object motion
transfer as well as complex camera motion transfer. Additionally, we introduce
MotionBench, the first motion transfer dataset consisting of 200 source videos
and 1000 transferred motions, covering single/multi-object transfers, and
complex camera motions.
Global and Dense Embeddings of Earth: Major TOM Floating in the Latent
  Space
Authors: name: Mikolaj Czerkawski, name: Marcin Kluczek, name: Jędrzej S. Bojanowski
Published: 2024-12-07 09:49:47

With the ever-increasing volumes of the Earth observation data present in the
archives of large programmes such as Copernicus, there is a growing need for
efficient vector representations of the underlying raw data. The approach of
extracting feature representations from pretrained deep neural networks is a
powerful approach that can provide semantic abstractions of the input data.
However, the way this is done for imagery archives containing geospatial data
has not yet been defined. In this work, an extension is proposed to an existing
community project, Major TOM, focused on the provision and standardization of
open and free AI-ready datasets for Earth observation. Furthermore, four global
and dense embedding datasets are released openly and for free along with the
publication of this manuscript, resulting in the most comprehensive global open
dataset of geospatial visual embeddings in terms of covered Earth's surface.