Orionnnn/hub-stats huggingface dataset:
Beyond Sight: Finetuning Generalist Robot Policies with Heterogeneous
  Sensors via Language Grounding
Authors: name: Joshua Jones, name: Oier Mees, name: Carmelo Sferrazza, name: Kyle Stachowicz, name: Pieter Abbeel, name: Sergey Levine
Published: 2025-01-08 18:57:33

Interacting with the world is a multi-sensory experience: achieving effective
general-purpose interaction requires making use of all available modalities --
including vision, touch, and audio -- to fill in gaps from partial observation.
For example, when vision is occluded reaching into a bag, a robot should rely
on its senses of touch and sound. However, state-of-the-art generalist robot
policies are typically trained on large datasets to predict robot actions
solely from visual and proprioceptive observations. In this work, we propose
FuSe, a novel approach that enables finetuning visuomotor generalist policies
on heterogeneous sensor modalities for which large datasets are not readily
available by leveraging natural language as a common cross-modal grounding. We
combine a multimodal contrastive loss with a sensory-grounded language
generation loss to encode high-level semantics. In the context of robot
manipulation, we show that FuSe enables performing challenging tasks that
require reasoning jointly over modalities such as vision, touch, and sound in a
zero-shot setting, such as multimodal prompting, compositional cross-modal
prompting, and descriptions of objects it interacts with. We show that the same
recipe is applicable to widely different generalist policies, including both
diffusion-based generalist policies and large vision-language-action (VLA)
models. Extensive experiments in the real world show that FuSeis able to
increase success rates by over 20% compared to all considered baselines.
MINIMA: Modality Invariant Image Matching
Authors: name: Xingyu Jiang, name: Jiangwei Ren, name: Zizhuo Li, name: Xin Zhou, name: Dingkang Liang, name: Xiang Bai
Published: 2024-12-27 02:39:50

Image matching for both cross-view and cross-modality plays a critical role
in multimodal perception. In practice, the modality gap caused by different
imaging systems/styles poses great challenges to the matching task. Existing
works try to extract invariant features for specific modalities and train on
limited datasets, showing poor generalization. In this paper, we present
MINIMA, a unified image matching framework for multiple cross-modal cases.
Without pursuing fancy modules, our MINIMA aims to enhance universal
performance from the perspective of data scaling up. For such purpose, we
propose a simple yet effective data engine that can freely produce a large
dataset containing multiple modalities, rich scenarios, and accurate matching
labels. Specifically, we scale up the modalities from cheap but rich RGB-only
matching data, by means of generative models. Under this setting, the matching
labels and rich diversity of the RGB dataset are well inherited by the
generated multimodal data. Benefiting from this, we construct MD-syn, a new
comprehensive dataset that fills the data gap for general multimodal image
matching. With MD-syn, we can directly train any advanced matching pipeline on
randomly selected modality pairs to obtain cross-modal ability. Extensive
experiments on in-domain and zero-shot matching tasks, including 19
cross-modal cases, demonstrate that our MINIMA can significantly outperform the
baselines and even surpass modality-specific methods. The dataset and code are
available at https://github.com/LSXI7/MINIMA .
Towards Best Practices for Open Datasets for LLM Training
Authors: name: Stefan Baack, name: Stella Biderman, name: Kasia Odrozek, name: Aviya Skowron, name: Ayah Bdeir, name: Jillian Bommarito, name: Jennifer Ding, name: Maximilian Gahntz, name: Paul Keller, name: Pierre-Carl Langlais, name: Greg Lindahl, name: Sebastian Majstorovic, name: Nik Marda, name: Guilherme Penedo, name: Maarten Van Segbroeck, name: Jennifer Wang, name: Leandro von Werra, name: Mitchell Baker, name: Julie Belião, name: Kasia Chmielinski, name: Marzieh Fadaee, name: Lisa Gutermuth, name: Hynek Kydlíček, name: Greg Leppert, name: EM Lewis-Jong, name: Solana Larsen, name: Shayne Longpre, name: Angela Oduor Lungati, name: Cullen Miller, name: Victor Miller, name: Max Ryabinin, name: Kathleen Siminyu, name: Andrew Strait, name: Mark Surman, name: Anna Tumadóttir, name: Maurice Weber, name: Rebecca Weiss, name: Lee White, name: Thomas Wolf
Published: 2025-01-14 17:18:05

Many AI companies are training their large language models (LLMs) on data
without the permission of the copyright owners. The permissibility of doing so
varies by jurisdiction: in countries like the EU and Japan, this is allowed
under certain restrictions, while in the United States, the legal landscape is
more ambiguous. Regardless of the legal status, concerns from creative
producers have led to several high-profile copyright lawsuits, and the threat
of litigation is commonly cited as a reason for the recent trend towards
minimizing the information shared about training datasets by both corporate and
public interest actors. This trend in limiting data information causes harm by
hindering transparency, accountability, and innovation in the broader ecosystem
by denying researchers, auditors, and impacted individuals access to the
information needed to understand AI models.
  While this could be mitigated by training language models on open access and
public domain data, at the time of writing, there are no such models (trained
at a meaningful scale) due to the substantial technical and sociological
challenges in assembling the necessary corpus. These challenges include
incomplete and unreliable metadata, the cost and complexity of digitizing
physical records, and the diverse set of legal and technical skills required to
ensure relevance and responsibility in a quickly changing landscape. Building
towards a future where AI systems can be trained on openly licensed data that
is responsibly curated and governed requires collaboration across legal,
technical, and policy domains, along with investments in metadata standards,
digitization, and fostering a culture of openness.
Trusted Machine Learning Models Unlock Private Inference for Problems
  Currently Infeasible with Cryptography
Authors: name: Ilia Shumailov, name: Daniel Ramage, name: Sarah Meiklejohn, name: Peter Kairouz, name: Florian Hartmann, name: Borja Balle, name: Eugene Bagdasarian
Published: 2025-01-15 17:28:53

We often interact with untrusted parties. Prioritization of privacy can limit
the effectiveness of these interactions, as achieving certain goals
necessitates sharing private data. Traditionally, addressing this challenge has
involved either seeking trusted intermediaries or constructing cryptographic
protocols that restrict how much data is revealed, such as multi-party
computations or zero-knowledge proofs. While significant advances have been
made in scaling cryptographic approaches, they remain limited in terms of the
size and complexity of applications they can be used for. In this paper, we
argue that capable machine learning models can fulfill the role of a trusted
third party, thus enabling secure computations for applications that were
previously infeasible. In particular, we describe Trusted Capable Model
Environments (TCMEs) as an alternative approach for scaling secure computation,
where capable machine learning model(s) interact under input/output
constraints, with explicit information flow control and explicit statelessness.
This approach aims to achieve a balance between privacy and computational
efficiency, enabling private inference where classical cryptographic solutions
are currently infeasible. We describe a number of use cases that are enabled by
TCME, and show that even some simple classic cryptographic problems can already
be solved with TCME. Finally, we outline current limitations and discuss the
path forward in implementing them.
Parameter-Inverted Image Pyramid Networks for Visual Perception and
  Multimodal Understanding
Authors: name: Zhaokai Wang, name: Xizhou Zhu, name: Xue Yang, name: Gen Luo, name: Hao Li, name: Changyao Tian, name: Wenhan Dou, name: Junqi Ge, name: Lewei Lu, name: Yu Qiao, name: Jifeng Dai
Published: 2025-01-14 01:57:41

Image pyramids are widely adopted in top-performing methods to obtain
multi-scale features for precise visual perception and understanding. However,
current image pyramids use the same large-scale model to process multiple
resolutions of images, leading to significant computational cost. To address
this challenge, we propose a novel network architecture, called
Parameter-Inverted Image Pyramid Networks (PIIP). Specifically, PIIP uses
pretrained models (ViTs or CNNs) as branches to process multi-scale images,
where images of higher resolutions are processed by smaller network branches to
balance computational cost and performance. To integrate information from
different spatial scales, we further propose a novel cross-branch feature
interaction mechanism. To validate PIIP, we apply it to various perception
models and a representative multimodal large language model called LLaVA, and
conduct extensive experiments on various tasks such as object detection,
segmentation, image classification and multimodal understanding. PIIP achieves
superior performance compared to single-branch and existing multi-resolution
approaches with lower computational cost. When applied to InternViT-6B, a
large-scale vision foundation model, PIIP can improve its performance by 1%-2%
on detection and segmentation with only 40%-60% of the original computation,
finally achieving 60.0 box AP on MS COCO and 59.7 mIoU on ADE20K. For
multimodal understanding, our PIIP-LLaVA achieves 73.0% accuracy on TextVQA and
74.5% on MMBench with only 2.8M training data. Our code is released at
https://github.com/OpenGVLab/PIIP.
Multimodal LLMs Can Reason about Aesthetics in Zero-Shot
Authors: name: Ruixiang Jiang, name: Changwen Chen
Published: 2025-01-15 18:56:22

We present the first study on how Multimodal LLMs' (MLLMs) reasoning ability
shall be elicited to evaluate the aesthetics of artworks. To facilitate this
investigation, we construct MM-StyleBench, a novel high-quality dataset for
benchmarking artistic stylization. We then develop a principled method for
human preference modeling and perform a systematic correlation analysis between
MLLMs' responses and human preference. Our experiments reveal an inherent
hallucination issue of MLLMs in art evaluation, associated with response
subjectivity. ArtCoT is proposed, demonstrating that art-specific task
decomposition and the use of concrete language boost MLLMs' reasoning ability
for aesthetics. Our findings offer valuable insights into MLLMs for art and can
benefit a wide range of downstream applications, such as style transfer and
artistic image generation. Code available at
https://github.com/songrise/MLLM4Art.
CityDreamer4D: Compositional Generative Model of Unbounded 4D Cities
Authors: name: Haozhe Xie, name: Zhaoxi Chen, name: Fangzhou Hong, name: Ziwei Liu
Published: 2025-01-15 17:59:56

3D scene generation has garnered growing attention in recent years and has
made significant progress. Generating 4D cities is more challenging than 3D
scenes due to the presence of structurally complex, visually diverse objects
like buildings and vehicles, and heightened human sensitivity to distortions in
urban environments. To tackle these issues, we propose CityDreamer4D, a
compositional generative model specifically tailored for generating unbounded
4D cities. Our main insights are 1) 4D city generation should separate dynamic
objects (e.g., vehicles) from static scenes (e.g., buildings and roads), and 2)
all objects in the 4D scene should be composed of different types of neural
fields for buildings, vehicles, and background stuff. Specifically, we propose
Traffic Scenario Generator and Unbounded Layout Generator to produce dynamic
traffic scenarios and static city layouts using a highly compact BEV
representation. Objects in 4D cities are generated by combining stuff-oriented
and instance-oriented neural fields for background stuff, buildings, and
vehicles. To suit the distinct characteristics of background stuff and
instances, the neural fields employ customized generative hash grids and
periodic positional embeddings as scene parameterizations. Furthermore, we
offer a comprehensive suite of datasets for city generation, including OSM,
GoogleEarth, and CityTopia. The OSM dataset provides a variety of real-world
city layouts, while the Google Earth and CityTopia datasets deliver
large-scale, high-quality city imagery complete with 3D instance annotations.
Leveraging its compositional design, CityDreamer4D supports a range of
downstream applications, such as instance editing, city stylization, and urban
simulation, while delivering state-of-the-art performance in generating
realistic 4D cities.
MMDocIR: Benchmarking Multi-Modal Retrieval for Long Documents
Authors: name: Kuicai Dong, name: Yujing Chang, name: Xin Deik Goh, name: Dexun Li, name: Ruiming Tang, name: Yong Liu
Published: 2025-01-15 14:30:13

Multi-modal document retrieval is designed to identify and retrieve various
forms of multi-modal content, such as figures, tables, charts, and layout
information from extensive documents. Despite its significance, there is a
notable lack of a robust benchmark to effectively evaluate the performance of
systems in multi-modal document retrieval. To address this gap, this work
introduces a new benchmark, named as MMDocIR, encompassing two distinct tasks:
page-level and layout-level retrieval. The former focuses on localizing the
most relevant pages within a long document, while the latter targets the
detection of specific layouts, offering a more fine-grained granularity than
whole-page analysis. A layout can refer to a variety of elements such as
textual paragraphs, equations, figures, tables, or charts. The MMDocIR
benchmark comprises a rich dataset featuring expertly annotated labels for
1,685 questions and bootstrapped labels for 173,843 questions, making it a
pivotal resource for advancing multi-modal document retrieval for both training
and evaluation. Through rigorous experiments, we reveal that (i) visual
retrievers significantly outperform their text counterparts, (ii) MMDocIR train
set can effectively benefit the training process of multi-modal document
retrieval and (iii) text retrievers leveraging on VLM-text perform much better
than those using OCR-text. These findings underscores the potential advantages
of integrating visual elements for multi-modal document retrieval.
RepVideo: Rethinking Cross-Layer Representation for Video Generation
Authors: name: Chenyang Si, name: Weichen Fan, name: Zhengyao Lv, name: Ziqi Huang, name: Yu Qiao, name: Ziwei Liu
Published: 2025-01-15 18:20:37

Video generation has achieved remarkable progress with the introduction of
diffusion models, which have significantly improved the quality of generated
videos. However, recent research has primarily focused on scaling up model
training, while offering limited insights into the direct impact of
representations on the video generation process. In this paper, we initially
investigate the characteristics of features in intermediate layers, finding
substantial variations in attention maps across different layers. These
variations lead to unstable semantic representations and contribute to
cumulative differences between features, which ultimately reduce the similarity
between adjacent frames and negatively affect temporal coherence. To address
this, we propose RepVideo, an enhanced representation framework for
text-to-video diffusion models. By accumulating features from neighboring
layers to form enriched representations, this approach captures more stable
semantic information. These enhanced representations are then used as inputs to
the attention mechanism, thereby improving semantic expressiveness while
ensuring feature consistency across adjacent frames. Extensive experiments
demonstrate that our RepVideo not only significantly enhances the ability to
generate accurate spatial appearances, such as capturing complex spatial
relationships between multiple objects, but also improves temporal consistency
in video generation.
Ouroboros-Diffusion: Exploring Consistent Content Generation in
  Tuning-free Long Video Diffusion
Authors: name: Jingyuan Chen, name: Fuchen Long, name: Jie An, name: Zhaofan Qiu, name: Ting Yao, name: Jiebo Luo, name: Tao Mei
Published: 2025-01-15 18:59:15

The first-in-first-out (FIFO) video diffusion, built on a pre-trained
text-to-video model, has recently emerged as an effective approach for
tuning-free long video generation. This technique maintains a queue of video
frames with progressively increasing noise, continuously producing clean frames
at the queue's head while Gaussian noise is enqueued at the tail. However,
FIFO-Diffusion often struggles to keep long-range temporal consistency in the
generated videos due to the lack of correspondence modeling across frames. In
this paper, we propose Ouroboros-Diffusion, a novel video denoising framework
designed to enhance structural and content (subject) consistency, enabling the
generation of consistent videos of arbitrary length. Specifically, we introduce
a new latent sampling technique at the queue tail to improve structural
consistency, ensuring perceptually smooth transitions among frames. To enhance
subject consistency, we devise a Subject-Aware Cross-Frame Attention (SACFA)
mechanism, which aligns subjects across frames within short segments to achieve
better visual coherence. Furthermore, we introduce self-recurrent guidance.
This technique leverages information from all previous cleaner frames at the
front of the queue to guide the denoising of noisier frames at the end,
fostering rich and contextual global information interaction. Extensive
experiments of long video generation on the VBench benchmark demonstrate the
superiority of our Ouroboros-Diffusion, particularly in terms of subject
consistency, motion smoothness, and temporal consistency.
XMusic: Towards a Generalized and Controllable Symbolic Music Generation
  Framework
Authors: name: Sida Tian, name: Can Zhang, name: Wei Yuan, name: Wei Tan, name: Wenjie Zhu
Published: 2025-01-15 14:08:44

In recent years, remarkable advancements in artificial intelligence-generated
content (AIGC) have been achieved in the fields of image synthesis and text
generation, generating content comparable to that produced by humans. However,
the quality of AI-generated music has not yet reached this standard, primarily
due to the challenge of effectively controlling musical emotions and ensuring
high-quality outputs. This paper presents a generalized symbolic music
generation framework, XMusic, which supports flexible prompts (i.e., images,
videos, texts, tags, and humming) to generate emotionally controllable and
high-quality symbolic music. XMusic consists of two core components, XProjector
and XComposer. XProjector parses the prompts of various modalities into
symbolic music elements (i.e., emotions, genres, rhythms and notes) within the
projection space to generate matching music. XComposer contains a Generator and
a Selector. The Generator generates emotionally controllable and melodious
music based on our innovative symbolic music representation, whereas the
Selector identifies high-quality symbolic music by constructing a multi-task
learning scheme involving quality assessment, emotion recognition, and genre
recognition tasks. In addition, we build XMIDI, a large-scale symbolic music
dataset that contains 108,023 MIDI files annotated with precise emotion and
genre labels. Objective and subjective evaluations show that XMusic
significantly outperforms the current state-of-the-art methods with impressive
music quality. Our XMusic has been awarded as one of the nine Highlights of
Collectibles at WAIC 2023. The project homepage of XMusic is
https://xmusic-project.github.io.
MatchAnything: Universal Cross-Modality Image Matching with Large-Scale
  Pre-Training
Authors: name: Xingyi He, name: Hao Yu, name: Sida Peng, name: Dongli Tan, name: Zehong Shen, name: Hujun Bao, name: Xiaowei Zhou
Published: 2025-01-13 18:37:36

Image matching, which aims to identify corresponding pixel locations between
images, is crucial in a wide range of scientific disciplines, aiding in image
registration, fusion, and analysis. In recent years, deep learning-based image
matching algorithms have dramatically outperformed humans in rapidly and
accurately finding large amounts of correspondences. However, when dealing with
images captured under different imaging modalities that result in significant
appearance changes, the performance of these algorithms often deteriorates due
to the scarcity of annotated cross-modal training data. This limitation hinders
applications in various fields that rely on multiple image modalities to obtain
complementary information. To address this challenge, we propose a large-scale
pre-training framework that utilizes synthetic cross-modal training signals,
incorporating diverse data from various sources, to train models to recognize
and match fundamental structures across images. This capability is transferable
to real-world, unseen cross-modality image matching tasks. Our key finding is
that the matching model trained with our framework achieves remarkable
generalizability across more than eight unseen cross-modality registration
tasks using the same network weight, substantially outperforming existing
methods, whether designed for generalization or tailored for specific tasks.
This advancement significantly enhances the applicability of image matching
technologies across various scientific disciplines and paves the way for new
applications in multi-modality human and artificial intelligence analysis and
beyond.
3DIS-FLUX: simple and efficient multi-instance generation with DiT
  rendering
Authors: name: Dewei Zhou, name: Ji Xie, name: Zongxin Yang, name: Yi Yang
Published: 2025-01-09 10:34:00

The growing demand for controllable outputs in text-to-image generation has
driven significant advancements in multi-instance generation (MIG), enabling
users to define both instance layouts and attributes. Currently, the
state-of-the-art methods in MIG are primarily adapter-based. However, these
methods necessitate retraining a new adapter each time a more advanced model is
released, resulting in significant resource consumption. A methodology named
Depth-Driven Decoupled Instance Synthesis (3DIS) has been introduced, which
decouples MIG into two distinct phases: 1) depth-based scene construction and
2) detail rendering with widely pre-trained depth control models. The 3DIS
method requires adapter training solely during the scene construction phase,
while enabling various models to perform training-free detail rendering.
Initially, 3DIS focused on rendering techniques utilizing U-Net architectures
such as SD1.5, SD2, and SDXL, without exploring the potential of recent
DiT-based models like FLUX. In this paper, we present 3DIS-FLUX, an extension
of the 3DIS framework that integrates the FLUX model for enhanced rendering
capabilities. Specifically, we employ the FLUX.1-Depth-dev model for depth map
controlled image generation and introduce a detail renderer that manipulates
the Attention Mask in FLUX's Joint Attention mechanism based on layout
information. This approach allows for the precise rendering of fine-grained
attributes of each instance. Our experimental results indicate that 3DIS-FLUX,
leveraging the FLUX model, outperforms the original 3DIS method, which utilized
SD2 and SDXL, and surpasses current state-of-the-art adapter-based methods in
terms of both performance and image quality. Project Page:
https://limuloo.github.io/3DIS/.