Orionnnn/hub-stats huggingface dataset:
Graph Generative Pre-trained Transformer
Authors: name: Xiaohui Chen, name: Yinkai Wang, name: Jiaxing He, name: Yuanqi Du, name: Soha Hassoun, name: Xiaolin Xu, name: Li-Ping Liu
Published: 2025-01-02 05:44:11

Graph generation is a critical task in numerous domains, including molecular
design and social network analysis, due to its ability to model complex
relationships and structured data. While most modern graph generative models
utilize adjacency matrix representations, this work revisits an alternative
approach that represents graphs as sequences of node set and edge set. We
advocate for this approach due to its efficient encoding of graphs and propose
a novel representation. Based on this representation, we introduce the Graph
Generative Pre-trained Transformer (G2PT), an auto-regressive model that learns
graph structures via next-token prediction. To further exploit G2PT's
capabilities as a general-purpose foundation model, we explore fine-tuning
strategies for two downstream applications: goal-oriented generation and graph
property prediction. We conduct extensive experiments across multiple datasets.
Results indicate that G2PT achieves superior generative performance on both
generic graph and molecule datasets. Furthermore, G2PT exhibits strong
adaptability and versatility in downstream tasks from molecular design to
property prediction.
LUSIFER: Language Universal Space Integration for Enhanced Multilingual
  Embeddings with Large Language Models
Authors: name: Hieu Man, name: Nghia Trung Ngo, name: Viet Dac Lai, name: Ryan A. Rossi, name: Franck Dernoncourt, name: Thien Huu Nguyen
Published: 2025-01-01 15:43:07

Recent advancements in large language models (LLMs) based embedding models
have established new state-of-the-art benchmarks for text embedding tasks,
particularly in dense vector-based retrieval. However, these models
predominantly focus on English, leaving multilingual embedding capabilities
largely unexplored. To address this limitation, we present LUSIFER, a novel
zero-shot approach that adapts LLM-based embedding models for multilingual
tasks without requiring multilingual supervision. LUSIFER's architecture
combines a multilingual encoder, serving as a language-universal learner, with
an LLM-based embedding model optimized for embedding-specific tasks. These
components are seamlessly integrated through a minimal set of trainable
parameters that act as a connector, effectively transferring the multilingual
encoder's language understanding capabilities to the specialized embedding
model. Additionally, to comprehensively evaluate multilingual embedding
performance, we introduce a new benchmark encompassing 5 primary embedding
tasks, 123 diverse datasets, and coverage across 14 languages. Extensive
experimental results demonstrate that LUSIFER significantly enhances the
multilingual performance across various embedding tasks, particularly for
medium and low-resource languages, without requiring explicit multilingual
training data.
EnerVerse: Envisioning Embodied Future Space for Robotics Manipulation
Authors: name: Siyuan Huang, name: Liliang Chen, name: Pengfei Zhou, name: Shengcong Chen, name: Zhengkai Jiang, name: Yue Hu, name: Peng Gao, name: Hongsheng Li, name: Maoqing Yao, name: Guanghui Ren
Published: 2025-01-03 17:00:33

We introduce EnerVerse, a comprehensive framework for embodied future space
generation specifically designed for robotic manipulation tasks. EnerVerse
seamlessly integrates convolutional and bidirectional attention mechanisms for
inner-chunk space modeling, ensuring low-level consistency and continuity.
Recognizing the inherent redundancy in video data, we propose a sparse memory
context combined with a chunkwise unidirectional generative paradigm to enable
the generation of infinitely long sequences. To further augment robotic
capabilities, we introduce the Free Anchor View (FAV) space, which provides
flexible perspectives to enhance observation and analysis. The FAV space
mitigates motion modeling ambiguity, removes physical constraints in confined
environments, and significantly improves the robot's generalization and
adaptability across various tasks and settings. To address the prohibitive
costs and labor intensity of acquiring multi-camera observations, we present a
data engine pipeline that integrates a generative model with 4D Gaussian
Splatting (4DGS). This pipeline leverages the generative model's robust
generalization capabilities and the spatial constraints provided by 4DGS,
enabling an iterative enhancement of data quality and diversity, thus creating
a data flywheel effect that effectively narrows the sim-to-real gap. Finally,
our experiments demonstrate that the embodied future space generation prior
substantially enhances policy predictive capabilities, resulting in improved
overall performance, particularly in long-range robotic manipulation tasks.
VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interaction
Authors: name: Chaoyou Fu, name: Haojia Lin, name: Xiong Wang, name: Yi-Fan Zhang, name: Yunhang Shen, name: Xiaoyu Liu, name: Yangze Li, name: Zuwei Long, name: Heting Gao, name: Ke Li, name: Xiawu Zheng, name: Rongrong Ji, name: Xing Sun, name: Caifeng Shan, name: Ran He
Published: 2025-01-03 18:59:52

Recent Multimodal Large Language Models (MLLMs) have typically focused on
integrating visual and textual modalities, with less emphasis placed on the
role of speech in enhancing interaction. However, speech plays a crucial role
in multimodal dialogue systems, and implementing high-performance in both
vision and speech tasks remains a significant challenge due to the fundamental
modality differences. In this paper, we propose a carefully designed
multi-stage training methodology that progressively trains LLM to understand
both visual and speech information, ultimately enabling fluent vision and
speech interaction. Our approach not only preserves strong vision-language
capacity, but also enables efficient speech-to-speech dialogue capabilities
without separate ASR and TTS modules, significantly accelerating multimodal
end-to-end response speed. By comparing our method against state-of-the-art
counterparts across benchmarks for image, video, and speech tasks, we
demonstrate that our model is equipped with both strong visual and speech
capabilities, making near real-time vision and speech interaction.
Virgo: A Preliminary Exploration on Reproducing o1-like MLLM
Authors: name: Yifan Du, name: Zikang Liu, name: Yifan Li, name: Wayne Xin Zhao, name: Yuqi Huo, name: Bingning Wang, name: Weipeng Chen, name: Zheng Liu, name: Zhongyuan Wang, name: Ji-Rong Wen
Published: 2025-01-03 17:14:16

Recently, slow-thinking reasoning systems, built upon large language models
(LLMs), have garnered widespread attention by scaling the thinking time during
inference. There is also growing interest in adapting this capability to
multimodal large language models (MLLMs). Given that MLLMs handle more complex
data semantics across different modalities, it is intuitively more challenging
to implement multimodal slow-thinking systems.
  To address this issue, in this paper, we explore a straightforward approach
by fine-tuning a capable MLLM with a small amount of textual long-form thought
data, resulting in a multimodal slow-thinking system, Virgo (Visual reasoning
with long thought). We find that these long-form reasoning processes, expressed
in natural language, can be effectively transferred to MLLMs. Moreover, it
seems that such textual reasoning data can be even more effective than visual
reasoning data in eliciting the slow-thinking capacities of MLLMs. While this
work is preliminary, it demonstrates that slow-thinking capacities are
fundamentally associated with the language model component, which can be
transferred across modalities or domains. This finding can be leveraged to
guide the development of more powerful slow-thinking reasoning systems. We
release our resources at https://github.com/RUCAIBox/Virgo.
Unifying Specialized Visual Encoders for Video Language Models
Authors: name: Jihoon Chung, name: Tyler Zhu, name: Max Gonzalez Saez-Diez, name: Juan Carlos Niebles, name: Honglu Zhou, name: Olga Russakovsky
Published: 2025-01-02 18:59:45

The recent advent of Large Language Models (LLMs) has ushered sophisticated
reasoning capabilities into the realm of video through Video Large Language
Models (VideoLLMs). However, VideoLLMs currently rely on a single vision
encoder for all of their visual processing, which limits the amount and type of
visual information that can be conveyed to the LLM. Our method, MERV,
Multi-Encoder Representation of Videos, instead leverages multiple frozen
visual encoders to create a unified representation of a video, providing the
VideoLLM with a comprehensive set of specialized visual knowledge.
Spatio-temporally aligning the features from each encoder allows us to tackle a
wider range of open-ended and multiple-choice video understanding questions and
outperform prior state-of-the-art works. MERV is up to 3.7% better in accuracy
than Video-LLaVA across the standard suite video understanding benchmarks,
while also having a better Video-ChatGPT score. We also improve upon SeViLA,
the previous best on zero-shot Perception Test accuracy, by 2.2%. MERV
introduces minimal extra parameters and trains faster than equivalent
single-encoder methods while parallelizing the visual processing. Finally, we
provide qualitative evidence that MERV successfully captures domain knowledge
from each of its encoders. Our results offer promising directions in utilizing
multiple vision encoders for comprehensive video understanding.
Nested Attention: Semantic-aware Attention Values for Concept
  Personalization
Authors: name: Or Patashnik, name: Rinon Gal, name: Daniil Ostashev, name: Sergey Tulyakov, name: Kfir Aberman, name: Daniel Cohen-Or
Published: 2025-01-02 18:52:11

Personalizing text-to-image models to generate images of specific subjects
across diverse scenes and styles is a rapidly advancing field. Current
approaches often face challenges in maintaining a balance between identity
preservation and alignment with the input text prompt. Some methods rely on a
single textual token to represent a subject, which limits expressiveness, while
others employ richer representations but disrupt the model's prior, diminishing
prompt alignment. In this work, we introduce Nested Attention, a novel
mechanism that injects a rich and expressive image representation into the
model's existing cross-attention layers. Our key idea is to generate
query-dependent subject values, derived from nested attention layers that learn
to select relevant subject features for each region in the generated image. We
integrate these nested layers into an encoder-based personalization method, and
show that they enable high identity preservation while adhering to input text
prompts. Our approach is general and can be trained on various domains.
Additionally, its prior preservation allows us to combine multiple personalized
subjects from different domains in a single image.
Population Aware Diffusion for Time Series Generation
Authors: name: Yang Li, name: Han Meng, name: Zhenyu Bi, name: Ingolv T. Urnes, name: Haipeng Chen
Published: 2025-01-01 17:53:43

Diffusion models have shown promising ability in generating high-quality time
series (TS) data. Despite the initial success, existing works mostly focus on
the authenticity of data at the individual level, but pay less attention to
preserving the population-level properties on the entire dataset. Such
population-level properties include value distributions for each dimension and
distributions of certain functional dependencies (e.g., cross-correlation, CC)
between different dimensions. For instance, when generating house energy
consumption TS data, the value distributions of the outside temperature and the
kitchen temperature should be preserved, as well as the distribution of CC
between them. Preserving such TS population-level properties is critical in
maintaining the statistical insights of the datasets, mitigating model bias,
and augmenting downstream tasks like TS prediction. Yet, it is often overlooked
by existing models. Hence, data generated by existing models often bear
distribution shifts from the original data. We propose Population-aware
Diffusion for Time Series (PaD-TS), a new TS generation model that better
preserves the population-level properties. The key novelties of PaD-TS include
1) a new training method explicitly incorporating TS population-level property
preservation, and 2) a new dual-channel encoder model architecture that better
captures the TS data structure. Empirical results in major benchmark datasets
show that PaD-TS can improve the average CC distribution shift score between
real and synthetic data by 5.9x while maintaining a performance comparable to
state-of-the-art models on individual-level authenticity.
Rethinking Addressing in Language Models via Contexualized Equivariant
  Positional Encoding
Authors: name: Jiajun Zhu, name: Peihao Wang, name: Ruisi Cai, name: Jason D. Lee, name: Pan Li, name: Zhangyang Wang
Published: 2025-01-01 03:23:00

Transformers rely on both content-based and position-based addressing
mechanisms to make predictions, but existing positional encoding techniques
often diminish the effectiveness of position-based addressing. Many current
methods enforce rigid patterns in attention maps, limiting the ability to model
long-range dependencies and adapt to diverse tasks. Additionally, most
positional encodings are learned as general biases, lacking the specialization
required for different instances within a dataset. To address this, we propose
conTextualized equivariAnt Position
Embedding (TAPE), a novel framework that enhances
positional embeddings by incorporating sequence content across layers. TAPE
introduces dynamic, context-aware positional encodings, overcoming the
constraints of traditional fixed patterns. By enforcing permutation and
orthogonal equivariance, TAPE ensures the stability of positional encodings
during updates, improving robustness and adaptability. Our method can be easily
integrated into pre-trained transformers, offering parameter-efficient
fine-tuning with minimal overhead. Extensive experiments shows that TAPE
achieves superior performance in language modeling, arithmetic reasoning, and
long-context retrieval tasks compared to existing positional embedding
techniques.
LTX-Video: Realtime Video Latent Diffusion
Authors: name: Yoav HaCohen, name: Nisan Chiprut, name: Benny Brazowski, name: Daniel Shalem, name: Dudu Moshe, name: Eitan Richardson, name: Eran Levin, name: Guy Shiran, name: Nir Zabari, name: Ori Gordon, name: Poriya Panet, name: Sapir Weissbuch, name: Victor Kulikov, name: Yaki Bitterman, name: Zeev Melumian, name: Ofir Bibi
Published: 2024-12-30 19:00:25

We introduce LTX-Video, a transformer-based latent diffusion model that
adopts a holistic approach to video generation by seamlessly integrating the
responsibilities of the Video-VAE and the denoising transformer. Unlike
existing methods, which treat these components as independent, LTX-Video aims
to optimize their interaction for improved efficiency and quality. At its core
is a carefully designed Video-VAE that achieves a high compression ratio of
1:192, with spatiotemporal downscaling of 32 x 32 x 8 pixels per token, enabled
by relocating the patchifying operation from the transformer's input to the
VAE's input. Operating in this highly compressed latent space enables the
transformer to efficiently perform full spatiotemporal self-attention, which is
essential for generating high-resolution videos with temporal consistency.
However, the high compression inherently limits the representation of fine
details. To address this, our VAE decoder is tasked with both latent-to-pixel
conversion and the final denoising step, producing the clean result directly in
pixel space. This approach preserves the ability to generate fine details
without incurring the runtime cost of a separate upsampling module. Our model
supports diverse use cases, including text-to-video and image-to-video
generation, with both capabilities trained simultaneously. It achieves
faster-than-real-time generation, producing 5 seconds of 24 fps video at
768x512 resolution in just 2 seconds on an Nvidia H100 GPU, outperforming all
existing models of similar scale. The source code and pre-trained models are
publicly available, setting a new benchmark for accessible and scalable video
generation.
SeedVR: Seeding Infinity in Diffusion Transformer Towards Generic Video
  Restoration
Authors: name: Jianyi Wang, name: Zhijie Lin, name: Meng Wei, name: Yang Zhao, name: Ceyuan Yang, name: Chen Change Loy, name: Lu Jiang
Published: 2025-01-02 16:19:48

Video restoration poses non-trivial challenges in maintaining fidelity while
recovering temporally consistent details from unknown degradations in the wild.
Despite recent advances in diffusion-based restoration, these methods often
face limitations in generation capability and sampling efficiency. In this
work, we present SeedVR, a diffusion transformer designed to handle real-world
video restoration with arbitrary length and resolution. The core design of
SeedVR lies in the shifted window attention that facilitates effective
restoration on long video sequences. SeedVR further supports variable-sized
windows near the boundary of both spatial and temporal dimensions, overcoming
the resolution constraints of traditional window attention. Equipped with
contemporary practices, including causal video autoencoder, mixed image and
video training, and progressive training, SeedVR achieves highly-competitive
performance on both synthetic and real-world benchmarks, as well as
AI-generated videos. Extensive experiments demonstrate SeedVR's superiority
over existing methods for generic video restoration.
MapEval: A Map-Based Evaluation of Geo-Spatial Reasoning in Foundation
  Models
Authors: name: Mahir Labib Dihan, name: Md Tanvir Hassan, name: Md Tanvir Parvez, name: Md Hasebul Hasan, name: Md Almash Alam, name: Muhammad Aamir Cheema, name: Mohammed Eunus Ali, name: Md Rizwan Parvez
Published: 2024-12-31 07:20:32

Recent advancements in foundation models have enhanced AI systems'
capabilities in autonomous tool usage and reasoning. However, their ability in
location or map-based reasoning - which improves daily life by optimizing
navigation, facilitating resource discovery, and streamlining logistics - has
not been systematically studied. To bridge this gap, we introduce MapEval, a
benchmark designed to assess diverse and complex map-based user queries with
geo-spatial reasoning. MapEval features three task types (textual, API-based,
and visual) that require collecting world information via map tools, processing
heterogeneous geo-spatial contexts (e.g., named entities, travel distances,
user reviews or ratings, images), and compositional reasoning, which all
state-of-the-art foundation models find challenging. Comprising 700 unique
multiple-choice questions about locations across 180 cities and 54 countries,
MapEval evaluates foundation models' ability to handle spatial relationships,
map infographics, travel planning, and navigation challenges. Using MapEval, we
conducted a comprehensive evaluation of 28 prominent foundation models. While
no single model excelled across all tasks, Claude-3.5-Sonnet, GPT-4o, and
Gemini-1.5-Pro achieved competitive performance overall. However, substantial
performance gaps emerged, particularly in MapEval, where agents with
Claude-3.5-Sonnet outperformed GPT-4o and Gemini-1.5-Pro by 16% and 21%,
respectively, and the gaps became even more amplified when compared to
open-source LLMs. Our detailed analyses provide insights into the strengths and
weaknesses of current models, though all models still fall short of human
performance by more than 20% on average, struggling with complex map images and
rigorous geo-spatial reasoning. This gap highlights MapEval's critical role in
advancing general-purpose foundation models with stronger geo-spatial
understanding.
MapQaTor: A System for Efficient Annotation of Map Query Datasets
Authors: name: Mahir Labib Dihan, name: Mohammed Eunus Ali, name: Md Rizwan Parvez
Published: 2024-12-30 15:33:19

Mapping and navigation services like Google Maps, Apple Maps, Openstreet
Maps, are essential for accessing various location-based data, yet they often
struggle to handle natural language geospatial queries. Recent advancements in
Large Language Models (LLMs) show promise in question answering (QA), but
creating reliable geospatial QA datasets from map services remains challenging.
We introduce MapQaTor, a web application that streamlines the creation of
reproducible, traceable map-based QA datasets. With its plug-and-play
architecture, MapQaTor enables seamless integration with any maps API, allowing
users to gather and visualize data from diverse sources with minimal setup. By
caching API responses, the platform ensures consistent ground truth, enhancing
the reliability of the data even as real-world information evolves. MapQaTor
centralizes data retrieval, annotation, and visualization within a single
platform, offering a unique opportunity to evaluate the current state of
LLM-based geospatial reasoning while advancing their capabilities for improved
geospatial understanding. Evaluation metrics show that, MapQaTor speeds up the
annotation process by at least 30 times compared to manual methods,
underscoring its potential for developing geospatial resources, such as complex
map reasoning datasets. The website is live at: https://mapqator.github.io/ and
a demo video is available at: https://youtu.be/7_aV9Wmhs6Q.
Understanding and Mitigating Bottlenecks of State Space Models through
  the Lens of Recency and Over-smoothing
Authors: name: Peihao Wang, name: Ruisi Cai, name: Yuehao Wang, name: Jiajun Zhu, name: Pragya Srivastava, name: Zhangyang Wang, name: Pan Li
Published: 2024-12-31 22:06:39

Structured State Space Models (SSMs) have emerged as alternatives to
transformers. While SSMs are often regarded as effective in capturing
long-sequence dependencies, we rigorously demonstrate that they are inherently
limited by strong recency bias. Our empirical studies also reveal that this
bias impairs the models' ability to recall distant information and introduces
robustness issues. Our scaling experiments then discovered that deeper
structures in SSMs can facilitate the learning of long contexts. However,
subsequent theoretical analysis reveals that as SSMs increase in depth, they
exhibit another inevitable tendency toward over-smoothing, e.g., token
representations becoming increasingly indistinguishable. This fundamental
dilemma between recency and over-smoothing hinders the scalability of existing
SSMs. Inspired by our theoretical findings, we propose to polarize two channels
of the state transition matrices in SSMs, setting them to zero and one,
respectively, simultaneously addressing recency bias and over-smoothing.
Experiments demonstrate that our polarization technique consistently enhances
the associative recall accuracy of long-range tokens and unlocks SSMs to
benefit further from deeper architectures. All source codes are released at
https://github.com/VITA-Group/SSM-Bottleneck.