Orionnnn/hub-stats huggingface dataset:
Lost in Time: Clock and Calendar Understanding Challenges in Multimodal
  LLMs
Authors: name: Rohit Saxena, name: Aryo Pradipta Gema, name: Pasquale Minervini
Published: 2025-02-07 17:11:23

Understanding time from visual representations is a fundamental cognitive
skill, yet it remains a challenge for multimodal large language models (MLLMs).
In this work, we investigate the capabilities of MLLMs in interpreting time and
date through analogue clocks and yearly calendars. To facilitate this, we
curated a structured dataset comprising two subsets: 1) ClockQA,
which comprises various types of clock styles-standard, black-dial,
no-second-hand, Roman numeral, and arrow-hand clocks-paired with time related
questions; and 2) CalendarQA, which consists of yearly calendar
images with questions ranging from commonly known dates (e.g., Christmas, New
Year's Day) to computationally derived ones (e.g., the 100th or 153rd day of
the year). We aim to analyse how MLLMs can perform visual recognition,
numerical reasoning, and temporal inference when presented with time-related
visual data. Our evaluations show that despite recent advancements, reliably
understanding time remains a significant challenge for MLLMs.
No Task Left Behind: Isotropic Model Merging with Common and
  Task-Specific Subspaces
Authors: name: Daniel Marczak, name: Simone Magistri, name: Sebastian Cygert, name: Bartłomiej Twardowski, name: Andrew D. Bagdanov, name: Joost van de Weijer
Published: 2025-02-07 14:22:56

Model merging integrates the weights of multiple task-specific models into a
single multi-task model. Despite recent interest in the problem, a significant
performance gap between the combined and single-task models remains. In this
paper, we investigate the key characteristics of task matrices -- weight update
matrices applied to a pre-trained model -- that enable effective merging. We
show that alignment between singular components of task-specific and merged
matrices strongly correlates with performance improvement over the pre-trained
model. Based on this, we propose an isotropic merging framework that flattens
the singular value spectrum of task matrices, enhances alignment, and reduces
the performance gap. Additionally, we incorporate both common and task-specific
subspaces to further improve alignment and performance. Our proposed approach
achieves state-of-the-art performance across multiple scenarios, including
various sets of tasks and model scales. This work advances the understanding of
model merging dynamics, offering an effective methodology to merge models
without requiring additional training. Code is available at
https://github.com/danielm1405/iso-merging .
CMoE: Fast Carving of Mixture-of-Experts for Efficient LLM Inference
Authors: name: Zehua Pei, name: Lancheng Zou, name: Hui-Ling Zhen, name: Xianzhi Yu, name: Wulong Liu, name: Sinno Jialin Pan, name: Mingxuan Yuan, name: Bei Yu
Published: 2025-02-06 14:05:30

Large language models (LLMs) achieve impressive performance by scaling model
parameters, but this comes with significant inference overhead. Feed-forward
networks (FFNs), which dominate LLM parameters, exhibit high activation
sparsity in hidden neurons. To exploit this, researchers have proposed using a
mixture-of-experts (MoE) architecture, where only a subset of parameters is
activated. However, existing approaches often require extensive training data
and resources, limiting their practicality. We propose CMoE (Carved MoE), a
novel framework to efficiently carve MoE models from dense models. CMoE
achieves remarkable performance through efficient expert grouping and
lightweight adaptation. First, neurons are grouped into shared and routed
experts based on activation rates. Next, we construct a routing mechanism
without training from scratch, incorporating a differentiable routing process
and load balancing. Using modest data, CMoE produces a well-designed, usable
MoE from a 7B dense model within five minutes. With lightweight fine-tuning, it
achieves high-performance recovery in under an hour. We make our code publicly
available at https://github.com/JarvisPei/CMoE.
ARR: Question Answering with Large Language Models via Analyzing,
  Retrieving, and Reasoning
Authors: name: Yuwei Yin, name: Giuseppe Carenini
Published: 2025-02-07 06:30:33

Large language models (LLMs) achieve remarkable performance on challenging
benchmarks that are often structured as multiple-choice question-answering (QA)
tasks. Zero-shot Chain-of-Thought (CoT) prompting enhances reasoning in LLMs
but provides only vague and generic guidance ("think step by step"). This paper
introduces ARR, an intuitive and effective zero-shot prompting method that
explicitly incorporates three key steps in QA solving: analyzing the intent of
the question, retrieving relevant information, and reasoning step by step.
Comprehensive experiments across diverse and challenging QA tasks demonstrate
that ARR consistently improves the Baseline (without ARR prompting) and
outperforms CoT. Ablation and case studies further validate the positive
contributions of each component: analyzing, retrieving, and reasoning. Notably,
intent analysis plays a vital role in ARR. Additionally, extensive evaluations
across various model sizes, LLM series, and generation settings solidify the
effectiveness, robustness, and generalizability of ARR.
QuEST: Stable Training of LLMs with 1-Bit Weights and Activations
Authors: name: Andrei Panferov, name: Jiale Chen, name: Soroush Tabesh, name: Roberto L. Castro, name: Mahdi Nikdan, name: Dan Alistarh
Published: 2025-02-07 15:23:34

One approach to reducing the massive costs of large language models (LLMs) is
the use of quantized or sparse representations for training or deployment.
While post-training compression methods are very popular, the question of
obtaining even more accurate compressed models by directly training over such
representations, i.e., Quantization-Aware Training (QAT), is still open: for
example, a recent study (arXiv:2411.04330v2) put the "optimal" bit-width at
which models can be trained using QAT, while staying accuracy-competitive with
standard FP16/BF16 precision, at 8-bits weights and activations.
  We advance this state-of-the-art via a new method called QuEST, which is
Pareto-competitive with FP16, i.e., it provides better accuracy at lower model
size, while training models with weights and activations in 4-bits or less.
Moreover, QuEST allows stable training with 1-bit weights and activations.
QuEST achieves this by improving two key aspects of QAT methods: (1) accurate
and fast quantization of the (continuous) distributions of weights and
activations via Hadamard normalization and MSE-optimal fitting; (2) a new trust
gradient estimator based on the idea of explicitly minimizing the error between
the noisy gradient computed over quantized states and the "true" (but unknown)
full-precision gradient. Experiments on Llama-type architectures show that
QuEST induces stable scaling laws across the entire range of hardware-supported
precisions, and can be extended to sparse representations. We provide GPU
kernel support showing that models produced by QuEST can be executed
efficiently. Our code is available at https://github.com/IST-DASLab/QuEST.
Scaling Laws in Patchification: An Image Is Worth 50,176 Tokens And More
Authors: name: Feng Wang, name: Yaodong Yu, name: Guoyizhe Wei, name: Wei Shao, name: Yuyin Zhou, name: Alan Yuille, name: Cihang Xie
Published: 2025-02-06 03:01:38

Since the introduction of Vision Transformer (ViT), patchification has long
been regarded as a de facto image tokenization approach for plain visual
architectures. By compressing the spatial size of images, this approach can
effectively shorten the token sequence and reduce the computational cost of
ViT-like plain architectures. In this work, we aim to thoroughly examine the
information loss caused by this patchification-based compressive encoding
paradigm and how it affects visual understanding. We conduct extensive patch
size scaling experiments and excitedly observe an intriguing scaling law in
patchification: the models can consistently benefit from decreased patch sizes
and attain improved predictive performance, until it reaches the minimum patch
size of 1x1, i.e., pixel tokenization. This conclusion is broadly applicable
across different vision tasks, various input scales, and diverse architectures
such as ViT and the recent Mamba models. Moreover, as a by-product, we discover
that with smaller patches, task-specific decoder heads become less critical for
dense prediction. In the experiments, we successfully scale up the visual
sequence to an exceptional length of 50,176 tokens, achieving a competitive
test accuracy of 84.6% with a base-sized model on the ImageNet-1k benchmark. We
hope this study can provide insights and theoretical foundations for future
works of building non-compressive vision models. Code is available at
https://github.com/wangf3014/Patch_Scaling.
YINYANG-ALIGN: Benchmarking Contradictory Objectives and Proposing
  Multi-Objective Optimization based DPO for Text-to-Image Alignment
Authors: name: Amitava Das, name: Yaswanth Narsupalli, name: Gurpreet Singh, name: Vinija Jain, name: Vasu Sharma, name: Suranjana Trivedy, name: Aman Chadha, name: Amit Sheth
Published: 2025-02-05 18:46:20

Precise alignment in Text-to-Image (T2I) systems is crucial to ensure that
generated visuals not only accurately encapsulate user intents but also conform
to stringent ethical and aesthetic benchmarks. Incidents like the Google Gemini
fiasco, where misaligned outputs triggered significant public backlash,
underscore the critical need for robust alignment mechanisms. In contrast,
Large Language Models (LLMs) have achieved notable success in alignment.
Building on these advancements, researchers are eager to apply similar
alignment techniques, such as Direct Preference Optimization (DPO), to T2I
systems to enhance image generation fidelity and reliability.
  We present YinYangAlign, an advanced benchmarking framework that
systematically quantifies the alignment fidelity of T2I systems, addressing six
fundamental and inherently contradictory design objectives. Each pair
represents fundamental tensions in image generation, such as balancing
adherence to user prompts with creative modifications or maintaining diversity
alongside visual coherence. YinYangAlign includes detailed axiom datasets
featuring human prompts, aligned (chosen) responses, misaligned (rejected)
AI-generated outputs, and explanations of the underlying contradictions.
QLIP: Text-Aligned Visual Tokenization Unifies Auto-Regressive
  Multimodal Understanding and Generation
Authors: name: Yue Zhao, name: Fuzhao Xue, name: Scott Reed, name: Linxi Fan, name: Yuke Zhu, name: Jan Kautz, name: Zhiding Yu, name: Philipp Krähenbühl, name: De-An Huang
Published: 2025-02-07 18:59:57

We introduce Quantized Language-Image Pretraining (QLIP), a visual
tokenization method that combines state-of-the-art reconstruction quality with
state-of-the-art zero-shot image understanding. QLIP trains a
binary-spherical-quantization-based autoencoder with reconstruction and
language-image alignment objectives. We are the first to show that the two
objectives do not need to be at odds. We balance the two loss terms dynamically
during training and show that a two-stage training pipeline effectively mixes
the large-batch requirements of image-language pre-training with the memory
bottleneck imposed by the reconstruction objective. We validate the
effectiveness of QLIP for multimodal understanding and text-conditioned image
generation with a single model. Specifically, QLIP serves as a drop-in
replacement for the visual encoder for LLaVA and the image tokenizer for
LlamaGen with comparable or even better performance. Finally, we demonstrate
that QLIP enables a unified mixed-modality auto-regressive model for
understanding and generation.
MEETING DELEGATE: Benchmarking LLMs on Attending Meetings on Our Behalf
Authors: name: Lingxiang Hu, name: Shurun Yuan, name: Xiaoting Qin, name: Jue Zhang, name: Qingwei Lin, name: Dongmei Zhang, name: Saravan Rajmohan, name: Qi Zhang
Published: 2025-02-05 16:25:43

In contemporary workplaces, meetings are essential for exchanging ideas and
ensuring team alignment but often face challenges such as time consumption,
scheduling conflicts, and inefficient participation. Recent advancements in
Large Language Models (LLMs) have demonstrated their strong capabilities in
natural language generation and reasoning, prompting the question: can LLMs
effectively delegate participants in meetings? To explore this, we develop a
prototype LLM-powered meeting delegate system and create a comprehensive
benchmark using real meeting transcripts. Our evaluation reveals that GPT-4/4o
maintain balanced performance between active and cautious engagement
strategies. In contrast, Gemini 1.5 Pro tends to be more cautious, while Gemini
1.5 Flash and Llama3-8B/70B display more active tendencies. Overall, about 60\%
of responses address at least one key point from the ground-truth. However,
improvements are needed to reduce irrelevant or repetitive content and enhance
tolerance for transcription errors commonly found in real-world settings.
Additionally, we implement the system in practical settings and collect
real-world feedback from demos. Our findings underscore the potential and
challenges of utilizing LLMs as meeting delegates, offering valuable insights
into their practical application for alleviating the burden of meetings.
DuoGuard: A Two-Player RL-Driven Framework for Multilingual LLM
  Guardrails
Authors: name: Yihe Deng, name: Yu Yang, name: Junkai Zhang, name: Wei Wang, name: Bo Li
Published: 2025-02-07 18:45:03

The rapid advancement of large language models (LLMs) has increased the need
for guardrail models to ensure responsible use, particularly in detecting
unsafe and illegal content. While substantial safety data exist in English,
multilingual guardrail modeling remains underexplored due to the scarcity of
open-source safety data in other languages. To address this gap, we propose a
novel two-player Reinforcement Learning (RL) framework, where a generator and a
guardrail model co-evolve adversarially to produce high-quality synthetic data
for multilingual guardrail training. We theoretically formalize this
interaction as a two-player game, proving convergence to a Nash equilibrium.
Empirical evaluations show that our model \ours outperforms state-of-the-art
models, achieving nearly 10% improvement over LlamaGuard3 (8B) on English
benchmarks while being 4.5x faster at inference with a significantly smaller
model (0.5B). We achieve substantial advancements in multilingual safety tasks,
particularly in addressing the imbalance for lower-resource languages in a
collected real dataset. Ablation studies emphasize the critical role of
synthetic data generation in bridging the imbalance in open-source data between
English and other languages. These findings establish a scalable and efficient
approach to synthetic data generation, paving the way for improved multilingual
guardrail models to enhance LLM safety. Code, model, and data will be
open-sourced at https://github.com/yihedeng9/DuoGuard.
FlashVideo:Flowing Fidelity to Detail for Efficient High-Resolution
  Video Generation
Authors: name: Shilong Zhang, name: Wenbo Li, name: Shoufa Chen, name: Chongjian Ge, name: Peize Sun, name: Yida Zhang, name: Yi Jiang, name: Zehuan Yuan, name: Binyue Peng, name: Ping Luo
Published: 2025-02-07 18:59:59

DiT diffusion models have achieved great success in text-to-video generation,
leveraging their scalability in model capacity and data scale. High content and
motion fidelity aligned with text prompts, however, often require large model
parameters and a substantial number of function evaluations (NFEs). Realistic
and visually appealing details are typically reflected in high resolution
outputs, further amplifying computational demands especially for single stage
DiT models. To address these challenges, we propose a novel two stage
framework, FlashVideo, which strategically allocates model capacity and NFEs
across stages to balance generation fidelity and quality. In the first stage,
prompt fidelity is prioritized through a low resolution generation process
utilizing large parameters and sufficient NFEs to enhance computational
efficiency. The second stage establishes flow matching between low and high
resolutions, effectively generating fine details with minimal NFEs.
Quantitative and visual results demonstrate that FlashVideo achieves
state-of-the-art high resolution video generation with superior computational
efficiency. Additionally, the two-stage design enables users to preview the
initial output before committing to full resolution generation, thereby
significantly reducing computational costs and wait times as well as enhancing
commercial viability .
Fast Video Generation with Sliding Tile Attention
Authors: name: Peiyuan Zhang, name: Yongqi Chen, name: Runlong Su, name: Hangliang Ding, name: Ion Stoica, name: Zhenghong Liu, name: Hao Zhang
Published: 2025-02-06 21:17:09

Diffusion Transformers (DiTs) with 3D full attention power state-of-the-art
video generation, but suffer from prohibitive compute cost -- when generating
just a 5-second 720P video, attention alone takes 800 out of 945 seconds of
total inference time. This paper introduces sliding tile attention (STA) to
address this challenge. STA leverages the observation that attention scores in
pretrained video diffusion models predominantly concentrate within localized 3D
windows. By sliding and attending over the local spatial-temporal region, STA
eliminates redundancy from full attention. Unlike traditional token-wise
sliding window attention (SWA), STA operates tile-by-tile with a novel
hardware-aware sliding window design, preserving expressiveness while being
hardware-efficient. With careful kernel-level optimizations, STA offers the
first efficient 2D/3D sliding-window-like attention implementation, achieving
58.79% MFU. Precisely, STA accelerates attention by 2.8-17x over
FlashAttention-2 (FA2) and 1.6-10x over FlashAttention-3 (FA3). On the leading
video DiT, HunyuanVideo, STA reduces end-to-end latency from 945s (FA3) to 685s
without quality degradation, requiring no training. Enabling finetuning further
lowers latency to 268s with only a 0.09% drop on VBench.
AuraFusion360: Augmented Unseen Region Alignment for Reference-based
  360° Unbounded Scene Inpainting
Authors: name: Chung-Ho Wu, name: Yang-Jung Chen, name: Ying-Huan Chen, name: Jie-Ying Lee, name: Bo-Hsu Ke, name: Chun-Wei Tuan Mu, name: Yi-Chuan Huang, name: Chin-Yang Lin, name: Min-Hung Chen, name: Yen-Yu Lin, name: Yu-Lun Liu
Published: 2025-02-07 18:59:55

Three-dimensional scene inpainting is crucial for applications from virtual
reality to architectural visualization, yet existing methods struggle with view
consistency and geometric accuracy in 360{\deg} unbounded scenes. We present
AuraFusion360, a novel reference-based method that enables high-quality object
removal and hole filling in 3D scenes represented by Gaussian Splatting. Our
approach introduces (1) depth-aware unseen mask generation for accurate
occlusion identification, (2) Adaptive Guided Depth Diffusion, a zero-shot
method for accurate initial point placement without requiring additional
training, and (3) SDEdit-based detail enhancement for multi-view coherence. We
also introduce 360-USID, the first comprehensive dataset for 360{\deg}
unbounded scene inpainting with ground truth. Extensive experiments demonstrate
that AuraFusion360 significantly outperforms existing methods, achieving
superior perceptual quality while maintaining geometric accuracy across
dramatic viewpoint changes. See our project page for video results and the
dataset at https://kkennethwu.github.io/aurafusion360/.
Goku: Flow Based Video Generative Foundation Models
Authors: name: Shoufa Chen, name: Chongjian Ge, name: Yuqi Zhang, name: Yida Zhang, name: Fengda Zhu, name: Hao Yang, name: Hongxiang Hao, name: Hui Wu, name: Zhichao Lai, name: Yifei Hu, name: Ting-Che Lin, name: Shilong Zhang, name: Fu Li, name: Chuan Li, name: Xing Wang, name: Yanghua Peng, name: Peize Sun, name: Ping Luo, name: Yi Jiang, name: Zehuan Yuan, name: Bingyue Peng, name: Xiaobing Liu
Published: 2025-02-07 13:03:55

This paper introduces Goku, a state-of-the-art family of joint
image-and-video generation models leveraging rectified flow Transformers to
achieve industry-leading performance. We detail the foundational elements
enabling high-quality visual generation, including the data curation pipeline,
model architecture design, flow formulation, and advanced infrastructure for
efficient and robust large-scale training. The Goku models demonstrate superior
performance in both qualitative and quantitative evaluations, setting new
benchmarks across major tasks. Specifically, Goku achieves 0.76 on GenEval and
83.65 on DPG-Bench for text-to-image generation, and 84.85 on VBench for
text-to-video tasks. We believe that this work provides valuable insights and
practical advancements for the research community in developing joint
image-and-video generation models.
On-device Sora: Enabling Diffusion-Based Text-to-Video Generation for
  Mobile Devices
Authors: name: Bosung Kim, name: Kyuhwan Lee, name: Isu Jeong, name: Jungmin Cheon, name: Yeojin Lee, name: Seulki Lee
Published: 2025-02-05 05:42:29

We present On-device Sora, a first pioneering solution for diffusion-based
on-device text-to-video generation that operates efficiently on
smartphone-grade devices. Building on Open-Sora, On-device Sora applies three
novel techniques to address the challenges of diffusion-based text-to-video
generation on computation- and memory-limited mobile devices. First, Linear
Proportional Leap (LPL) reduces the excessive denoising steps required in video
diffusion through an efficient leap-based approach. Second, Temporal Dimension
Token Merging (TDTM) minimizes intensive token-processing computation in
attention layers by merging consecutive tokens along the temporal dimension.
Third, Concurrent Inference with Dynamic Loading (CI-DL) dynamically partitions
large models into smaller blocks and loads them into memory for concurrent
model inference, effectively addressing the challenges of limited device
memory. We implement On-device Sora on the iPhone 15 Pro, and the experimental
evaluations demonstrate that it is capable of generating high-quality videos on
the device, comparable to those produced by Open-Sora running on high-end GPUs.
These results show that On-device Sora enables efficient and high-quality video
generation on resource-constrained mobile devices, expanding accessibility,
ensuring user privacy, reducing dependence on cloud infrastructure, and
lowering associated costs. We envision the proposed On-device Sora as a
significant first step toward democratizing state-of-the-art generative
technologies, enabling video generation capabilities on commodity mobile and
embedded devices. The code implementation is publicly available at an GitHub
repository: https://github.com/eai-lab/On-device-Sora.