Orionnnn/hub-stats huggingface dataset:
Promote, Suppress, Iterate: How Language Models Answer One-to-Many
  Factual Queries
Authors: name: Tianyi Lorena Yan, name: Robin Jia
Published: 2025-02-27 19:23:15

To answer one-to-many factual queries (e.g., listing cities of a country), a
language model (LM) must simultaneously recall knowledge and avoid repeating
previous answers. How are these two subtasks implemented and integrated
internally? Across multiple datasets and models, we identify a
promote-then-suppress mechanism: the model first recalls all answers, and then
suppresses previously generated ones. Specifically, LMs use both the subject
and previous answer tokens to perform knowledge recall, with attention
propagating subject information and MLPs promoting the answers. Then, attention
attends to and suppresses previous answer tokens, while MLPs amplify the
suppression signal. Our mechanism is corroborated by extensive experimental
evidence: in addition to using early decoding and causal tracing, we analyze
how components use different tokens by introducing both Token Lens, which
decodes aggregated attention updates from specified tokens, and a knockout
method that analyzes changes in MLP outputs after removing attention to
specified tokens. Overall, we provide new insights into how LMs' internal
components interact with different input tokens to support complex factual
recall. Code is available at
https://github.com/Lorenayannnnn/how-lms-answer-one-to-many-factual-queries.
VACE: All-in-One Video Creation and Editing
Authors: name: Zeyinzi Jiang, name: Zhen Han, name: Chaojie Mao, name: Jingfeng Zhang, name: Yulin Pan, name: Yu Liu
Published: 2025-03-10 17:57:04

Diffusion Transformer has demonstrated powerful capability and scalability in
generating high-quality images and videos. Further pursuing the unification of
generation and editing tasks has yielded significant progress in the domain of
image content creation. However, due to the intrinsic demands for consistency
across both temporal and spatial dynamics, achieving a unified approach for
video synthesis remains challenging. We introduce VACE, which enables users to
perform Video tasks within an All-in-one framework for Creation and Editing.
These tasks include reference-to-video generation, video-to-video editing, and
masked video-to-video editing. Specifically, we effectively integrate the
requirements of various tasks by organizing video task inputs, such as editing,
reference, and masking, into a unified interface referred to as the Video
Condition Unit (VCU). Furthermore, by utilizing a Context Adapter structure, we
inject different task concepts into the model using formalized representations
of temporal and spatial dimensions, allowing it to handle arbitrary video
synthesis tasks flexibly. Extensive experiments demonstrate that the unified
model of VACE achieves performance on par with task-specific models across
various subtasks. Simultaneously, it enables diverse applications through
versatile task combinations. Project page:
https://ali-vilab.github.io/VACE-Page/.
REF-VLM: Triplet-Based Referring Paradigm for Unified Visual Decoding
Authors: name: Yan Tai, name: Luhao Zhu, name: Zhiqiang Chen, name: Ynan Ding, name: Yiying Dong, name: Xiaohong Liu, name: Guodong Guo
Published: 2025-03-10 14:59:14

Multimodal Large Language Models (MLLMs) demonstrate robust zero-shot
capabilities across diverse vision-language tasks after training on mega-scale
datasets. However, dense prediction tasks, such as semantic segmentation and
keypoint detection, pose significant challenges for MLLMs when represented
solely as text outputs. Simultaneously, current MLLMs utilizing latent
embeddings for visual task decoding generally demonstrate limited adaptability
to both multi-task learning and multi-granularity scenarios. In this work, we
present REF-VLM, an end-to-end framework for unified training of various visual
decoding tasks. To address complex visual decoding scenarios, we introduce the
Triplet-Based Referring Paradigm (TRP), which explicitly decouples three
critical dimensions in visual decoding tasks through a triplet structure:
concepts, decoding types, and targets. TRP employs symbolic delimiters to
enforce structured representation learning, enhancing the parsability and
interpretability of model outputs. Additionally, we construct Visual-Task
Instruction Following Dataset (VTInstruct), a large-scale multi-task dataset
containing over 100 million multimodal dialogue samples across 25 task types.
Beyond text inputs and outputs, VT-Instruct incorporates various visual prompts
such as point, box, scribble, and mask, and generates outputs composed of text
and visual units like box, keypoint, depth and mask. The combination of
different visual prompts and visual units generates a wide variety of task
types, expanding the applicability of REF-VLM significantly. Both qualitative
and quantitative experiments demonstrate that our REF-VLM outperforms other
MLLMs across a variety of standard benchmarks. The code, dataset, and demo
available at https://github.com/MacavityT/REF-VLM.
Unleashing the Potential of Large Language Models for Text-to-Image
  Generation through Autoregressive Representation Alignment
Authors: name: Xing Xie, name: Jiawei Liu, name: Ziyue Lin, name: Huijie Fan, name: Zhi Han, name: Yandong Tang, name: Liangqiong Qu
Published: 2025-03-10 13:49:28

We present Autoregressive Representation Alignment (ARRA), a new training
framework that unlocks global-coherent text-to-image generation in
autoregressive LLMs without architectural changes. Unlike prior work that
requires complex architectural redesigns, ARRA aligns LLM hidden states with
visual representations from external visual foundational models via a global
visual alignment loss and a hybrid token, <HYBNEXT>. This token enforces dual
constraints: local next-token prediction and global semantic distillation,
enabling LLMs to implicitly learn spatial and contextual coherence while
retaining their original autoregressive paradigm. Extensive experiments
validate ARRA's plug-and-play versatility. When training from
text-generation-only LLMs or random initialization, ARRA reduces FID by 25.5%
(MIMIC-CXR), 8.8% (DeepEyeNet), and 7.5% (ImageNet) for advanced autoregressive
LLMs like Chameleon and LlamaGen, all without framework modifications. For
domain adaption, ARRA aligns general-purpose LLMs with specialized models
(e.g., BioMedCLIP), achieving an 18.6% FID reduction over direct fine-tuning on
medical imaging (MIMIC-CXR). By demonstrating that training objective redesign
-- not just architectural innovation -- can resolve cross-modal global
coherence challenges, ARRA offers a complementary paradigm for advancing
autoregressive models. Code and models will be released to advance
autoregressive image generation.
Symbolic Mixture-of-Experts: Adaptive Skill-based Routing for
  Heterogeneous Reasoning
Authors: name: Justin Chih-Yao Chen, name: Sukwon Yun, name: Elias Stengel-Eskin, name: Tianlong Chen, name: Mohit Bansal
Published: 2025-03-07 18:03:13

Combining existing pre-trained expert LLMs is a promising avenue for scalably
tackling large-scale and diverse tasks. However, selecting experts at the task
level is often too coarse-grained, as heterogeneous tasks may require different
expertise for each instance. To enable adaptive instance-level mixing of
pre-trained LLM experts, we propose Symbolic-MoE, a symbolic, text-based, and
gradient-free Mixture-of-Experts framework. Symbolic-MoE takes a fine-grained
approach to selection by emphasizing skills, e.g., algebra in math or molecular
biology in biomedical reasoning. We propose a skill-based recruiting strategy
that dynamically selects the most relevant set of expert LLMs for diverse
reasoning tasks based on their strengths. Each selected expert then generates
its own reasoning, resulting in k outputs from k experts, which are then
synthesized into a final high-quality response by an aggregator chosen based on
its ability to integrate diverse reasoning outputs. We show that Symbolic-MoE's
instance-level expert selection improves performance by a large margin but --
when implemented naively -- can introduce a high computational overhead due to
the need for constant model loading and offloading. To address this, we
implement a batch inference strategy that groups instances based on their
assigned experts, loading each model only once. This allows us to integrate 16
expert models on 1 GPU with a time cost comparable to or better than prior
multi-agent baselines using 4 GPUs. Through extensive evaluations on diverse
benchmarks (MMLU-Pro, GPQA, AIME, and MedMCQA), we demonstrate that
Symbolic-MoE outperforms strong LLMs like GPT4o-mini, as well as multi-agent
approaches, with an absolute average improvement of 8.15% over the best
multi-agent baseline. Moreover, Symbolic-MoE removes the need for expensive
multi-round discussions, outperforming discussion baselines with less
computation.
A Data-Centric Revisit of Pre-Trained Vision Models for Robot Learning
Authors: name: Xin Wen, name: Bingchen Zhao, name: Yilun Chen, name: Jiangmiao Pang, name: Xiaojuan Qi
Published: 2025-03-10 06:18:31

Pre-trained vision models (PVMs) are fundamental to modern robotics, yet
their optimal configuration remains unclear. Through systematic evaluation, we
find that while DINO and iBOT outperform MAE across visuomotor control and
perception tasks, they struggle when trained on non-(single-)object-centric
(NOC) data--a limitation strongly correlated with their diminished ability to
learn object-centric representations. This investigation indicates that the
ability to form object-centric representations from the non-object-centric
robotics dataset is the key to success for PVMs. Motivated by this discovery,
we designed SlotMIM, a method that induces object-centric representations by
introducing a semantic bottleneck to reduce the number of prototypes to
encourage the emergence of objectness as well as cross-view consistency
regularization for encouraging multiview invariance. Our experiments encompass
pre-training on object-centric, scene-centric, web-crawled, and ego-centric
data. Across all settings, our approach learns transferrable representations
and achieves significant improvements over prior work in image recognition,
scene understanding, and robot learning evaluations. When scaled up with
million-scale datasets, our method also demonstrates superior data efficiency
and scalability. Our code and models are publicly available at
https://github.com/CVMI-Lab/SlotMIM.
TRCE: Towards Reliable Malicious Concept Erasure in Text-to-Image
  Diffusion Models
Authors: name: Ruidong Chen, name: Honglin Guo, name: Lanjun Wang, name: Chenyu Zhang, name: Weizhi Nie, name: An-An Liu
Published: 2025-03-10 14:37:53

Recent advances in text-to-image diffusion models enable photorealistic image
generation, but they also risk producing malicious content, such as NSFW
images. To mitigate risk, concept erasure methods are studied to facilitate the
model to unlearn specific concepts. However, current studies struggle to fully
erase malicious concepts implicitly embedded in prompts (e.g., metaphorical
expressions or adversarial prompts) while preserving the model's normal
generation capability. To address this challenge, our study proposes TRCE,
using a two-stage concept erasure strategy to achieve an effective trade-off
between reliable erasure and knowledge preservation. Firstly, TRCE starts by
erasing the malicious semantics implicitly embedded in textual prompts. By
identifying a critical mapping objective(i.e., the [EoT] embedding), we
optimize the cross-attention layers to map malicious prompts to contextually
similar prompts but with safe concepts. This step prevents the model from being
overly influenced by malicious semantics during the denoising process.
Following this, considering the deterministic properties of the sampling
trajectory of the diffusion model, TRCE further steers the early denoising
prediction toward the safe direction and away from the unsafe one through
contrastive learning, thus further avoiding the generation of malicious
content. Finally, we conduct comprehensive evaluations of TRCE on multiple
malicious concept erasure benchmarks, and the results demonstrate its
effectiveness in erasing malicious concepts while better preserving the model's
original generation ability. The code is available at:
http://github.com/ddgoodgood/TRCE. CAUTION: This paper includes model-generated
content that may contain offensive material.
DiffCLIP: Differential Attention Meets CLIP
Authors: name: Hasan Abed Al Kader Hammoud, name: Bernard Ghanem
Published: 2025-03-09 14:04:09

We propose DiffCLIP, a novel vision-language model that extends the
differential attention mechanism to CLIP architectures. Differential attention
was originally developed for large language models to amplify relevant context
while canceling out noisy information. In this work, we integrate this
mechanism into CLIP's dual encoder (image and text) framework. With minimal
additional parameters, DiffCLIP achieves superior performance on image-text
understanding tasks. Across zero-shot classification, retrieval, and robustness
benchmarks, DiffCLIP consistently outperforms baseline CLIP models. Notably,
these gains come with negligible computational overhead, demonstrating that
differential attention can significantly enhance multi-modal representations
without sacrificing efficiency. Code can be found at
https://github.com/hammoudhasan/DiffCLIP.
HumanMM: Global Human Motion Recovery from Multi-shot Videos
Authors: name: Yuhong Zhang, name: Guanlin Wu, name: Ling-Hao Chen, name: Zhuokai Zhao, name: Jing Lin, name: Xiaoke Jiang, name: Jiamin Wu, name: Zhuoheng Li, name: Hao Frank Yang, name: Haoqian Wang, name: Lei Zhang
Published: 2025-03-10 17:57:03

In this paper, we present a novel framework designed to reconstruct
long-sequence 3D human motion in the world coordinates from in-the-wild videos
with multiple shot transitions. Such long-sequence in-the-wild motions are
highly valuable to applications such as motion generation and motion
understanding, but are of great challenge to be recovered due to abrupt shot
transitions, partial occlusions, and dynamic backgrounds presented in such
videos. Existing methods primarily focus on single-shot videos, where
continuity is maintained within a single camera view, or simplify multi-shot
alignment in camera space only. In this work, we tackle the challenges by
integrating an enhanced camera pose estimation with Human Motion Recovery (HMR)
by incorporating a shot transition detector and a robust alignment module for
accurate pose and orientation continuity across shots. By leveraging a custom
motion integrator, we effectively mitigate the problem of foot sliding and
ensure temporal consistency in human pose. Extensive evaluations on our created
multi-shot dataset from public 3D human datasets demonstrate the robustness of
our method in reconstructing realistic human motion in world coordinates.
Novel Object 6D Pose Estimation with a Single Reference View
Authors: name: Jian Liu, name: Wei Sun, name: Kai Zeng, name: Jin Zheng, name: Hui Yang, name: Lin Wang, name: Hossein Rahmani, name: Ajmal Mian
Published: 2025-03-07 17:00:41

Existing novel object 6D pose estimation methods typically rely on CAD models
or dense reference views, which are both difficult to acquire. Using only a
single reference view is more scalable, but challenging due to large pose
discrepancies and limited geometric and spatial information. To address these
issues, we propose a Single-Reference-based novel object 6D (SinRef-6D) pose
estimation method. Our key idea is to iteratively establish point-wise
alignment in the camera coordinate system based on state space models (SSMs).
Specifically, iterative camera-space point-wise alignment can effectively
handle large pose discrepancies, while our proposed RGB and Points SSMs can
capture long-range dependencies and spatial information from a single view,
offering linear complexity and superior spatial modeling capability. Once
pre-trained on synthetic data, SinRef-6D can estimate the 6D pose of a novel
object using only a single reference view, without requiring retraining or a
CAD model. Extensive experiments on six popular datasets and real-world robotic
scenes demonstrate that we achieve on-par performance with CAD-based and dense
reference view-based methods, despite operating in the more challenging single
reference setting. Code will be released at
https://github.com/CNJianLiu/SinRef-6D.
Beyond RAG: Task-Aware KV Cache Compression for Comprehensive Knowledge
  Reasoning
Authors: name: Giulio Corallo, name: Orion Weller, name: Fabio Petroni, name: Paolo Papotti
Published: 2025-03-06 21:07:41

Incorporating external knowledge in large language models (LLMs) enhances
their utility across diverse applications, but existing methods have
trade-offs. Retrieval-Augmented Generation (RAG) fetches evidence via
similarity search, but key information may fall outside top ranked results.
Long-context models can process multiple documents but are computationally
expensive and limited by context window size. Inspired by students condensing
study material for open-book exams, we propose task-aware key-value (KV) cache
compression, which compresses external knowledge in a zero- or few-shot setup.
This enables LLMs to reason efficiently over a compacted representation of all
relevant information. Experiments show our approach outperforms both RAG and
task-agnostic compression methods. On LongBench v2, it improves accuracy by up
to 7 absolute points over RAG with a 30x compression rate, while reducing
inference latency from 0.43s to 0.16s. A synthetic dataset highlights that RAG
performs well when sparse evidence suffices, whereas task-aware compression is
superior for broad knowledge tasks.
WISE: A World Knowledge-Informed Semantic Evaluation for Text-to-Image
  Generation
Authors: name: Yuwei Niu, name: Munan Ning, name: Mengren Zheng, name: Bin Lin, name: Peng Jin, name: Jiaqi Liao, name: Kunpeng Ning, name: Bin Zhu, name: Li Yuan
Published: 2025-03-10 12:47:53

Text-to-Image (T2I) models are capable of generating high-quality artistic
creations and visual content. However, existing research and evaluation
standards predominantly focus on image realism and shallow text-image
alignment, lacking a comprehensive assessment of complex semantic understanding
and world knowledge integration in text to image generation. To address this
challenge, we propose WISE, the first benchmark specifically
designed for World Knowledge-Informed Semantic
Evaluation. WISE moves beyond simple word-pixel mapping by
challenging models with 1000 meticulously crafted prompts across 25 sub-domains
in cultural common sense, spatio-temporal reasoning, and natural science. To
overcome the limitations of traditional CLIP metric, we introduce
WiScore, a novel quantitative metric for assessing knowledge-image
alignment. Through comprehensive testing of 20 models (10 dedicated T2I models
and 10 unified multimodal models) using 1,000 structured prompts spanning 25
subdomains, our findings reveal significant limitations in their ability to
effectively integrate and apply world knowledge during image generation,
highlighting critical pathways for enhancing knowledge incorporation and
application in next-generation T2I models. Code and data are available at
https://github.com/PKU-YuanGroup/WISE.
Zero-AVSR: Zero-Shot Audio-Visual Speech Recognition with LLMs by
  Learning Language-Agnostic Speech Representations
Authors: name: Jeong Hun Yeo, name: Minsu Kim, name: Chae Won Kim, name: Stavros Petridis, name: Yong Man Ro
Published: 2025-03-08 16:40:13

We explore a novel zero-shot Audio-Visual Speech Recognition (AVSR)
framework, dubbed Zero-AVSR, which enables speech recognition in target
languages without requiring any audio-visual speech data in those languages.
Specifically, we introduce the Audio-Visual Speech Romanizer (AV-Romanizer),
which learns language-agnostic speech representations by predicting Roman text.
Then, by leveraging the strong multilingual modeling capabilities of Large
Language Models (LLMs), we propose converting the predicted Roman text into
language-specific graphemes, forming the proposed Cascaded Zero-AVSR. Taking it
a step further, we explore a unified Zero-AVSR approach by directly integrating
the audio-visual speech representations encoded by the AV-Romanizer into the
LLM. This is achieved through finetuning the adapter and the LLM using our
proposed multi-task learning scheme. To capture the wide spectrum of phonetic
and linguistic diversity, we also introduce a Multilingual Audio-Visual
Romanized Corpus (MARC) consisting of 2,916 hours of audio-visual speech data
across 82 languages, along with transcriptions in both language-specific
graphemes and Roman text. Extensive analysis and experiments confirm that the
proposed Zero-AVSR framework has the potential to expand language support
beyond the languages seen during the training of the AV-Romanizer.
Efficient Distillation of Classifier-Free Guidance using Adapters
Authors: name: Cristian Perez Jensen, name: Seyedmorteza Sadat
Published: 2025-03-10 12:55:08

While classifier-free guidance (CFG) is essential for conditional diffusion
models, it doubles the number of neural function evaluations (NFEs) per
inference step. To mitigate this inefficiency, we introduce adapter guidance
distillation (AGD), a novel approach that simulates CFG in a single forward
pass. AGD leverages lightweight adapters to approximate CFG, effectively
doubling the sampling speed while maintaining or even improving sample quality.
Unlike prior guidance distillation methods that tune the entire model, AGD
keeps the base model frozen and only trains minimal additional parameters
(sim2%) to significantly reduce the resource requirement of the distillation
phase. Additionally, this approach preserves the original model weights and
enables the adapters to be seamlessly combined with other checkpoints derived
from the same base model. We also address a key mismatch between training and
inference in existing guidance distillation methods by training on CFG-guided
trajectories instead of standard diffusion trajectories. Through extensive
experiments, we show that AGD achieves comparable or superior FID to CFG across
multiple architectures with only half the NFEs. Notably, our method enables the
distillation of large models (sim2.6B parameters) on a single consumer GPU
with 24 GB of VRAM, making it more accessible than previous approaches that
require multiple high-end GPUs. We will publicly release the implementation of
our method.
This Is Your Doge, If It Please You: Exploring Deception and Robustness
  in Mixture of LLMs
Authors: name: Lorenz Wolf, name: Sangwoong Yoon, name: Ilija Bogunovic
Published: 2025-03-07 14:46:39

Mixture of large language model (LLMs) Agents (MoA) architectures achieve
state-of-the-art performance on prominent benchmarks like AlpacaEval 2.0 by
leveraging the collaboration of multiple LLMs at inference time. Despite these
successes, an evaluation of the safety and reliability of MoA is missing. We
present the first comprehensive study of MoA's robustness against deceptive LLM
agents that deliberately provide misleading responses. We examine factors like
the propagation of deceptive information, model size, and information
availability, and uncover critical vulnerabilities. On AlpacaEval 2.0, the
popular LLaMA 3.1-70B model achieves a length-controlled Win Rate (LC WR) of
49.2% when coupled with 3-layer MoA (6 LLM agents). However, we demonstrate
that introducing only a single carefully-instructed deceptive agent
into the MoA can reduce performance to 37.9%, effectively nullifying all MoA
gains. On QuALITY, a multiple-choice comprehension task, the impact is also
severe, with accuracy plummeting by a staggering 48.5%. Inspired in part by the
historical Doge of Venice voting process, designed to minimize influence and
deception, we propose a range of unsupervised defense mechanisms that recover
most of the lost performance.