Orionnnn/hub-stats huggingface dataset:
LogQuant: Log-Distributed 2-Bit Quantization of KV Cache with Superior
  Accuracy Preservation
Authors: name: Han Chen, name: Zicong Jiang, name: Zining Zhang, name: Bingsheng He, name: Pingyi Luo, name: Mian Lu, name: Yuqiang Chen
Published: 2025-03-25 16:24:45

We introduce LogQuant, a groundbreaking 2-bit quantization technique for KV
Cache in large language model (LLM) inference, delivering substantial memory
savings while preserving superior performance. Previous methods either assume
that later tokens are more important or attempt to predict important tokens
based on earlier attention patterns. Both approaches, however, can result in
performance bottlenecks or frequent mispredictions.
  LogQuant takes a different approach. By applying a log-based filtering
mechanism, it selectively compresses the KV Cache across the entire context,
achieving better performance with the same or even reduced memory footprint
compared to existing methods. In benchmark tests, it enhances throughput by 25%
and boosts batch size by 60% without increasing memory consumption. For
challenging tasks such as Math and Code Completion, LogQuant improves accuracy
by 40% to 200% at the same compression ratio, outperforming comparable
techniques.LogQuant integrates effortlessly with popular inference frameworks
like Python's transformers library. Implementation can be available in
https://github.com/Concyclics/LogQuantKV.
Sparse Logit Sampling: Accelerating Knowledge Distillation in LLMs
Authors: name: Anshumann, name: Mohd Abbas Zaidi, name: Akhil Kedia, name: Jinwoo Ahn, name: Taehwak Kwon, name: Kangwook Lee, name: Haejun Lee, name: Joohyung Lee
Published: 2025-03-21 05:58:18

Knowledge distillation can be a cost-effective technique to distill knowledge
in Large Language Models, if the teacher output logits can be pre-computed and
cached. However, successfully applying this to pre-training remains largely
unexplored. In this work, we prove that naive approaches for sparse knowledge
distillation such as caching Top-K probabilities, while intuitive, provide
biased estimates of teacher probability distribution to the student, resulting
in suboptimal performance and calibration. We propose an
importance-sampling-based method `Random Sampling Knowledge Distillation',
which provides unbiased estimates, preserves the gradient in expectation, and
requires storing significantly sparser logits. Our method enables faster
training of student models with marginal overhead (<10%) compared to
cross-entropy based training, while maintaining competitive performance
compared to full distillation, across a range of model sizes from 300M to 3B.
Attention IoU: Examining Biases in CelebA using Attention Maps
Authors: name: Aaron Serianni, name: Tyler Zhu, name: Olga Russakovsky, name: Vikram V. Ramaswamy
Published: 2025-03-25 17:11:39

Computer vision models have been shown to exhibit and amplify biases across a
wide array of datasets and tasks. Existing methods for quantifying bias in
classification models primarily focus on dataset distribution and model
performance on subgroups, overlooking the internal workings of a model. We
introduce the Attention-IoU (Attention Intersection over Union) metric and
related scores, which use attention maps to reveal biases within a model's
internal representations and identify image features potentially causing the
biases. First, we validate Attention-IoU on the synthetic Waterbirds dataset,
showing that the metric accurately measures model bias. We then analyze the
CelebA dataset, finding that Attention-IoU uncovers correlations beyond
accuracy disparities. Through an investigation of individual attributes through
the protected attribute of Male, we examine the distinct ways biases are
represented in CelebA. Lastly, by subsampling the training set to change
attribute correlations, we demonstrate that Attention-IoU reveals potential
confounding variables not present in dataset labels.
MCTS-RAG: Enhancing Retrieval-Augmented Generation with Monte Carlo Tree
  Search
Authors: name: Yunhai Hu, name: Yilun Zhao, name: Chen Zhao, name: Arman Cohan
Published: 2025-03-26 17:46:08

We introduce MCTS-RAG, a novel approach that enhances the reasoning
capabilities of small language models on knowledge-intensive tasks by
leveraging retrieval-augmented generation (RAG) to provide relevant context and
Monte Carlo Tree Search (MCTS) to refine reasoning paths. MCTS-RAG dynamically
integrates retrieval and reasoning through an iterative decision-making
process. Unlike standard RAG methods, which typically retrieve information
independently from reasoning and thus integrate knowledge suboptimally, or
conventional MCTS reasoning, which depends solely on internal model knowledge
without external facts, MCTS-RAG combines structured reasoning with adaptive
retrieval. This integrated approach enhances decision-making, reduces
hallucinations, and ensures improved factual accuracy and response consistency.
The experimental results on multiple reasoning and knowledge-intensive datasets
datasets (i.e., ComplexWebQA, GPQA, and FoolMeTwice) show that our method
enables small-scale LMs to achieve performance comparable to frontier LLMs like
GPT-4o by effectively scaling inference-time compute, setting a new standard
for reasoning in small-scale models.
Dita: Scaling Diffusion Transformer for Generalist
  Vision-Language-Action Policy
Authors: name: Zhi Hou, name: Tianyi Zhang, name: Yuwen Xiong, name: Haonan Duan, name: Hengjun Pu, name: Ronglei Tong, name: Chengyang Zhao, name: Xizhou Zhu, name: Yu Qiao, name: Jifeng Dai, name: Yuntao Chen
Published: 2025-03-25 15:19:56

While recent vision-language-action models trained on diverse robot datasets
exhibit promising generalization capabilities with limited in-domain data,
their reliance on compact action heads to predict discretized or continuous
actions constrains adaptability to heterogeneous action spaces. We present
Dita, a scalable framework that leverages Transformer architectures to directly
denoise continuous action sequences through a unified multimodal diffusion
process. Departing from prior methods that condition denoising on fused
embeddings via shallow networks, Dita employs in-context conditioning --
enabling fine-grained alignment between denoised actions and raw visual tokens
from historical observations. This design explicitly models action deltas and
environmental nuances. By scaling the diffusion action denoiser alongside the
Transformer's scalability, Dita effectively integrates cross-embodiment
datasets across diverse camera perspectives, observation scenes, tasks, and
action spaces. Such synergy enhances robustness against various variances and
facilitates the successful execution of long-horizon tasks. Evaluations across
extensive benchmarks demonstrate state-of-the-art or comparative performance in
simulation. Notably, Dita achieves robust real-world adaptation to
environmental variances and complex long-horizon tasks through 10-shot
finetuning, using only third-person camera inputs. The architecture establishes
a versatile, lightweight and open-source baseline for generalist robot policy
learning. Project Page: https://robodita.github.io.
GenHancer: Imperfect Generative Models are Secretly Strong
  Vision-Centric Enhancers
Authors: name: Shijie Ma, name: Yuying Ge, name: Teng Wang, name: Yuxin Guo, name: Yixiao Ge, name: Ying Shan
Published: 2025-03-25 09:15:34

The synergy between generative and discriminative models receives growing
attention. While discriminative Contrastive Language-Image Pre-Training (CLIP)
excels in high-level semantics, it struggles with perceiving fine-grained
visual details. Generally, to enhance representations, generative models take
CLIP's visual features as conditions for reconstruction. However, the
underlying principle remains underexplored. In this work, we empirically found
that visually perfect generations are not always optimal for representation
enhancement. The essence lies in effectively extracting fine-grained knowledge
from generative models while mitigating irrelevant information. To explore
critical factors, we delve into three aspects: (1) Conditioning mechanisms: We
found that even a small number of local tokens can drastically reduce the
difficulty of reconstruction, leading to collapsed training. We thus conclude
that utilizing only global visual tokens as conditions is the most effective
strategy. (2) Denoising configurations: We observed that end-to-end training
introduces extraneous information. To address this, we propose a two-stage
training strategy to prioritize learning useful visual knowledge. Additionally,
we demonstrate that lightweight denoisers can yield remarkable improvements.
(3) Generation paradigms: We explore both continuous and discrete denoisers
with desirable outcomes, validating the versatility of our method. Through our
in-depth explorations, we have finally arrived at an effective method, namely
GenHancer, which consistently outperforms prior arts on the MMVP-VLM benchmark,
e.g., 6.0% on OpenAICLIP. The enhanced CLIP can be further plugged into
multimodal large language models for better vision-centric performance. All the
models and codes are made publicly available.
Beyond Words: Advancing Long-Text Image Generation via Multimodal
  Autoregressive Models
Authors: name: Alex Jinpeng Wang, name: Linjie Li, name: Zhengyuan Yang, name: Lijuan Wang, name: Min Li
Published: 2025-03-26 03:44:25

Recent advancements in autoregressive and diffusion models have led to strong
performance in image generation with short scene text words. However,
generating coherent, long-form text in images, such as paragraphs in slides or
documents, remains a major challenge for current generative models. We present
the first work specifically focused on long text image generation, addressing a
critical gap in existing text-to-image systems that typically handle only brief
phrases or single sentences. Through comprehensive analysis of state-of-the-art
autoregressive generation models, we identify the image tokenizer as a critical
bottleneck in text generating quality. To address this, we introduce a novel
text-focused, binary tokenizer optimized for capturing detailed scene text
features. Leveraging our tokenizer, we develop \ModelName, a multimodal
autoregressive model that excels in generating high-quality long-text images
with unprecedented fidelity. Our model offers robust controllability, enabling
customization of text properties such as font style, size, color, and
alignment. Extensive experiments demonstrate that \ModelName~significantly
outperforms SD3.5 Large~sd3 and GPT4o~gpt4o with DALL-E
3~dalle3 in generating long text accurately, consistently, and flexibly.
Beyond its technical achievements, \ModelName~opens up exciting opportunities
for innovative applications like interleaved document and PowerPoint
generation, establishing a new frontier in long-text image generating.
BizGen: Advancing Article-level Visual Text Rendering for Infographics
  Generation
Authors: name: Yuyang Peng, name: Shishi Xiao, name: Keming Wu, name: Qisheng Liao, name: Bohan Chen, name: Kevin Lin, name: Danqing Huang, name: Ji Li, name: Yuhui Yuan
Published: 2025-03-26 16:04:57

Recently, state-of-the-art text-to-image generation models, such as Flux and
Ideogram 2.0, have made significant progress in sentence-level visual text
rendering. In this paper, we focus on the more challenging scenarios of
article-level visual text rendering and address a novel task of generating
high-quality business content, including infographics and slides, based on user
provided article-level descriptive prompts and ultra-dense layouts. The
fundamental challenges are twofold: significantly longer context lengths and
the scarcity of high-quality business content data.
  In contrast to most previous works that focus on a limited number of
sub-regions and sentence-level prompts, ensuring precise adherence to
ultra-dense layouts with tens or even hundreds of sub-regions in business
content is far more challenging. We make two key technical contributions: (i)
the construction of scalable, high-quality business content dataset, i.e.,
Infographics-650K, equipped with ultra-dense layouts and prompts by
implementing a layer-wise retrieval-augmented infographic generation scheme;
and (ii) a layout-guided cross attention scheme, which injects tens of
region-wise prompts into a set of cropped region latent space according to the
ultra-dense layouts, and refine each sub-regions flexibly during inference
using a layout conditional CFG.
  We demonstrate the strong results of our system compared to previous SOTA
systems such as Flux and SD3 on our BizEval prompt set. Additionally, we
conduct thorough ablation experiments to verify the effectiveness of each
component. We hope our constructed Infographics-650K and BizEval can encourage
the broader community to advance the progress of business content generation.
Wan: Open and Advanced Large-Scale Video Generative Models
Authors: name: WanTeam, name: Ang Wang, name: Baole Ai, name: Bin Wen, name: Chaojie Mao, name: Chen-Wei Xie, name: Di Chen, name: Feiwu Yu, name: Haiming Zhao, name: Jianxiao Yang, name: Jianyuan Zeng, name: Jiayu Wang, name: Jingfeng Zhang, name: Jingren Zhou, name: Jinkai Wang, name: Jixuan Chen, name: Kai Zhu, name: Kang Zhao, name: Keyu Yan, name: Lianghua Huang, name: Mengyang Feng, name: Ningyi Zhang, name: Pandeng Li, name: Pingyu Wu, name: Ruihang Chu, name: Ruili Feng, name: Shiwei Zhang, name: Siyang Sun, name: Tao Fang, name: Tianxing Wang, name: Tianyi Gui, name: Tingyu Weng, name: Tong Shen, name: Wei Lin, name: Wei Wang, name: Wei Wang, name: Wenmeng Zhou, name: Wente Wang, name: Wenting Shen, name: Wenyuan Yu, name: Xianzhong Shi, name: Xiaoming Huang, name: Xin Xu, name: Yan Kou, name: Yangyu Lv, name: Yifei Li, name: Yijing Liu, name: Yiming Wang, name: Yingya Zhang, name: Yitong Huang, name: Yong Li, name: You Wu, name: Yu Liu, name: Yulin Pan, name: Yun Zheng, name: Yuntao Hong, name: Yupeng Shi, name: Yutong Feng, name: Zeyinzi Jiang, name: Zhen Han, name: Zhi-Fan Wu, name: Ziyu Liu
Published: 2025-03-26 08:25:43

This report presents Wan, a comprehensive and open suite of video foundation
models designed to push the boundaries of video generation. Built upon the
mainstream diffusion transformer paradigm, Wan achieves significant
advancements in generative capabilities through a series of innovations,
including our novel VAE, scalable pre-training strategies, large-scale data
curation, and automated evaluation metrics. These contributions collectively
enhance the model's performance and versatility. Specifically, Wan is
characterized by four key features: Leading Performance: The 14B model of Wan,
trained on a vast dataset comprising billions of images and videos,
demonstrates the scaling laws of video generation with respect to both data and
model size. It consistently outperforms the existing open-source models as well
as state-of-the-art commercial solutions across multiple internal and external
benchmarks, demonstrating a clear and significant performance superiority.
Comprehensiveness: Wan offers two capable models, i.e., 1.3B and 14B
parameters, for efficiency and effectiveness respectively. It also covers
multiple downstream applications, including image-to-video, instruction-guided
video editing, and personal video generation, encompassing up to eight tasks.
Consumer-Grade Efficiency: The 1.3B model demonstrates exceptional resource
efficiency, requiring only 8.19 GB VRAM, making it compatible with a wide range
of consumer-grade GPUs. Openness: We open-source the entire series of Wan,
including source code and all models, with the goal of fostering the growth of
the video generation community. This openness seeks to significantly expand the
creative possibilities of video production in the industry and provide academia
with high-quality video foundation models. All the code and models are
available at https://github.com/Wan-Video/Wan2.1.
Self-Supervised Learning of Motion Concepts by Optimizing
  Counterfactuals
Authors: name: Stefan Stojanov, name: David Wendt, name: Seungwoo Kim, name: Rahul Venkatesh, name: Kevin Feigelis, name: Jiajun Wu, name: Daniel LK Yamins
Published: 2025-03-25 17:58:52

Estimating motion in videos is an essential computer vision problem with many
downstream applications, including controllable video generation and robotics.
Current solutions are primarily trained using synthetic data or require tuning
of situation-specific heuristics, which inherently limits these models'
capabilities in real-world contexts. Despite recent developments in large-scale
self-supervised learning from videos, leveraging such representations for
motion estimation remains relatively underexplored. In this work, we develop
Opt-CWM, a self-supervised technique for flow and occlusion estimation from a
pre-trained next-frame prediction model. Opt-CWM works by learning to optimize
counterfactual probes that extract motion information from a base video model,
avoiding the need for fixed heuristics while training on unrestricted video
inputs. We achieve state-of-the-art performance for motion estimation on
real-world videos while requiring no labeled data.
AccVideo: Accelerating Video Diffusion Model with Synthetic Dataset
Authors: name: Haiyu Zhang, name: Xinyuan Chen, name: Yaohui Wang, name: Xihui Liu, name: Yunhong Wang, name: Yu Qiao
Published: 2025-03-25 08:52:07

Diffusion models have achieved remarkable progress in the field of video
generation. However, their iterative denoising nature requires a large number
of inference steps to generate a video, which is slow and computationally
expensive. In this paper, we begin with a detailed analysis of the challenges
present in existing diffusion distillation methods and propose a novel
efficient method, namely AccVideo, to reduce the inference steps for
accelerating video diffusion models with synthetic dataset. We leverage the
pretrained video diffusion model to generate multiple valid denoising
trajectories as our synthetic dataset, which eliminates the use of useless data
points during distillation. Based on the synthetic dataset, we design a
trajectory-based few-step guidance that utilizes key data points from the
denoising trajectories to learn the noise-to-video mapping, enabling video
generation in fewer steps. Furthermore, since the synthetic dataset captures
the data distribution at each diffusion timestep, we introduce an adversarial
training strategy to align the output distribution of the student model with
that of our synthetic dataset, thereby enhancing the video quality. Extensive
experiments demonstrate that our model achieves 8.5x improvements in generation
speed compared to the teacher model while maintaining comparable performance.
Compared to previous accelerating methods, our approach is capable of
generating videos with higher quality and resolution, i.e., 5-seconds,
720x1280, 24fps.
ADS-Edit: A Multimodal Knowledge Editing Dataset for Autonomous Driving
  Systems
Authors: name: Chenxi Wang, name: Jizhan Fang, name: Xiang Chen, name: Bozhong Tian, name: Ziwen Xu, name: Huajun Chen, name: Ningyu Zhang
Published: 2025-03-26 17:45:29

Recent advancements in Large Multimodal Models (LMMs) have shown promise in
Autonomous Driving Systems (ADS). However, their direct application to ADS is
hindered by challenges such as misunderstanding of traffic knowledge, complex
road conditions, and diverse states of vehicle. To address these challenges, we
propose the use of Knowledge Editing, which enables targeted modifications to a
model's behavior without the need for full retraining. Meanwhile, we introduce
ADS-Edit, a multimodal knowledge editing dataset specifically designed for ADS,
which includes various real-world scenarios, multiple data types, and
comprehensive evaluation metrics. We conduct comprehensive experiments and
derive several interesting conclusions. We hope that our work will contribute
to the further advancement of knowledge editing applications in the field of
autonomous driving. Code and data are available in
https://github.com/zjunlp/EasyEdit.
Open Deep Search: Democratizing Search with Open-source Reasoning Agents
Authors: name: Salaheddin Alzubi, name: Creston Brooks, name: Purva Chiniya, name: Edoardo Contente, name: Chiara von Gerlach, name: Lucas Irwin, name: Yihan Jiang, name: Arda Kaz, name: Windsor Nguyen, name: Sewoong Oh, name: Himanshu Tyagi, name: Pramod Viswanath
Published: 2025-03-26 03:51:32

We introduce Open Deep Search (ODS) to close the increasing gap between the
proprietary search AI solutions, such as Perplexity's Sonar Reasoning Pro and
OpenAI's GPT-4o Search Preview, and their open-source counterparts. The main
innovation introduced in ODS is to augment the reasoning capabilities of the
latest open-source LLMs with reasoning agents that can judiciously use web
search tools to answer queries. Concretely, ODS consists of two components that
work with a base LLM chosen by the user: Open Search Tool and Open Reasoning
Agent. Open Reasoning Agent interprets the given task and completes it by
orchestrating a sequence of actions that includes calling tools, one of which
is the Open Search Tool. Open Search Tool is a novel web search tool that
outperforms proprietary counterparts. Together with powerful open-source
reasoning LLMs, such as DeepSeek-R1, ODS nearly matches and sometimes surpasses
the existing state-of-the-art baselines on two benchmarks: SimpleQA and FRAMES.
For example, on the FRAMES evaluation benchmark, ODS improves the best existing
baseline of the recently released GPT-4o Search Preview by 9.7% in accuracy.
ODS is a general framework for seamlessly augmenting any LLMs -- for example,
DeepSeek-R1 that achieves 82.4% on SimpleQA and 30.1% on FRAMES -- with search
and reasoning capabilities to achieve state-of-the-art performance: 88.3% on
SimpleQA and 75.3% on FRAMES.
DINeMo: Learning Neural Mesh Models with no 3D Annotations
Authors: name: Weijie Guo, name: Guofeng Zhang, name: Wufei Ma, name: Alan Yuille
Published: 2025-03-26 04:23:53

Category-level 3D/6D pose estimation is a crucial step towards comprehensive
3D scene understanding, which would enable a broad range of applications in
robotics and embodied AI. Recent works explored neural mesh models that
approach a range of 2D and 3D tasks from an analysis-by-synthesis perspective.
Despite the largely enhanced robustness to partial occlusion and domain shifts,
these methods depended heavily on 3D annotations for part-contrastive learning,
which confines them to a narrow set of categories and hinders efficient
scaling. In this work, we present DINeMo, a novel neural mesh model that is
trained with no 3D annotations by leveraging pseudo-correspondence obtained
from large visual foundation models. We adopt a bidirectional
pseudo-correspondence generation method, which produce pseudo correspondence
utilize both local appearance features and global context information.
Experimental results on car datasets demonstrate that our DINeMo outperforms
previous zero- and few-shot 3D pose estimation by a wide margin, narrowing the
gap with fully-supervised methods by 67.3%. Our DINeMo also scales effectively
and efficiently when incorporating more unlabeled images during training, which
demonstrate the advantages over supervised learning methods that rely on 3D
annotations. Our project page is available at
https://analysis-by-synthesis.github.io/DINeMo/.