Orionnnn/hub-stats huggingface dataset:
Inside-Out: Hidden Factual Knowledge in LLMs
Authors: name: Zorik Gekhman, name: Eyal Ben David, name: Hadas Orgad, name: Eran Ofek, name: Yonatan Belinkov, name: Idan Szpector, name: Jonathan Herzig, name: Roi Reichart
Published: 2025-03-19 15:21:48

This work presents a framework for assessing whether large language models
(LLMs) encode more factual knowledge in their parameters than what they express
in their outputs. While a few studies hint at this possibility, none has
clearly defined or demonstrated this phenomenon. We first propose a formal
definition of knowledge, quantifying it for a given question as the fraction of
correct-incorrect answer pairs where the correct one is ranked higher. This
gives rise to external and internal knowledge, depending on the information
used to score individual answer candidates: either the model's observable
token-level probabilities or its intermediate computations. Hidden knowledge
arises when internal knowledge exceeds external knowledge. We then present a
case study, applying this framework to three popular open-weights LLMs in a
closed-book QA setup. Our results indicate that: (1) LLMs consistently encode
more factual knowledge internally than what they express externally, with an
average gap of 40%. (2) Surprisingly, some knowledge is so deeply hidden that a
model can internally know an answer perfectly, yet fail to generate it even
once, despite large-scale repeated sampling of 1,000 answers. This reveals
fundamental limitations in the generation capabilities of LLMs, which (3) puts
a practical constraint on scaling test-time compute via repeated answer
sampling in closed-book QA: significant performance improvements remain
inaccessible because some answers are practically never sampled, yet if they
were, we would be guaranteed to rank them first.
Sonata: Self-Supervised Learning of Reliable Point Representations
Authors: name: Xiaoyang Wu, name: Daniel DeTone, name: Duncan Frost, name: Tianwei Shen, name: Chris Xie, name: Nan Yang, name: Jakob Engel, name: Richard Newcombe, name: Hengshuang Zhao, name: Julian Straub
Published: 2025-03-20 17:59:59

In this paper, we question whether we have a reliable self-supervised point
cloud model that can be used for diverse 3D tasks via simple linear probing,
even with limited data and minimal computation. We find that existing 3D
self-supervised learning approaches fall short when evaluated on representation
quality through linear probing. We hypothesize that this is due to what we term
the "geometric shortcut", which causes representations to collapse to low-level
spatial features. This challenge is unique to 3D and arises from the sparse
nature of point cloud data. We address it through two key strategies: obscuring
spatial information and enhancing the reliance on input features, ultimately
composing a Sonata of 140k point clouds through self-distillation. Sonata is
simple and intuitive, yet its learned representations are strong and reliable:
zero-shot visualizations demonstrate semantic grouping, alongside strong
spatial reasoning through nearest-neighbor relationships. Sonata demonstrates
exceptional parameter and data efficiency, tripling linear probing accuracy
(from 21.8% to 72.5%) on ScanNet and nearly doubling performance with only 1%
of the data compared to previous approaches. Full fine-tuning further advances
SOTA across both 3D indoor and outdoor perception tasks.
AIMI: Leveraging Future Knowledge and Personalization in Sparse Event
  Forecasting for Treatment Adherence
Authors: name: Abdullah Mamun, name: Diane J. Cook, name: Hassan Ghasemzadeh
Published: 2025-03-20 12:32:35

Adherence to prescribed treatments is crucial for individuals with chronic
conditions to avoid costly or adverse health outcomes. For certain patient
groups, intensive lifestyle interventions are vital for enhancing medication
adherence. Accurate forecasting of treatment adherence can open pathways to
developing an on-demand intervention tool, enabling timely and personalized
support. With the increasing popularity of smartphones and wearables, it is now
easier than ever to develop and deploy smart activity monitoring systems.
However, effective forecasting systems for treatment adherence based on
wearable sensors are still not widely available. We close this gap by proposing
Adherence Forecasting and Intervention with Machine Intelligence (AIMI). AIMI
is a knowledge-guided adherence forecasting system that leverages smartphone
sensors and previous medication history to estimate the likelihood of
forgetting to take a prescribed medication. A user study was conducted with 27
participants who took daily medications to manage their cardiovascular
diseases. We designed and developed CNN and LSTM-based forecasting models with
various combinations of input features and found that LSTM models can forecast
medication adherence with an accuracy of 0.932 and an F-1 score of 0.936.
Moreover, through a series of ablation studies involving convolutional and
recurrent neural network architectures, we demonstrate that leveraging known
knowledge about future and personalized training enhances the accuracy of
medication adherence forecasting. Code available:
https://github.com/ab9mamun/AIMI.
BigO(Bench) -- Can LLMs Generate Code with Controlled Time and Space
  Complexity?
Authors: name: Pierre Chambon, name: Baptiste Roziere, name: Benoit Sagot, name: Gabriel Synnaeve
Published: 2025-03-19 14:19:57

We introduce BigO(Bench), a novel coding benchmark designed to evaluate the
capabilities of generative language models in understanding and generating code
with specified time and space complexities. This benchmark addresses the gap in
current evaluations that often overlook the ability of models to comprehend and
produce code constrained by computational complexity. BigO(Bench) includes
tooling to infer the algorithmic complexity of any Python function from
profiling measurements, including human- or LLM-generated solutions.
BigO(Bench) also includes of set of 3,105 coding problems and 1,190,250
solutions from Code Contests annotated with inferred (synthetic) time and space
complexity labels from the complexity framework, as well as corresponding
runtime and memory footprint values for a large set of input sizes. We present
results from evaluating multiple state-of-the-art language models on this
benchmark, highlighting their strengths and weaknesses in handling complexity
requirements. In particular, token-space reasoning models are unrivaled in code
generation but not in complexity understanding, hinting that they may not
generalize well to tasks for which no reward was given at training time.
Tokenize Image as a Set
Authors: name: Zigang Geng, name: Mengde Xu, name: Han Hu, name: Shuyang Gu
Published: 2025-03-20 17:59:51

This paper proposes a fundamentally new paradigm for image generation through
set-based tokenization and distribution modeling. Unlike conventional methods
that serialize images into fixed-position latent codes with a uniform
compression ratio, we introduce an unordered token set representation to
dynamically allocate coding capacity based on regional semantic complexity.
This TokenSet enhances global context aggregation and improves robustness
against local perturbations. To address the critical challenge of modeling
discrete sets, we devise a dual transformation mechanism that bijectively
converts sets into fixed-length integer sequences with summation constraints.
Further, we propose Fixed-Sum Discrete Diffusion--the first framework to
simultaneously handle discrete values, fixed sequence length, and summation
invariance--enabling effective set distribution modeling. Experiments
demonstrate our method's superiority in semantic-aware representation and
generation quality. Our innovations, spanning novel representation and modeling
strategies, advance visual generation beyond traditional sequential token
paradigms. Our code and models are publicly available at
https://github.com/Gengzigang/TokenSet.
Scale-wise Distillation of Diffusion Models
Authors: name: Nikita Starodubcev, name: Denis Kuznedelev, name: Artem Babenko, name: Dmitry Baranchuk
Published: 2025-03-20 17:54:02

We present SwD, a scale-wise distillation framework for diffusion models
(DMs), which effectively employs next-scale prediction ideas for
diffusion-based few-step generators. In more detail, SwD is inspired by the
recent insights relating diffusion processes to the implicit spectral
autoregression. We suppose that DMs can initiate generation at lower data
resolutions and gradually upscale the samples at each denoising step without
loss in performance while significantly reducing computational costs. SwD
naturally integrates this idea into existing diffusion distillation methods
based on distribution matching. Also, we enrich the family of distribution
matching approaches by introducing a novel patch loss enforcing finer-grained
similarity to the target distribution. When applied to state-of-the-art
text-to-image diffusion models, SwD approaches the inference times of two full
resolution steps and significantly outperforms the counterparts under the same
computation budget, as evidenced by automated metrics and human preference
studies.
One-Step Residual Shifting Diffusion for Image Super-Resolution via
  Distillation
Authors: name: Daniil Selikhanovych, name: David Li, name: Aleksei Leonov, name: Nikita Gushchin, name: Sergei Kushneriuk, name: Alexander Filippov, name: Evgeny Burnaev, name: Iaroslav Koshelev, name: Alexander Korotin
Published: 2025-03-17 16:44:08

Diffusion models for super-resolution (SR) produce high-quality visual
results but require expensive computational costs. Despite the development of
several methods to accelerate diffusion-based SR models, some (e.g., SinSR)
fail to produce realistic perceptual details, while others (e.g., OSEDiff) may
hallucinate non-existent structures. To overcome these issues, we present RSD,
a new distillation method for ResShift, one of the top diffusion-based SR
models. Our method is based on training the student network to produce such
images that a new fake ResShift model trained on them will coincide with the
teacher model. RSD achieves single-step restoration and outperforms the teacher
by a large margin. We show that our distillation method can surpass the other
distillation-based method for ResShift - SinSR - making it on par with
state-of-the-art diffusion-based SR distillation methods. Compared to SR
methods based on pre-trained text-to-image models, RSD produces competitive
perceptual quality, provides images with better alignment to degraded input
images, and requires fewer parameters and GPU memory. We provide experimental
results on various real-world and synthetic datasets, including RealSR,
RealSet65, DRealSR, ImageNet, and DIV2K.
Plug-and-Play 1.x-Bit KV Cache Quantization for Video Large Language
  Models
Authors: name: Keda Tao, name: Haoxuan You, name: Yang Sui, name: Can Qin, name: Huan Wang
Published: 2025-03-20 15:52:43

Video large language models (VideoLLMs) have demonstrated the capability to
process longer video inputs and enable complex reasoning and analysis. However,
due to the thousands of visual tokens from the video frames, key-value (KV)
cache can significantly increase memory requirements, becoming a bottleneck for
inference speed and memory usage. KV cache quantization is a widely used
approach to address this problem. In this paper, we find that 2-bit KV
quantization of VideoLLMs can hardly hurt the model performance, while the
limit of KV cache quantization in even lower bits has not been investigated. To
bridge this gap, we introduce VidKV, a plug-and-play KV cache quantization
method to compress the KV cache to lower than 2 bits. Specifically, (1) for
key, we propose a mixed-precision quantization strategy in the channel
dimension, where we perform 2-bit quantization for anomalous channels and 1-bit
quantization combined with FFT for normal channels; (2) for value, we implement
1.58-bit quantization while selectively filtering semantically salient visual
tokens for targeted preservation, for a better trade-off between precision and
model performance. Importantly, our findings suggest that the value cache of
VideoLLMs should be quantized in a per-channel fashion instead of the per-token
fashion proposed by prior KV cache quantization works for LLMs. Empirically,
extensive results with LLaVA-OV-7B and Qwen2.5-VL-7B on six benchmarks show
that VidKV effectively compresses the KV cache to 1.5-bit and 1.58-bit
precision with almost no performance drop compared to the FP16 counterparts.
VideoRFSplat: Direct Scene-Level Text-to-3D Gaussian Splatting
  Generation with Flexible Pose and Multi-View Joint Modeling
Authors: name: Hyojun Go, name: Byeongjun Park, name: Hyelin Nam, name: Byung-Hoon Kim, name: Hyungjin Chung, name: Changick Kim
Published: 2025-03-20 05:26:09

We propose VideoRFSplat, a direct text-to-3D model leveraging a video
generation model to generate realistic 3D Gaussian Splatting (3DGS) for
unbounded real-world scenes. To generate diverse camera poses and unbounded
spatial extent of real-world scenes, while ensuring generalization to arbitrary
text prompts, previous methods fine-tune 2D generative models to jointly model
camera poses and multi-view images. However, these methods suffer from
instability when extending 2D generative models to joint modeling due to the
modality gap, which necessitates additional models to stabilize training and
inference. In this work, we propose an architecture and a sampling strategy to
jointly model multi-view images and camera poses when fine-tuning a video
generation model. Our core idea is a dual-stream architecture that attaches a
dedicated pose generation model alongside a pre-trained video generation model
via communication blocks, generating multi-view images and camera poses through
separate streams. This design reduces interference between the pose and image
modalities. Additionally, we propose an asynchronous sampling strategy that
denoises camera poses faster than multi-view images, allowing rapidly denoised
poses to condition multi-view generation, reducing mutual ambiguity and
enhancing cross-modal consistency. Trained on multiple large-scale real-world
datasets (RealEstate10K, MVImgNet, DL3DV-10K, ACID), VideoRFSplat outperforms
existing text-to-3D direct generation methods that heavily depend on post-hoc
refinement via score distillation sampling, achieving superior results without
such refinement.
Make Your Training Flexible: Towards Deployment-Efficient Video Models
Authors: name: Chenting Wang, name: Kunchang Li, name: Tianxiang Jiang, name: Xiangyu Zeng, name: Yi Wang, name: Limin Wang
Published: 2025-03-18 13:15:58

Popular video training methods mainly operate on a fixed number of tokens
sampled from a predetermined spatiotemporal grid, resulting in sub-optimal
accuracy-computation trade-offs due to inherent video redundancy. They also
lack adaptability to varying computational budgets for downstream tasks,
hindering applications of the most competitive model in real-world scenes. We
thus propose a new test setting, Token Optimization, for maximized input
information across budgets, which optimizes the size-limited set of input
tokens through token selection from more suitably sampled videos. To this end,
we propose a novel augmentation tool termed Flux. By making the sampling grid
flexible and leveraging token selection, it is easily adopted in most popular
video training frameworks, boosting model robustness with nearly no additional
cost. We integrate Flux in large-scale video pre-training, and the resulting
FluxViT establishes new state-of-the-art results across extensive tasks at
standard costs. Notably, with 1/4 tokens only, it can still match the
performance of previous state-of-the-art models with Token Optimization,
yielding nearly 90\% savings. All models and data are available at
https://github.com/OpenGVLab/FluxViT.
NuiScene: Exploring Efficient Generation of Unbounded Outdoor Scenes
Authors: name: Han-Hung Lee, name: Qinghong Han, name: Angel X. Chang
Published: 2025-03-20 17:37:43

In this paper, we explore the task of generating expansive outdoor scenes,
ranging from castles to high-rises. Unlike indoor scene generation, which has
been a primary focus of prior work, outdoor scene generation presents unique
challenges, including wide variations in scene heights and the need for a
method capable of rapidly producing large landscapes. To address this, we
propose an efficient approach that encodes scene chunks as uniform vector sets,
offering better compression and performance than the spatially structured
latents used in prior methods. Furthermore, we train an explicit outpainting
model for unbounded generation, which improves coherence compared to prior
resampling-based inpainting schemes while also speeding up generation by
eliminating extra diffusion steps. To facilitate this task, we curate
NuiScene43, a small but high-quality set of scenes, preprocessed for joint
training. Notably, when trained on scenes of varying styles, our model can
blend different environments, such as rural houses and city skyscrapers, within
the same scene, highlighting the potential of our curation process to leverage
heterogeneous scenes for joint training.
Why Do Multi-Agent LLM Systems Fail?
Authors: name: Mert Cemri, name: Melissa Z. Pan, name: Shuyi Yang, name: Lakshya A. Agrawal, name: Bhavya Chopra, name: Rishabh Tiwari, name: Kurt Keutzer, name: Aditya Parameswaran, name: Dan Klein, name: Kannan Ramchandran, name: Matei Zaharia, name: Joseph E. Gonzalez, name: Ion Stoica
Published: 2025-03-17 19:04:38

Despite growing enthusiasm for Multi-Agent Systems (MAS), where multiple LLM
agents collaborate to accomplish tasks, their performance gains across popular
benchmarks remain minimal compared to single-agent frameworks. This gap
highlights the need to analyze the challenges hindering MAS effectiveness.
  In this paper, we present the first comprehensive study of MAS challenges. We
analyze five popular MAS frameworks across over 150 tasks, involving six expert
human annotators. We identify 14 unique failure modes and propose a
comprehensive taxonomy applicable to various MAS frameworks. This taxonomy
emerges iteratively from agreements among three expert annotators per study,
achieving a Cohen's Kappa score of 0.88. These fine-grained failure modes are
organized into 3 categories, (i) specification and system design failures, (ii)
inter-agent misalignment, and (iii) task verification and termination. To
support scalable evaluation, we integrate MASFT with LLM-as-a-Judge. We also
explore if identified failures could be easily prevented by proposing two
interventions: improved specification of agent roles and enhanced orchestration
strategies. Our findings reveal that identified failures require more complex
solutions, highlighting a clear roadmap for future research. We open-source our
dataset and LLM annotator.
Reinforcement Learning for Reasoning in Small LLMs: What Works and What
  Doesn't
Authors: name: Quy-Anh Dang, name: Chris Ngo
Published: 2025-03-20 15:13:23

Enhancing the reasoning capabilities of large language models (LLMs)
typically relies on massive computational resources and extensive datasets,
limiting accessibility for resource-constrained settings. Our study
investigates the potential of reinforcement learning (RL) to improve reasoning
in small LLMs, focusing on a 1.5-billion-parameter model,
DeepSeek-R1-Distill-Qwen-1.5B, under strict constraints: training on 4 NVIDIA
A40 GPUs (48 GB VRAM each) within 24 hours. Adapting the Group Relative Policy
Optimization (GRPO) algorithm and curating a compact, high-quality mathematical
reasoning dataset, we conducted three experiments to explore model behavior and
performance. Our results demonstrate rapid reasoning gains - e.g., AMC23
accuracy rising from 63% to 80% and AIME24 reaching 46.7%, surpassing
o1-preview - using only 7,000 samples and a $42 training cost, compared to
thousands of dollars for baseline models. However, challenges such as
optimization instability and length constraints emerged with prolonged
training. These findings highlight the efficacy of RL-based fine-tuning for
small LLMs, offering a cost-effective alternative to large-scale approaches. We
release our code and datasets as open-source resources, providing insights into
trade-offs and laying a foundation for scalable, reasoning-capable LLMs in
resource-limited environments. All are available at
https://github.com/knoveleng/open-rs.
DiffMoE: Dynamic Token Selection for Scalable Diffusion Transformers
Authors: name: Minglei Shi, name: Ziyang Yuan, name: Haotian Yang, name: Xintao Wang, name: Mingwu Zheng, name: Xin Tao, name: Wenliang Zhao, name: Wenzhao Zheng, name: Jie Zhou, name: Jiwen Lu, name: Pengfei Wan, name: Di Zhang, name: Kun Gai
Published: 2025-03-18 17:57:07

Diffusion models have demonstrated remarkable success in various image
generation tasks, but their performance is often limited by the uniform
processing of inputs across varying conditions and noise levels. To address
this limitation, we propose a novel approach that leverages the inherent
heterogeneity of the diffusion process. Our method, DiffMoE, introduces a
batch-level global token pool that enables experts to access global token
distributions during training, promoting specialized expert behavior. To
unleash the full potential of the diffusion process, DiffMoE incorporates a
capacity predictor that dynamically allocates computational resources based on
noise levels and sample complexity. Through comprehensive evaluation, DiffMoE
achieves state-of-the-art performance among diffusion models on ImageNet
benchmark, substantially outperforming both dense architectures with 3x
activated parameters and existing MoE approaches while maintaining 1x activated
parameters. The effectiveness of our approach extends beyond class-conditional
generation to more challenging tasks such as text-to-image generation,
demonstrating its broad applicability across different diffusion model
applications. Project Page: https://shiml20.github.io/DiffMoE/
Survey on Evaluation of LLM-based Agents
Authors: name: Asaf Yehudai, name: Lilach Eden, name: Alan Li, name: Guy Uziel, name: Yilun Zhao, name: Roy Bar-Haim, name: Arman Cohan, name: Michal Shmueli-Scheuer
Published: 2025-03-20 17:59:23

The emergence of LLM-based agents represents a paradigm shift in AI, enabling
autonomous systems to plan, reason, use tools, and maintain memory while
interacting with dynamic environments. This paper provides the first
comprehensive survey of evaluation methodologies for these increasingly capable
agents. We systematically analyze evaluation benchmarks and frameworks across
four critical dimensions: (1) fundamental agent capabilities, including
planning, tool use, self-reflection, and memory; (2) application-specific
benchmarks for web, software engineering, scientific, and conversational
agents; (3) benchmarks for generalist agents; and (4) frameworks for evaluating
agents. Our analysis reveals emerging trends, including a shift toward more
realistic, challenging evaluations with continuously updated benchmarks. We
also identify critical gaps that future research must address-particularly in
assessing cost-efficiency, safety, and robustness, and in developing
fine-grained, and scalable evaluation methods. This survey maps the rapidly
evolving landscape of agent evaluation, reveals the emerging trends in the
field, identifies current limitations, and proposes directions for future
research.