Orionnnn/hub-stats huggingface dataset:
Self-Taught Self-Correction for Small Language Models
Authors: name: Viktor Moskvoretskii, name: Chris Biemann, name: Irina Nikishina
Published: 2025-03-11 17:57:44

Although large language models (LLMs) have achieved remarkable performance
across various tasks, they remain prone to errors. A key challenge is enabling
them to self-correct. While prior research has relied on external tools or
large proprietary models, this work explores self-correction in small language
models (SLMs) through iterative fine-tuning using solely self-generated data.
We introduce the Self-Taught Self-Correction (STaSC) algorithm, which
incorporates multiple algorithmic design choices. Experimental results on a
question-answering task demonstrate that STaSC effectively learns
self-correction, leading to significant performance improvements. Our analysis
further provides insights into the mechanisms of self-correction and the impact
of different design choices on learning dynamics and overall performance. To
support future research, we release our user-friendly codebase and lightweight
models.
Monte Carlo Diffusion for Generalizable Learning-Based RANSAC
Authors: name: Jiale Wang, name: Chen Zhao, name: Wei Ke, name: Tong Zhang
Published: 2025-03-12 14:01:18

Random Sample Consensus (RANSAC) is a fundamental approach for robustly
estimating parametric models from noisy data. Existing learning-based RANSAC
methods utilize deep learning to enhance the robustness of RANSAC against
outliers. However, these approaches are trained and tested on the data
generated by the same algorithms, leading to limited generalization to
out-of-distribution data during inference. Therefore, in this paper, we
introduce a novel diffusion-based paradigm that progressively injects noise
into ground-truth data, simulating the noisy conditions for training
learning-based RANSAC. To enhance data diversity, we incorporate Monte Carlo
sampling into the diffusion paradigm, approximating diverse data distributions
by introducing different types of randomness at multiple stages. We evaluate
our approach in the context of feature matching through comprehensive
experiments on the ScanNet and MegaDepth datasets. The experimental results
demonstrate that our Monte Carlo diffusion mechanism significantly improves the
generalization ability of learning-based RANSAC. We also develop extensive
ablation studies that highlight the effectiveness of key components in our
framework.
Multi Agent based Medical Assistant for Edge Devices
Authors: name: Sakharam Gawade, name: Shivam Akhouri, name: Chinmay Kulkarni, name: Jagdish Samant, name: Pragya Sahu, name: Aastik, name: Jai Pahal, name: Saswat Meher
Published: 2025-03-07 13:20:12

Large Action Models (LAMs) have revolutionized intelligent automation, but
their application in healthcare faces challenges due to privacy concerns,
latency, and dependency on internet access. This report introduces an ondevice,
multi-agent healthcare assistant that overcomes these limitations. The system
utilizes smaller, task-specific agents to optimize resources, ensure
scalability and high performance. Our proposed system acts as a one-stop
solution for health care needs with features like appointment booking, health
monitoring, medication reminders, and daily health reporting. Powered by the
Qwen Code Instruct 2.5 7B model, the Planner and Caller Agents achieve an
average RougeL score of 85.5 for planning and 96.5 for calling for our tasks
while being lightweight for on-device deployment. This innovative approach
combines the benefits of ondevice systems with multi-agent architectures,
paving the way for user-centric healthcare solutions.
When Large Vision-Language Model Meets Large Remote Sensing Imagery:
  Coarse-to-Fine Text-Guided Token Pruning
Authors: name: Junwei Luo, name: Yingying Zhang, name: Xue Yang, name: Kang Wu, name: Qi Zhu, name: Lei Liang, name: Jingdong Chen, name: Yansheng Li
Published: 2025-03-10 17:51:16

Efficient vision-language understanding of large Remote Sensing Images (RSIs)
is meaningful but challenging. Current Large Vision-Language Models (LVLMs)
typically employ limited pre-defined grids to process images, leading to
information loss when handling gigapixel RSIs. Conversely, using unlimited
grids significantly increases computational costs. To preserve image details
while reducing computational complexity, we propose a text-guided token pruning
method with Dynamic Image Pyramid (DIP) integration. Our method introduces: (i)
a Region Focus Module (RFM) that leverages text-aware region localization
capability to identify critical vision tokens, and (ii) a coarse-to-fine image
tile selection and vision token pruning strategy based on DIP, which is guided
by RFM outputs and avoids directly processing the entire large imagery.
Additionally, existing benchmarks for evaluating LVLMs' perception ability on
large RSI suffer from limited question diversity and constrained image sizes.
We construct a new benchmark named LRS-VQA, which contains 7,333 QA pairs
across 8 categories, with image length up to 27,328 pixels. Our method
outperforms existing high-resolution strategies on four datasets using the same
data. Moreover, compared to existing token reduction methods, our approach
demonstrates higher efficiency under high-resolution settings. Dataset and code
are in https://github.com/VisionXLab/LRS-VQA.
RewardSDS: Aligning Score Distillation via Reward-Weighted Sampling
Authors: name: Itay Chachy, name: Guy Yariv, name: Sagie Benaim
Published: 2025-03-12 17:59:47

Score Distillation Sampling (SDS) has emerged as an effective technique for
leveraging 2D diffusion priors for tasks such as text-to-3D generation. While
powerful, SDS struggles with achieving fine-grained alignment to user intent.
To overcome this, we introduce RewardSDS, a novel approach that weights noise
samples based on alignment scores from a reward model, producing a weighted SDS
loss. This loss prioritizes gradients from noise samples that yield aligned
high-reward output. Our approach is broadly applicable and can extend SDS-based
methods. In particular, we demonstrate its applicability to Variational Score
Distillation (VSD) by introducing RewardVSD. We evaluate RewardSDS and
RewardVSD on text-to-image, 2D editing, and text-to-3D generation tasks,
showing significant improvements over SDS and VSD on a diverse set of metrics
measuring generation quality and alignment to desired reward models, enabling
state-of-the-art performance. Project page is available at https://itaychachy.
github.io/reward-sds/.
WildIFEval: Instruction Following in the Wild
Authors: name: Gili Lior, name: Asaf Yehudai, name: Ariel Gera, name: Liat Ein-Dor
Published: 2025-03-09 12:06:29

Recent LLMs have shown remarkable success in following user instructions, yet
handling instructions with multiple constraints remains a significant
challenge. In this work, we introduce WildIFEval - a large-scale dataset of 12K
real user instructions with diverse, multi-constraint conditions. Unlike prior
datasets, our collection spans a broad lexical and topical spectrum of
constraints, in natural user prompts. We categorize these constraints into
eight high-level classes to capture their distribution and dynamics in
real-world scenarios. Leveraging WildIFEval, we conduct extensive experiments
to benchmark the instruction-following capabilities of leading LLMs. Our
findings reveal that all evaluated models experience performance degradation
with an increasing number of constraints. Thus, we show that all models have a
large room for improvement on such tasks. Moreover, we observe that the
specific type of constraint plays a critical role in model performance. We
release our dataset to promote further research on instruction-following under
complex, realistic conditions.
MoC: Mixtures of Text Chunking Learners for Retrieval-Augmented
  Generation System
Authors: name: Jihao Zhao, name: Zhiyuan Ji, name: Zhaoxin Fan, name: Hanyu Wang, name: Simin Niu, name: Bo Tang, name: Feiyu Xiong, name: Zhiyu Li
Published: 2025-03-12 17:59:42

Retrieval-Augmented Generation (RAG), while serving as a viable complement to
large language models (LLMs), often overlooks the crucial aspect of text
chunking within its pipeline. This paper initially introduces a dual-metric
evaluation method, comprising Boundary Clarity and Chunk Stickiness, to enable
the direct quantification of chunking quality. Leveraging this assessment
method, we highlight the inherent limitations of traditional and semantic
chunking in handling complex contextual nuances, thereby substantiating the
necessity of integrating LLMs into chunking process. To address the inherent
trade-off between computational efficiency and chunking precision in LLM-based
approaches, we devise the granularity-aware Mixture-of-Chunkers (MoC)
framework, which consists of a three-stage processing mechanism. Notably, our
objective is to guide the chunker towards generating a structured list of
chunking regular expressions, which are subsequently employed to extract chunks
from the original text. Extensive experiments demonstrate that both our
proposed metrics and the MoC framework effectively settle challenges of the
chunking task, revealing the chunking kernel while enhancing the performance of
the RAG system.
Quantizing Large Language Models for Code Generation: A Differentiated
  Replication
Authors: name: Alessandro Giagnorio, name: Antonio Mastropaolo, name: Saima Afrin, name: Massimiliano Di Penta, name: Gabriele Bavota
Published: 2025-03-10 09:26:08

Large Language Models (LLMs) have shown an impressive capability in code
generation and, specifically, to automatically implement requirements described
in natural language. The LLM effectiveness generally increases with its size:
The higher the number of LLM's trainable parameters the better its ability to
implement code. However, when it comes to deploying LLM-based code generators,
larger LLMs pose significant challenges related to their memory (and,
consequently, carbon) footprint. A previous work by Wei et al. proposed to
leverage quantization techniques to reduce the memory footprint of LLM-based
code generators without substantially degrading their effectiveness. In short,
they studied LLMs featuring up to 16B parameters, quantizing their precision
from floating point 32 bits down to int 8 bits and showing their limited impact
on code generation performance. Given the fast pace at which LLM capabilities
and quantization techniques are evolving, in this work we present a
differentiated replication of the work by Wei et al. in which we consider (i)
on the one side, more recent and larger code-related LLMs, of up to 34B
parameters; (ii) the latest advancements in model quantization techniques,
which allow pushing the compression to the extreme quantization level of 2 bits
per model parameter and; (iii) different types of calibration datasets to guide
the quantization process, including code-specific ones. Our empirical
evaluation reveals that the new frontier for LLM quantization is 4-bit
precision, resulting in an average memory footprint reduction of 70% compared
to the original model without observing any significant decrease in
performance. Additionally, when the quantization becomes even more extreme (3
and 2 bits), a code-specific calibration dataset helps to limit the loss of
performance.
More Documents, Same Length: Isolating the Challenge of Multiple
  Documents in RAG
Authors: name: Shahar Levy, name: Nir Mazor, name: Lihi Shalmon, name: Michael Hassid, name: Gabriel Stanovsky
Published: 2025-03-06 12:38:17

Retrieval-augmented generation (RAG) provides LLMs with relevant documents.
Although previous studies noted that retrieving many documents can degrade
performance, they did not isolate how the quantity of documents affects
performance while controlling for context length. We evaluate various language
models on custom datasets derived from a multi-hop QA task. We keep the context
length and position of relevant information constant while varying the number
of documents, and find that increasing the document count in RAG settings poses
significant challenges for LLMs. Additionally, our results indicate that
processing multiple documents is a separate challenge from handling long
contexts. We also make the datasets and code available:
https://github.com/shaharl6000/MoreDocsSameLen .
Cost-Optimal Grouped-Query Attention for Long-Context LLMs
Authors: name: Yingfa Chen, name: Yutong Wu, name: Xu Han, name: Zhiyuan Liu, name: Maosong Sun
Published: 2025-03-12 17:50:42

Building effective and efficient Transformer-based large language models
(LLMs) has recently become a research focus, requiring maximizing model
language capabilities and minimizing training and deployment costs. Existing
efforts have primarily described complex relationships among model performance,
parameter size, and data size, as well as searched for the optimal compute
allocation to train LLMs. However, they overlook the impacts of context length
and attention head configuration (the number of query and key-value heads in
grouped-query attention) on training and inference. In this paper, we
systematically compare models with different parameter sizes, context lengths,
and attention head configurations in terms of model performance, computational
cost, and memory cost. Then, we extend the existing scaling methods, which are
based solely on parameter size and training compute, to guide the construction
of cost-optimal LLMs during both training and inference. Our quantitative
scaling studies show that, when processing sufficiently long sequences, a
larger model with fewer attention heads can achieve a lower loss while
incurring lower computational and memory costs. Our findings provide valuable
insights for developing practical LLMs, especially in long-context processing
scenarios. We will publicly release our code and data.
GTR: Guided Thought Reinforcement Prevents Thought Collapse in RL-based
  VLM Agent Training
Authors: name: Tong Wei, name: Yijun Yang, name: Junliang Xing, name: Yuanchun Shi, name: Zongqing Lu, name: Deheng Ye
Published: 2025-03-11 15:17:02

Reinforcement learning with verifiable outcome rewards (RLVR) has effectively
scaled up chain-of-thought (CoT) reasoning in large language models (LLMs).
Yet, its efficacy in training vision-language model (VLM) agents for
goal-directed action reasoning in visual environments is less established. This
work investigates this problem through extensive experiments on complex card
games, such as 24 points, and embodied tasks from ALFWorld. We find that when
rewards are based solely on action outcomes, RL fails to incentivize CoT
reasoning in VLMs, instead leading to a phenomenon we termed thought collapse,
characterized by a rapid loss of diversity in the agent's thoughts,
state-irrelevant and incomplete reasoning, and subsequent invalid actions,
resulting in negative rewards. To counteract thought collapse, we highlight the
necessity of process guidance and propose an automated corrector that evaluates
and refines the agent's reasoning at each RL step. This simple and scalable GTR
(Guided Thought Reinforcement) framework trains reasoning and action
simultaneously without the need for dense, per-step human labeling. Our
experiments demonstrate that GTR significantly enhances the performance and
generalization of the LLaVA-7b model across various visual environments,
achieving 3-5 times higher task success rates compared to SoTA models with
notably smaller model sizes.
VLog: Video-Language Models by Generative Retrieval of Narration
  Vocabulary
Authors: name: Kevin Qinghong Lin, name: Mike Zheng Shou
Published: 2025-03-12 13:53:30

Human daily activities can be concisely narrated as sequences of routine
events (e.g., turning off an alarm) in video streams, forming an event
vocabulary. Motivated by this, we introduce VLog, a novel video understanding
framework that define video narrations as vocabulary, going beyond the typical
subword vocabularies in existing generative video-language models. Built on the
lightweight language model GPT-2, VLog feature three key innovations: (i) A
generative retrieval model, marrying language model's complex reasoning
capabilities with contrastive retrieval's efficient similarity search. (ii) A
hierarchical vocabulary derived from large-scale video narrations using our
narration pair encoding algorithm, enabling efficient indexing of specific
events (e.g., cutting a tomato) by identifying broader scenarios (e.g.,
kitchen) with expressive postfixes (e.g., by the left hand). (iii) A vocabulary
update strategy leveraging generative models to extend the vocabulary for novel
events encountered during inference. To validate our approach, we introduce
VidCap-Eval, a development set requiring concise narrations with reasoning
relationships (e.g., before and after). Experiments on EgoSchema, COIN, and
HiREST further demonstrate the effectiveness of VLog, highlighting its ability
to generate concise, contextually accurate, and efficient narrations, offering
a novel perspective on video understanding. Codes are released at
https://github.com/showlab/VLog.
TPDiff: Temporal Pyramid Video Diffusion Model
Authors: name: Lingmin Ran, name: Mike Zheng Shou
Published: 2025-03-12 17:33:22

The development of video diffusion models unveils a significant challenge:
the substantial computational demands. To mitigate this challenge, we note that
the reverse process of diffusion exhibits an inherent entropy-reducing nature.
Given the inter-frame redundancy in video modality, maintaining full frame
rates in high-entropy stages is unnecessary. Based on this insight, we propose
TPDiff, a unified framework to enhance training and inference efficiency. By
dividing diffusion into several stages, our framework progressively increases
frame rate along the diffusion process with only the last stage operating on
full frame rate, thereby optimizing computational efficiency. To train the
multi-stage diffusion model, we introduce a dedicated training framework:
stage-wise diffusion. By solving the partitioned probability flow ordinary
differential equations (ODE) of diffusion under aligned data and noise, our
training strategy is applicable to various diffusion forms and further enhances
training efficiency. Comprehensive experimental evaluations validate the
generality of our method, demonstrating 50% reduction in training cost and 1.5x
improvement in inference efficiency.
Reangle-A-Video: 4D Video Generation as Video-to-Video Translation
Authors: name: Hyeonho Jeong, name: Suhyeon Lee, name: Jong Chul Ye
Published: 2025-03-12 08:26:15

We introduce Reangle-A-Video, a unified framework for generating synchronized
multi-view videos from a single input video. Unlike mainstream approaches that
train multi-view video diffusion models on large-scale 4D datasets, our method
reframes the multi-view video generation task as video-to-videos translation,
leveraging publicly available image and video diffusion priors. In essence,
Reangle-A-Video operates in two stages. (1) Multi-View Motion Learning: An
image-to-video diffusion transformer is synchronously fine-tuned in a
self-supervised manner to distill view-invariant motion from a set of warped
videos. (2) Multi-View Consistent Image-to-Images Translation: The first frame
of the input video is warped and inpainted into various camera perspectives
under an inference-time cross-view consistency guidance using DUSt3R,
generating multi-view consistent starting images. Extensive experiments on
static view transport and dynamic camera control show that Reangle-A-Video
surpasses existing methods, establishing a new solution for multi-view video
generation. We will publicly release our code and data. Project page:
https://hyeonho99.github.io/reangle-a-video/
Motion Anything: Any to Motion Generation
Authors: name: Zeyu Zhang, name: Yiran Wang, name: Wei Mao, name: Danning Li, name: Rui Zhao, name: Biao Wu, name: Zirui Song, name: Bohan Zhuang, name: Ian Reid, name: Richard Hartley
Published: 2025-03-10 06:04:31

Conditional motion generation has been extensively studied in computer
vision, yet two critical challenges remain. First, while masked autoregressive
methods have recently outperformed diffusion-based approaches, existing masking
models lack a mechanism to prioritize dynamic frames and body parts based on
given conditions. Second, existing methods for different conditioning
modalities often fail to integrate multiple modalities effectively, limiting
control and coherence in generated motion. To address these challenges, we
propose Motion Anything, a multimodal motion generation framework that
introduces an Attention-based Mask Modeling approach, enabling fine-grained
spatial and temporal control over key frames and actions. Our model adaptively
encodes multimodal conditions, including text and music, improving
controllability. Additionally, we introduce Text-Music-Dance (TMD), a new
motion dataset consisting of 2,153 pairs of text, music, and dance, making it
twice the size of AIST++, thereby filling a critical gap in the community.
Extensive experiments demonstrate that Motion Anything surpasses
state-of-the-art methods across multiple benchmarks, achieving a 15%
improvement in FID on HumanML3D and showing consistent performance gains on
AIST++ and TMD. See our project website
https://steve-zeyu-zhang.github.io/MotionAnything
Alias-Free Latent Diffusion Models:Improving Fractional Shift
  Equivariance of Diffusion Latent Space
Authors: name: Yifan Zhou, name: Zeqi Xiao, name: Shuai Yang, name: Xingang Pan
Published: 2025-03-12 14:16:30

Latent Diffusion Models (LDMs) are known to have an unstable generation
process, where even small perturbations or shifts in the input noise can lead
to significantly different outputs. This hinders their applicability in
applications requiring consistent results. In this work, we redesign LDMs to
enhance consistency by making them shift-equivariant. While introducing
anti-aliasing operations can partially improve shift-equivariance, significant
aliasing and inconsistency persist due to the unique challenges in LDMs,
including 1) aliasing amplification during VAE training and multiple U-Net
inferences, and 2) self-attention modules that inherently lack
shift-equivariance. To address these issues, we redesign the attention modules
to be shift-equivariant and propose an equivariance loss that effectively
suppresses the frequency bandwidth of the features in the continuous domain.
The resulting alias-free LDM (AF-LDM) achieves strong shift-equivariance and is
also robust to irregular warping. Extensive experiments demonstrate that AF-LDM
produces significantly more consistent results than vanilla LDM across various
applications, including video editing and image-to-image translation. Code is
available at: https://github.com/SingleZombie/AFLDM
Block Diffusion: Interpolating Between Autoregressive and Diffusion
  Language Models
Authors: name: Marianne Arriola, name: Aaron Gokaslan, name: Justin T Chiu, name: Zhihan Yang, name: Zhixuan Qi, name: Jiaqi Han, name: Subham Sekhar Sahoo, name: Volodymyr Kuleshov
Published: 2025-03-12 17:43:40

Diffusion language models offer unique benefits over autoregressive models
due to their potential for parallelized generation and controllability, yet
they lag in likelihood modeling and are limited to fixed-length generation. In
this work, we introduce a class of block diffusion language models that
interpolate between discrete denoising diffusion and autoregressive models.
Block diffusion overcomes key limitations of both approaches by supporting
flexible-length generation and improving inference efficiency with KV caching
and parallel token sampling. We propose a recipe for building effective block
diffusion models that includes an efficient training algorithm, estimators of
gradient variance, and data-driven noise schedules to minimize the variance.
Block diffusion sets a new state-of-the-art performance among diffusion models
on language modeling benchmarks and enables generation of arbitrary-length
sequences. We provide the code, along with the model weights and blog post on
the project page: https://m-arriola.com/bd3lms/