Orionnnn/hub-stats huggingface dataset:
COCONut-PanCap: Joint Panoptic Segmentation and Grounded Captions for
  Fine-Grained Understanding and Generation
Authors: name: Xueqing Deng, name: Qihang Yu, name: Ali Athar, name: Chenglin Yang, name: Linjie Yang, name: Xiaojie Jin, name: Xiaohui Shen, name: Liang-Chieh Chen
Published: 2025-02-04 18:59:46

This paper introduces the COCONut-PanCap dataset, created to enhance panoptic
segmentation and grounded image captioning. Building upon the COCO dataset with
advanced COCONut panoptic masks, this dataset aims to overcome limitations in
existing image-text datasets that often lack detailed, scene-comprehensive
descriptions. The COCONut-PanCap dataset incorporates fine-grained,
region-level captions grounded in panoptic segmentation masks, ensuring
consistency and improving the detail of generated captions. Through
human-edited, densely annotated descriptions, COCONut-PanCap supports improved
training of vision-language models (VLMs) for image understanding and
generative models for text-to-image tasks. Experimental results demonstrate
that COCONut-PanCap significantly boosts performance across understanding and
generation tasks, offering complementary benefits to large-scale datasets. This
dataset sets a new benchmark for evaluating models on joint panoptic
segmentation and grounded captioning tasks, addressing the need for
high-quality, detailed image-text annotations in multi-modal learning.
Sample, Scrutinize and Scale: Effective Inference-Time Search by Scaling
  Verification
Authors: name: Eric Zhao, name: Pranjal Awasthi, name: Sreenivas Gollapudi
Published: 2025-02-03 21:31:07

Sampling-based search, a simple paradigm for utilizing test-time compute,
involves generating multiple candidate responses and selecting the best one --
typically by verifying each response for correctness. In this paper, we study
the scaling trends governing sampling-based search. Among our findings is that
simply scaling up a minimalist implementation that uses only random sampling
and direct self-verification results in sustained performance improvements
that, for example, elevate the Gemini v1.5 Pro model's reasoning capabilities
past that of o1-Preview on popular benchmarks. We partially attribute the
scalability of sampling-based search to a phenomenon of implicit scaling, where
sampling a larger pool of responses in turn improves verification accuracy. We
further identify two useful principles for improving self-verification
capabilities with test-time compute: (1) comparing across responses provides
helpful signals about the locations of errors and hallucinations, and (2)
different model output styles are useful for different contexts -- chains of
thought are useful for reasoning but harder to verify. We also find that,
though accurate verification can be elicited, frontier models demonstrate
remarkably weak out-of-box verification capabilities and introduce a benchmark
to measure progress on these deficiencies.
Rethinking Mixture-of-Agents: Is Mixing Different Large Language Models
  Beneficial?
Authors: name: Wenzhe Li, name: Yong Lin, name: Mengzhou Xia, name: Chi Jin
Published: 2025-02-02 05:23:29

Ensembling outputs from diverse sources is a straightforward yet effective
approach to boost performance. Mixture-of-Agents (MoA) is one such popular
ensemble method that aggregates outputs from multiple different Large Language
Models (LLMs). This paper raises the question in the context of language
models: is mixing different LLMs truly beneficial? We propose Self-MoA -- an
ensemble method that aggregates outputs from only the single top-performing
LLM. Our extensive experiments reveal that, surprisingly, Self-MoA outperforms
standard MoA that mixes different LLMs in a large number of scenarios: Self-MoA
achieves 6.6% improvement over MoA on the AlpacaEval 2.0 benchmark, and an
average of 3.8% improvement across various benchmarks, including MMLU, CRUX,
and MATH. Applying Self-MoA to one of the top-ranking models in AlpacaEval 2.0
directly achieves the new state-of-the-art performance on the leaderboard. To
understand the effectiveness of Self-MoA, we systematically investigate the
trade-off between diversity and quality of outputs under various MoA settings.
We confirm that the MoA performance is rather sensitive to the quality, and
mixing different LLMs often lowers the average quality of the models. To
complement the study, we identify the scenarios where mixing different LLMs
could be helpful. This paper further introduces a sequential version of
Self-MoA, that is capable of aggregating a large number of LLM outputs
on-the-fly over multiple rounds, and is as effective as aggregating all outputs
at once.
Concept Steerers: Leveraging K-Sparse Autoencoders for Controllable
  Generations
Authors: name: Dahye Kim, name: Deepti Ghadiyaram
Published: 2025-01-31 11:52:47

Despite the remarkable progress in text-to-image generative models, they are
prone to adversarial attacks and inadvertently generate unsafe, unethical
content. Existing approaches often rely on fine-tuning models to remove
specific concepts, which is computationally expensive, lack scalability, and/or
compromise generation quality. In this work, we propose a novel framework
leveraging k-sparse autoencoders (k-SAEs) to enable efficient and interpretable
concept manipulation in diffusion models. Specifically, we first identify
interpretable monosemantic concepts in the latent space of text embeddings and
leverage them to precisely steer the generation away or towards a given concept
(e.g., nudity) or to introduce a new concept (e.g., photographic style).
Through extensive experiments, we demonstrate that our approach is very simple,
requires no retraining of the base model nor LoRA adapters, does not compromise
the generation quality, and is robust to adversarial prompt manipulations. Our
method yields an improvement of 20.01% in unsafe concept removal,
is effective in style manipulation, and is sim5x faster than
current state-of-the-art.
Inverse Bridge Matching Distillation
Authors: name: Nikita Gushchin, name: David Li, name: Daniil Selikhanovych, name: Evgeny Burnaev, name: Dmitry Baranchuk, name: Alexander Korotin
Published: 2025-02-03 13:56:03

Learning diffusion bridge models is easy; making them fast and practical is
an art. Diffusion bridge models (DBMs) are a promising extension of diffusion
models for applications in image-to-image translation. However, like many
modern diffusion and flow models, DBMs suffer from the problem of slow
inference. To address it, we propose a novel distillation technique based on
the inverse bridge matching formulation and derive the tractable objective to
solve it in practice. Unlike previously developed DBM distillation techniques,
the proposed method can distill both conditional and unconditional types of
DBMs, distill models in a one-step generator, and use only the corrupted images
for training. We evaluate our approach for both conditional and unconditional
types of bridge matching on a wide set of setups, including super-resolution,
JPEG restoration, sketch-to-image, and other tasks, and show that our
distillation technique allows us to accelerate the inference of DBMs from 4x to
100x and even provide better generation quality than used teacher model
depending on particular setup.
Generating Multi-Image Synthetic Data for Text-to-Image Customization
Authors: name: Nupur Kumari, name: Xi Yin, name: Jun-Yan Zhu, name: Ishan Misra, name: Samaneh Azadi
Published: 2025-02-03 18:59:41

Customization of text-to-image models enables users to insert custom concepts
and generate the concepts in unseen settings. Existing methods either rely on
costly test-time optimization or train encoders on single-image training
datasets without multi-image supervision, leading to worse image quality. We
propose a simple approach that addresses both limitations. We first leverage
existing text-to-image models and 3D datasets to create a high-quality
Synthetic Customization Dataset (SynCD) consisting of multiple images of the
same object in different lighting, backgrounds, and poses. We then propose a
new encoder architecture based on shared attention mechanisms that better
incorporate fine-grained visual details from input images. Finally, we propose
a new inference technique that mitigates overexposure issues during inference
by normalizing the text and image guidance vectors. Through extensive
experiments, we show that our model, trained on the synthetic dataset with the
proposed encoder and inference algorithm, outperforms existing tuning-free
methods on standard customization benchmarks.
VideoJAM: Joint Appearance-Motion Representations for Enhanced Motion
  Generation in Video Models
Authors: name: Hila Chefer, name: Uriel Singer, name: Amit Zohar, name: Yuval Kirstain, name: Adam Polyak, name: Yaniv Taigman, name: Lior Wolf, name: Shelly Sheynin
Published: 2025-02-04 17:07:10

Despite tremendous recent progress, generative video models still struggle to
capture real-world motion, dynamics, and physics. We show that this limitation
arises from the conventional pixel reconstruction objective, which biases
models toward appearance fidelity at the expense of motion coherence. To
address this, we introduce VideoJAM, a novel framework that instills an
effective motion prior to video generators, by encouraging the model to learn a
joint appearance-motion representation. VideoJAM is composed of two
complementary units. During training, we extend the objective to predict both
the generated pixels and their corresponding motion from a single learned
representation. During inference, we introduce Inner-Guidance, a mechanism that
steers the generation toward coherent motion by leveraging the model's own
evolving motion prediction as a dynamic guidance signal. Notably, our framework
can be applied to any video model with minimal adaptations, requiring no
modifications to the training data or scaling of the model. VideoJAM achieves
state-of-the-art performance in motion coherence, surpassing highly competitive
proprietary models while also enhancing the perceived visual quality of the
generations. These findings emphasize that appearance and motion can be
complementary and, when effectively integrated, enhance both the visual quality
and the coherence of video generation. Project website:
https://hila-chefer.github.io/videojam-paper.github.io/
Can LLMs Maintain Fundamental Abilities under KV Cache Compression?
Authors: name: Xiang Liu, name: Zhenheng Tang, name: Hong Chen, name: Peijie Dong, name: Zeyu Li, name: Xiuze Zhou, name: Bo Li, name: Xuming Hu, name: Xiaowen Chu
Published: 2025-02-04 02:23:06

This paper investigates an under-explored challenge in large language models
(LLMs): the impact of KV cache compression methods on LLMs' fundamental
capabilities. While existing methods achieve impressive compression ratios on
long-context benchmarks, their effects on core model capabilities remain
understudied. We present a comprehensive empirical study evaluating prominent
KV cache compression methods across diverse tasks, spanning world knowledge,
commonsense reasoning, arithmetic reasoning, code generation, safety, and
long-context understanding and generation.Our analysis reveals that KV cache
compression methods exhibit task-specific performance degradation. Arithmetic
reasoning tasks prove particularly sensitive to aggressive compression, with
different methods showing performance drops of 17.4%-43.3%. Notably, the
DeepSeek R1 Distill model exhibits more robust compression tolerance compared
to instruction-tuned models, showing only 9.67%-25.53% performance
degradation. Based on our analysis of attention patterns and cross-task
compression performance, we propose ShotKV, a novel compression approach that
distinctly handles prefill and decoding phases while maintaining shot-level
semantic coherence. Empirical results show that ShotKV achieves 9%-18%
performance improvements on long-context generation tasks under aggressive
compression ratios.
ACECODER: Acing Coder RL via Automated Test-Case Synthesis
Authors: name: Huaye Zeng, name: Dongfu Jiang, name: Haozhe Wang, name: Ping Nie, name: Xiaotong Chen, name: Wenhu Chen
Published: 2025-02-03 18:46:04

Most progress in recent coder models has been driven by supervised
fine-tuning (SFT), while the potential of reinforcement learning (RL) remains
largely unexplored, primarily due to the lack of reliable reward data/model in
the code domain. In this paper, we address this challenge by leveraging
automated large-scale test-case synthesis to enhance code model training.
Specifically, we design a pipeline that generates extensive (question,
test-cases) pairs from existing code data. Using these test cases, we construct
preference pairs based on pass rates over sampled programs to train reward
models with Bradley-Terry loss. It shows an average of 10-point improvement for
Llama-3.1-8B-Ins and 5-point improvement for Qwen2.5-Coder-7B-Ins through
best-of-32 sampling, making the 7B model on par with 236B DeepSeek-V2.5.
Furthermore, we conduct reinforcement learning with both reward models and
test-case pass rewards, leading to consistent improvements across HumanEval,
MBPP, BigCodeBench, and LiveCodeBench (V4). Notably, we follow the R1-style
training to start from Qwen2.5-Coder-base directly and show that our RL
training can improve model on HumanEval-plus by over 25\% and MBPP-plus by 6\%
for merely 80 optimization steps. We believe our results highlight the huge
potential of reinforcement learning in coder models.
QLASS: Boosting Language Agent Inference via Q-Guided Stepwise Search
Authors: name: Zongyu Lin, name: Yao Tang, name: Xingcheng Yao, name: Da Yin, name: Ziniu Hu, name: Yizhou Sun, name: Kai-Wei Chang
Published: 2025-02-04 18:58:31

Language agents have become a promising solution to complex interactive
tasks. One of the key ingredients to the success of language agents is the
reward model on the trajectory of the agentic workflow, which provides valuable
guidance during training or inference. However, due to the lack of annotations
of intermediate interactions, most existing works use an outcome reward model
to optimize policies across entire trajectories. This may lead to sub-optimal
policies and hinder the overall performance. To address this, we propose QLASS
(Q-guided Language Agent Stepwise Search), to automatically generate
annotations by estimating Q-values in a stepwise manner for open language
agents. By introducing a reasoning tree and performing process reward modeling,
QLASS provides effective intermediate guidance for each step. With the stepwise
guidance, we propose a Q-guided generation strategy to enable language agents
to better adapt to long-term value, resulting in significant performance
improvement during model inference on complex interactive agent tasks. Notably,
even with almost half the annotated data, QLASS retains strong performance,
demonstrating its efficiency in handling limited supervision. We also
empirically demonstrate that QLASS can lead to more effective decision making
through qualitative analysis. We will release our code and data.
Satori: Reinforcement Learning with Chain-of-Action-Thought Enhances LLM
  Reasoning via Autoregressive Search
Authors: name: Maohao Shen, name: Guangtao Zeng, name: Zhenting Qi, name: Zhang-Wei Hong, name: Zhenfang Chen, name: Wei Lu, name: Gregory Wornell, name: Subhro Das, name: David Cox, name: Chuang Gan
Published: 2025-02-04 17:26:58

Large language models (LLMs) have demonstrated remarkable reasoning
capabilities across diverse domains. Recent studies have shown that increasing
test-time computation enhances LLMs' reasoning capabilities. This typically
involves extensive sampling at inference time guided by an external LLM
verifier, resulting in a two-player system. Despite external guidance, the
effectiveness of this system demonstrates the potential of a single LLM to
tackle complex tasks. Thus, we pose a new research problem: Can we internalize
the searching capabilities to fundamentally enhance the reasoning abilities of
a single LLM? This work explores an orthogonal direction focusing on
post-training LLMs for autoregressive searching (i.e., an extended reasoning
process with self-reflection and self-exploration of new strategies). To
achieve this, we propose the Chain-of-Action-Thought (COAT) reasoning and a
two-stage training paradigm: 1) a small-scale format tuning stage to
internalize the COAT reasoning format and 2) a large-scale self-improvement
stage leveraging reinforcement learning. Our approach results in Satori, a 7B
LLM trained on open-source models and data. Extensive empirical evaluations
demonstrate that Satori achieves state-of-the-art performance on mathematical
reasoning benchmarks while exhibits strong generalization to out-of-domain
tasks. Code, data, and models will be fully open-sourced.
LongDPO: Unlock Better Long-form Generation Abilities for LLMs via
  Critique-augmented Stepwise Information
Authors: name: Bowen Ping, name: Jiali Zeng, name: Fandong Meng, name: Shuo Wang, name: Jie Zhou, name: Shanghang Zhang
Published: 2025-02-04 08:25:17

Long-form generation is crucial for academic writing papers and repo-level
code generation. Despite this, current models, including GPT-4o, still exhibit
unsatisfactory performance. Existing methods that utilize preference learning
with outcome supervision often fail to provide detailed feedback for extended
contexts. This shortcoming can lead to content that does not fully satisfy
query requirements, resulting in issues like length deviations, and diminished
quality. In this paper, we propose enhancing long-form generation by
incorporating process supervision. We employ Monte Carlo Tree Search to gather
stepwise preference pairs, utilizing a global memory pool to maintain
consistency. To address the issue of suboptimal candidate selection, we
integrate external critiques to refine and improve the quality of the
preference pairs. Finally, we apply step-level DPO using the collected stepwise
preference pairs. Experimental results show that our method improves length and
quality on long-form generation benchmarks, with almost lossless performance on
general benchmarks across various model backbones.
MakeAnything: Harnessing Diffusion Transformers for Multi-Domain
  Procedural Sequence Generation
Authors: name: Yiren Song, name: Cheng Liu, name: Mike Zheng Shou
Published: 2025-02-03 17:55:30

A hallmark of human intelligence is the ability to create complex artifacts
through structured multi-step processes. Generating procedural tutorials with
AI is a longstanding but challenging goal, facing three key obstacles: (1)
scarcity of multi-task procedural datasets, (2) maintaining logical continuity
and visual consistency between steps, and (3) generalizing across multiple
domains. To address these challenges, we propose a multi-domain dataset
covering 21 tasks with over 24,000 procedural sequences. Building upon this
foundation, we introduce MakeAnything, a framework based on the diffusion
transformer (DIT), which leverages fine-tuning to activate the in-context
capabilities of DIT for generating consistent procedural sequences. We
introduce asymmetric low-rank adaptation (LoRA) for image generation, which
balances generalization capabilities and task-specific performance by freezing
encoder parameters while adaptively tuning decoder layers. Additionally, our
ReCraft model enables image-to-process generation through spatiotemporal
consistency constraints, allowing static images to be decomposed into plausible
creation sequences. Extensive experiments demonstrate that MakeAnything
surpasses existing methods, setting new performance benchmarks for procedural
generation tasks.
RandLoRA: Full-rank parameter-efficient fine-tuning of large models
Authors: name: Paul Albert, name: Frederic Z. Zhang, name: Hemanth Saratchandran, name: Cristian Rodriguez-Opazo, name: Anton van den Hengel, name: Ehsan Abbasnejad
Published: 2025-02-03 01:59:45

Low-Rank Adaptation (LoRA) and its variants have shown impressive results in
reducing the number of trainable parameters and memory requirements of large
transformer networks while maintaining fine-tuning performance. However, the
low-rank nature of the weight update inherently limits the representation power
of fine-tuned models, potentially compromising performance on complex tasks.
This raises a critical question: when a performance gap between LoRA and
standard fine-tuning is observed, is it due to the reduced number of trainable
parameters or the rank deficiency? This paper aims to answer this question by
introducing RandLoRA, a parameter-efficient method that performs full-rank
updates using a learned linear combinations of low-rank, non-trainable random
matrices. Our method limits the number of trainable parameters by restricting
optimization to diagonal scaling matrices applied to the fixed random matrices.
This allows us to effectively overcome the low-rank limitations while
maintaining parameter and memory efficiency during training. Through extensive
experimentation across vision, language, and vision-language benchmarks, we
systematically evaluate the limitations of LoRA and existing random basis
methods. Our findings reveal that full-rank updates are beneficial across
vision and language tasks individually, and even more so for vision-language
tasks, where RandLoRA significantly reduces -- and sometimes eliminates -- the
performance gap between standard fine-tuning and LoRA, demonstrating its
efficacy.
Learning to Generate Unit Tests for Automated Debugging
Authors: name: Archiki Prasad, name: Elias Stengel-Eskin, name: Justin Chih-Yao Chen, name: Zaid Khan, name: Mohit Bansal
Published: 2025-02-03 18:51:43

Unit tests (UTs) play an instrumental role in assessing code correctness as
well as providing feedback to a large language model (LLM) as it iteratively
debugs faulty code, motivating automated test generation. However, we uncover a
trade-off between generating unit test inputs that reveal errors when given a
faulty code and correctly predicting the unit test output without access to the
gold solution. To address this trade-off, we propose UTGen, which teaches LLMs
to generate unit test inputs that reveal errors along with their correct
expected outputs based on task descriptions and candidate code. We integrate
UTGen into UTDebug, a robust debugging pipeline that uses generated tests to
help LLMs debug effectively. Since model-generated tests can provide noisy
signals (e.g., from incorrectly predicted outputs), UTDebug (i) scales UTGen
via test-time compute to improve UT output prediction, and (ii) validates and
back-tracks edits based on multiple generated UTs to avoid overfitting. We show
that UTGen outperforms UT generation baselines by 7.59% based on a metric
measuring the presence of both error-revealing UT inputs and correct UT
outputs. When used with UTDebug, we find that feedback from UTGen's unit tests
improves pass@1 accuracy of Qwen-2.5 7B on HumanEvalFix and our own harder
debugging split of MBPP+ by over 3% and 12.35% (respectively) over other
LLM-based UT generation baselines.