Orionnnn/hub-stats huggingface dataset:
Unveiling Downstream Performance Scaling of LLMs: A Clustering-Based
  Perspective
Authors: name: Chengyin Xu, name: Kaiyuan Chen, name: Xiao Li, name: Ke Shen, name: Chenggang Li
Published: 2025-02-24 15:44:57

The rapid advancements in computing dramatically increase the scale and cost
of training Large Language Models (LLMs). Accurately predicting downstream task
performance prior to model training is crucial for efficient resource
allocation, yet remains challenging due to two primary constraints: (1) the
"emergence phenomenon", wherein downstream performance metrics become
meaningful only after extensive training, which limits the ability to use
smaller models for prediction; (2) Uneven task difficulty distributions and the
absence of consistent scaling laws, resulting in substantial metric
variability. Existing performance prediction methods suffer from limited
accuracy and reliability, thereby impeding the assessment of potential LLM
capabilities. To address these challenges, we propose a
Clustering-On-Difficulty (COD) downstream performance prediction framework. COD
first constructs a predictable support subset by clustering tasks based on
difficulty features, strategically excluding non-emergent and non-scalable
clusters. The scores on the selected subset serve as effective intermediate
predictors of downstream performance on the full evaluation set. With
theoretical support, we derive a mapping function that transforms performance
metrics from the predictable subset to the full evaluation set, thereby
ensuring accurate extrapolation of LLM downstream performance. The proposed
method has been applied to predict performance scaling for a 70B LLM, providing
actionable insights for training resource allocation and assisting in
monitoring the training process. Notably, COD achieves remarkable predictive
accuracy on the 70B LLM by leveraging an ensemble of small models,
demonstrating an absolute mean deviation of 1.36% across eight important LLM
evaluation benchmarks.
SpargeAttn: Accurate Sparse Attention Accelerating Any Model Inference
Authors: name: Jintao Zhang, name: Chendong Xiang, name: Haofeng Huang, name: Jia Wei, name: Haocheng Xi, name: Jun Zhu, name: Jianfei Chen
Published: 2025-02-25 12:02:17

An efficient attention implementation is essential for large models due to
its quadratic time complexity. Fortunately, attention commonly exhibits
sparsity, i.e., many values in the attention map are near zero, allowing for
the omission of corresponding computations. Many studies have utilized the
sparse pattern to accelerate attention. However, most existing works focus on
optimizing attention within specific models by exploiting certain sparse
patterns of the attention map. A universal sparse attention that guarantees
both the speedup and end-to-end performance of diverse models remains elusive.
In this paper, we propose SpargeAttn, a universal sparse and quantized
attention for any model. Our method uses a two-stage online filter: in the
first stage, we rapidly and accurately predict the attention map, enabling the
skip of some matrix multiplications in attention. In the second stage, we
design an online softmax-aware filter that incurs no extra overhead and further
skips some matrix multiplications. Experiments show that our method
significantly accelerates diverse models, including language, image, and video
generation, without sacrificing end-to-end metrics. The codes are available at
https://github.com/thu-ml/SpargeAttn.
SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open
  Software Evolution
Authors: name: Yuxiang Wei, name: Olivier Duchenne, name: Jade Copet, name: Quentin Carbonneaux, name: Lingming Zhang, name: Daniel Fried, name: Gabriel Synnaeve, name: Rishabh Singh, name: Sida I. Wang
Published: 2025-02-25 18:45:04

The recent DeepSeek-R1 release has demonstrated the immense potential of
reinforcement learning (RL) in enhancing the general reasoning capabilities of
large language models (LLMs). While DeepSeek-R1 and other follow-up work
primarily focus on applying RL to competitive coding and math problems, this
paper introduces SWE-RL, the first approach to scale RL-based LLM reasoning for
real-world software engineering. Leveraging a lightweight rule-based reward
(e.g., the similarity score between ground-truth and LLM-generated solutions),
SWE-RL enables LLMs to autonomously recover a developer's reasoning processes
and solutions by learning from extensive open-source software evolution data --
the record of a software's entire lifecycle, including its code snapshots, code
changes, and events such as issues and pull requests. Trained on top of Llama
3, our resulting reasoning model, Llama3-SWE-RL-70B, achieves a 41.0% solve
rate on SWE-bench Verified -- a human-verified collection of real-world GitHub
issues. To our knowledge, this is the best performance reported for
medium-sized (<100B) LLMs to date, even comparable to leading proprietary LLMs
like GPT-4o. Surprisingly, despite performing RL solely on software evolution
data, Llama3-SWE-RL has even emerged with generalized reasoning skills. For
example, it shows improved results on five out-of-domain tasks, namely,
function coding, library use, code reasoning, mathematics, and general language
understanding, whereas a supervised-finetuning baseline even leads to
performance degradation on average. Overall, SWE-RL opens up a new direction to
improve the reasoning capabilities of LLMs through reinforcement learning on
massive software engineering data.
OmniAlign-V: Towards Enhanced Alignment of MLLMs with Human Preference
Authors: name: Xiangyu Zhao, name: Shengyuan Ding, name: Zicheng Zhang, name: Haian Huang, name: Maosong Cao, name: Weiyun Wang, name: Jiaqi Wang, name: Xinyu Fang, name: Wenhai Wang, name: Guangtao Zhai, name: Haodong Duan, name: Hua Yang, name: Kai Chen
Published: 2025-02-25 18:05:14

Recent advancements in open-source multi-modal large language models (MLLMs)
have primarily focused on enhancing foundational capabilities, leaving a
significant gap in human preference alignment. This paper introduces
OmniAlign-V, a comprehensive dataset of 200K high-quality training samples
featuring diverse images, complex questions, and varied response formats to
improve MLLMs' alignment with human preferences. We also present MM-AlignBench,
a human-annotated benchmark specifically designed to evaluate MLLMs' alignment
with human values. Experimental results show that finetuning MLLMs with
OmniAlign-V, using Supervised Fine-Tuning (SFT) or Direct Preference
Optimization (DPO), significantly enhances human preference alignment while
maintaining or enhancing performance on standard VQA benchmarks, preserving
their fundamental capabilities. Our datasets, benchmark, code and checkpoints
have been released at https://github.com/PhoenixZ810/OmniAlign-V.
ART: Anonymous Region Transformer for Variable Multi-Layer Transparent
  Image Generation
Authors: name: Yifan Pu, name: Yiming Zhao, name: Zhicong Tang, name: Ruihong Yin, name: Haoxing Ye, name: Yuhui Yuan, name: Dong Chen, name: Jianmin Bao, name: Sirui Zhang, name: Yanbin Wang, name: Lin Liang, name: Lijuan Wang, name: Ji Li, name: Xiu Li, name: Zhouhui Lian, name: Gao Huang, name: Baining Guo
Published: 2025-02-25 16:57:04

Multi-layer image generation is a fundamental task that enables users to
isolate, select, and edit specific image layers, thereby revolutionizing
interactions with generative models. In this paper, we introduce the Anonymous
Region Transformer (ART), which facilitates the direct generation of variable
multi-layer transparent images based on a global text prompt and an anonymous
region layout. Inspired by Schema theory suggests that knowledge is organized
in frameworks (schemas) that enable people to interpret and learn from new
information by linking it to prior knowledge.}, this anonymous region layout
allows the generative model to autonomously determine which set of visual
tokens should align with which text tokens, which is in contrast to the
previously dominant semantic layout for the image generation task. In addition,
the layer-wise region crop mechanism, which only selects the visual tokens
belonging to each anonymous region, significantly reduces attention computation
costs and enables the efficient generation of images with numerous distinct
layers (e.g., 50+). When compared to the full attention approach, our method is
over 12 times faster and exhibits fewer layer conflicts. Furthermore, we
propose a high-quality multi-layer transparent image autoencoder that supports
the direct encoding and decoding of the transparency of variable multi-layer
images in a joint manner. By enabling precise control and scalable layer
generation, ART establishes a new paradigm for interactive content creation.
KV-Edit: Training-Free Image Editing for Precise Background Preservation
Authors: name: Tianrui Zhu, name: Shiyi Zhang, name: Jiawei Shao, name: Yansong Tang
Published: 2025-02-24 17:40:09

Background consistency remains a significant challenge in image editing
tasks. Despite extensive developments, existing works still face a trade-off
between maintaining similarity to the original image and generating content
that aligns with the target. Here, we propose KV-Edit, a training-free approach
that uses KV cache in DiTs to maintain background consistency, where background
tokens are preserved rather than regenerated, eliminating the need for complex
mechanisms or expensive training, ultimately generating new content that
seamlessly integrates with the background within user-provided regions. We
further explore the memory consumption of the KV cache during editing and
optimize the space complexity to O(1) using an inversion-free method. Our
approach is compatible with any DiT-based generative model without additional
training. Experiments demonstrate that KV-Edit significantly outperforms
existing approaches in terms of both background and image quality, even
surpassing training-based methods. Project webpage is available at
https://xilluill.github.io/projectpages/KV-Edit
MutaGReP: Execution-Free Repository-Grounded Plan Search for Code-Use
Authors: name: Zaid Khan, name: Ali Farhadi, name: Ranjay Krishna, name: Luca Weihs, name: Mohit Bansal, name: Tanmay Gupta
Published: 2025-02-21 18:58:17

When a human requests an LLM to complete a coding task using functionality
from a large code repository, how do we provide context from the repo to the
LLM? One approach is to add the entire repo to the LLM's context window.
However, most tasks involve only fraction of symbols from a repo, longer
contexts are detrimental to the LLM's reasoning abilities, and context windows
are not unlimited. Alternatively, we could emulate the human ability to
navigate a large repo, pick out the right functionality, and form a plan to
solve the task. We propose MutaGReP (Mutation-guided Grounded Repository Plan
Search), an approach to search for plans that decompose a user request into
natural language steps grounded in the codebase. MutaGReP performs neural tree
search in plan space, exploring by mutating plans and using a symbol retriever
for grounding. On the challenging LongCodeArena benchmark, our plans use less
than 5% of the 128K context window for GPT-4o but rival the coding performance
of GPT-4o with a context window filled with the repo. Plans produced by
MutaGReP allow Qwen 2.5 Coder 32B and 72B to match the performance of GPT-4o
with full repo context and enable progress on the hardest LongCodeArena tasks.
Project page: zaidkhan.me/MutaGReP
Pandora3D: A Comprehensive Framework for High-Quality 3D Shape and
  Texture Generation
Authors: name: Jiayu Yang, name: Taizhang Shang, name: Weixuan Sun, name: Xibin Song, name: Ziang Cheng, name: Senbo Wang, name: Shenzhou Chen, name: Weizhe Liu, name: Hongdong Li, name: Pan Ji
Published: 2025-02-20 04:22:30

This report presents a comprehensive framework for generating high-quality 3D
shapes and textures from diverse input prompts, including single images,
multi-view images, and text descriptions. The framework consists of 3D shape
generation and texture generation. (1). The 3D shape generation pipeline
employs a Variational Autoencoder (VAE) to encode implicit 3D geometries into a
latent space and a diffusion network to generate latents conditioned on input
prompts, with modifications to enhance model capacity. An alternative
Artist-Created Mesh (AM) generation approach is also explored, yielding
promising results for simpler geometries. (2). Texture generation involves a
multi-stage process starting with frontal images generation followed by
multi-view images generation, RGB-to-PBR texture conversion, and
high-resolution multi-view texture refinement. A consistency scheduler is
plugged into every stage, to enforce pixel-wise consistency among multi-view
textures during inference, ensuring seamless integration.
  The pipeline demonstrates effective handling of diverse input formats,
leveraging advanced neural architectures and novel methodologies to produce
high-quality 3D content. This report details the system architecture,
experimental results, and potential future directions to improve and expand the
framework. The source code and pretrained weights are released at:
https://github.com/Tencent/Tencent-XR-3DGen.
Mind the Gap! Static and Interactive Evaluations of Large Audio Models
Authors: name: Minzhi Li, name: William Barr Held, name: Michael J Ryan, name: Kunat Pipatanakul, name: Potsawee Manakul, name: Hao Zhu, name: Diyi Yang
Published: 2025-02-21 20:29:02

As AI chatbots become ubiquitous, voice interaction presents a compelling way
to enable rapid, high-bandwidth communication for both semantic and social
signals. This has driven research into Large Audio Models (LAMs) to power
voice-native experiences. However, aligning LAM development with user goals
requires a clear understanding of user needs and preferences to establish
reliable progress metrics. This study addresses these challenges by introducing
an interactive approach to evaluate LAMs and collecting 7,500 LAM interactions
from 484 participants. Through topic modeling of user queries, we identify
primary use cases for audio interfaces. We then analyze user preference
rankings and qualitative feedback to determine which models best align with
user needs. Finally, we evaluate how static benchmarks predict interactive
performance - our analysis reveals no individual benchmark strongly correlates
with interactive results (tau leq 0.33 for all benchmarks). While combining
multiple coarse-grained features yields modest predictive power (R^2=0.30),
only two out of twenty datasets on spoken question answering and age prediction
show significantly positive correlations. This suggests a clear need to develop
LAM evaluations that better correlate with user preferences.
Self-Taught Agentic Long Context Understanding
Authors: name: Yufan Zhuang, name: Xiaodong Yu, name: Jialian Wu, name: Ximeng Sun, name: Ze Wang, name: Jiang Liu, name: Yusheng Su, name: Jingbo Shang, name: Zicheng Liu, name: Emad Barsoum
Published: 2025-02-21 20:29:36

Answering complex, long-context questions remains a major challenge for large
language models (LLMs) as it requires effective question clarifications and
context retrieval. We propose Agentic Long-Context Understanding (AgenticLU), a
framework designed to enhance an LLM's understanding of such queries by
integrating targeted self-clarification with contextual grounding within an
agentic workflow. At the core of AgenticLU is Chain-of-Clarifications (CoC),
where models refine their understanding through self-generated clarification
questions and corresponding contextual groundings. By scaling inference as a
tree search where each node represents a CoC step, we achieve 97.8% answer
recall on NarrativeQA with a search depth of up to three and a branching factor
of eight. To amortize the high cost of this search process to training, we
leverage the preference pairs for each step obtained by the CoC workflow and
perform two-stage model finetuning: (1) supervised finetuning to learn
effective decomposition strategies, and (2) direct preference optimization to
enhance reasoning quality. This enables AgenticLU models to generate
clarifications and retrieve relevant context effectively and efficiently in a
single inference pass. Extensive experiments across seven long-context tasks
demonstrate that AgenticLU significantly outperforms state-of-the-art prompting
methods and specialized long-context LLMs, achieving robust multi-hop reasoning
while sustaining consistent performance as context length grows.
Grounded Persuasive Language Generation for Automated Marketing
Authors: name: Jibang Wu, name: Chenghao Yang, name: Simon Mahns, name: Chaoqi Wang, name: Hao Zhu, name: Fei Fang, name: Haifeng Xu
Published: 2025-02-24 03:36:57

This paper develops an agentic framework that employs large language models
(LLMs) to automate the generation of persuasive and grounded marketing content,
using real estate listing descriptions as our focal application domain. Our
method is designed to align the generated content with user preferences while
highlighting useful factual attributes. This agent consists of three key
modules: (1) Grounding Module, mimicking expert human behavior to predict
marketable features; (2) Personalization Module, aligning content with user
preferences; (3) Marketing Module, ensuring factual accuracy and the inclusion
of localized features. We conduct systematic human-subject experiments in the
domain of real estate marketing, with a focus group of potential house buyers.
The results demonstrate that marketing descriptions generated by our approach
are preferred over those written by human experts by a clear margin. Our
findings suggest a promising LLM-based agentic framework to automate
large-scale targeted marketing while ensuring responsible generation using only
facts.
InductionBench: LLMs Fail in the Simplest Complexity Class
Authors: name: Wenyue Hua, name: Tyler Wong, name: Sun Fei, name: Liangming Pan, name: Adam Jardine, name: William Yang Wang
Published: 2025-02-20 03:48:00

Large language models (LLMs) have shown remarkable improvements in reasoning
and many existing benchmarks have been addressed by models such as o1 and o3
either fully or partially. However, a majority of these benchmarks emphasize
deductive reasoning, including mathematical and coding tasks in which rules
such as mathematical axioms or programming syntax are clearly defined, based on
which LLMs can plan and apply these rules to arrive at a solution. In contrast,
inductive reasoning, where one infers the underlying rules from observed data,
remains less explored. Such inductive processes lie at the heart of scientific
discovery, as they enable researchers to extract general principles from
empirical observations. To assess whether LLMs possess this capacity, we
introduce InductionBench, a new benchmark designed to evaluate the inductive
reasoning ability of LLMs. Our experimental findings reveal that even the most
advanced models available struggle to master the simplest complexity classes
within the subregular hierarchy of functions, highlighting a notable deficiency
in current LLMs' inductive reasoning capabilities. Coda and data are available
https://github.com/Wenyueh/inductive_reasoning_benchmark.
Investigating the Impact of Quantization Methods on the Safety and
  Reliability of Large Language Models
Authors: name: Artyom Kharinaev, name: Viktor Moskvoretskii, name: Egor Shvetsov, name: Kseniia Studenikina, name: Bykov Mikhail, name: Evgeny Burnaev
Published: 2025-02-18 20:32:05

Large Language Models (LLMs) have emerged as powerful tools for addressing
modern challenges and enabling practical applications. However, their
computational expense remains a significant barrier to widespread adoption.
Quantization has emerged as a promising technique to democratize access and
enable low resource device deployment. Despite these advancements, the safety
and trustworthiness of quantized models remain underexplored, as prior studies
often overlook contemporary architectures and rely on overly simplistic
benchmarks and evaluations. To address this gap, we introduce OpenSafetyMini, a
novel open-ended safety dataset designed to better distinguish between models.
We evaluate 4 state-of-the-art quantization techniques across LLaMA and Mistral
models using 4 benchmarks, including human evaluations. Our findings reveal
that the optimal quantization method varies for 4-bit precision, while vector
quantization techniques deliver the best safety and trustworthiness performance
at 2-bit precision, providing foundation for future research.
Diagnosing COVID-19 Severity from Chest X-Ray Images Using ViT and CNN
  Architectures
Authors: name: Luis Lara, name: Lucia Eve Berger, name: Rajesh Raju, name: Shawn Whitfield
Published: 2025-02-23 15:50:42

The COVID-19 pandemic strained healthcare resources and prompted discussion
about how machine learning can alleviate physician burdens and contribute to
diagnosis. Chest x-rays (CXRs) are used for diagnosis of COVID-19, but few
studies predict the severity of a patient's condition from CXRs. In this study,
we produce a large COVID severity dataset by merging three sources and
investigate the efficacy of transfer learning using ImageNet- and
CXR-pretrained models and vision transformers (ViTs) in both severity
regression and classification tasks. A pretrained DenseNet161 model performed
the best on the three class severity prediction problem, reaching 80% accuracy
overall and 77.3%, 83.9%, and 70% on mild, moderate and severe cases,
respectively. The ViT had the best regression results, with a mean absolute
error of 0.5676 compared to radiologist-predicted severity scores. The
project's source code is publicly available.
Early-Exit and Instant Confidence Translation Quality Estimation
Authors: name: Vilém Zouhar, name: Maike Züfle, name: Beni Egressy, name: Julius Cheng, name: Jan Niehues
Published: 2025-02-20 10:27:13

Quality estimation is omnipresent in machine translation, for both evaluation
and generation. Unfortunately, quality estimation models are often opaque and
computationally expensive, making them impractical to be part of large-scale
pipelines. In this work, we tackle two connected challenges: (1) reducing the
cost of quality estimation at scale, and (2) developing an inexpensive
uncertainty estimation method for quality estimation. To address the latter, we
introduce Instant Confidence COMET, an uncertainty-aware quality estimation
model that matches the performance of previous approaches at a fraction of
their costs. We extend this to Early-Exit COMET, a quality estimation model
that can compute quality scores and associated confidences already at early
model layers, allowing us to early-exit computations and reduce evaluation
costs. We also apply our model to machine translation reranking. We combine
Early-Exit COMET with an upper confidence bound bandit algorithm to find the
best candidate from a large pool without having to run the full evaluation
model on all candidates. In both cases (evaluation and reranking) our methods
reduce the required compute by 50% with very little degradation in performance.
MegaLoc: One Retrieval to Place Them All
Authors: name: Gabriele Berton, name: Carlo Masone
Published: 2025-02-24 15:14:55

Retrieving images from the same location as a given query is an important
component of multiple computer vision tasks, like Visual Place Recognition,
Landmark Retrieval, Visual Localization, 3D reconstruction, and SLAM. However,
existing solutions are built to specifically work for one of these tasks, and
are known to fail when the requirements slightly change or when they meet
out-of-distribution data. In this paper we combine a variety of existing
methods, training techniques, and datasets to train a retrieval model, called
MegaLoc, that is performant on multiple tasks. We find that MegaLoc (1)
achieves state of the art on a large number of Visual Place Recognition
datasets, (2) impressive results on common Landmark Retrieval datasets, and (3)
sets a new state of the art for Visual Localization on the LaMAR datasets,
where we only changed the retrieval method to the existing localization
pipeline. The code for MegaLoc is available at
https://github.com/gmberton/MegaLoc