Orionnnn/hub-stats huggingface dataset:
The Jumping Reasoning Curve? Tracking the Evolution of Reasoning
  Performance in GPT-[n] and o-[n] Models on Multimodal Puzzles
Authors: name: Vernon Y. H. Toh, name: Yew Ken Chia, name: Deepanway Ghosal, name: Soujanya Poria
Published: 2025-02-03 05:47:04

The releases of OpenAI's o1 and o3 mark a significant paradigm shift in Large
Language Models towards advanced reasoning capabilities. Notably, o3
outperformed humans in novel problem-solving and skill acquisition on the
Abstraction and Reasoning Corpus for Artificial General Intelligence (ARC-AGI).
However, this benchmark is limited to symbolic patterns, whereas humans often
perceive and reason about multimodal scenarios involving both vision and
language data. Thus, there is an urgent need to investigate advanced reasoning
capabilities in multimodal tasks. To this end, we track the evolution of the
GPT-[n] and o-[n] series models on challenging multimodal puzzles, requiring
fine-grained visual perception with abstract or algorithmic reasoning. The
superior performance of o1 comes at nearly 750 times the computational cost of
GPT-4o, raising concerns about its efficiency. Our results reveal a clear
upward trend in reasoning capabilities across model iterations, with notable
performance jumps across GPT-series models and subsequently to o1. Nonetheless,
we observe that the o1 model still struggles with simple multimodal puzzles
requiring abstract reasoning. Furthermore, its performance in algorithmic
puzzles remains poor. We plan to continuously track new models in the series
and update our results in this paper accordingly. All resources used in this
evaluation are openly available https://github.com/declare-lab/LLM-PuzzleTest.
Scaling Embedding Layers in Language Models
Authors: name: Da Yu, name: Edith Cohen, name: Badih Ghazi, name: Yangsibo Huang, name: Pritish Kamath, name: Ravi Kumar, name: Daogao Liu, name: Chiyuan Zhang
Published: 2025-02-03 18:59:32

We propose SCONE (Scalable, Contextualized,
Offloaded, N-gram Embedding), a method for
extending input embedding layers to enhance language model performance as layer
size scales. To avoid increased decoding costs, SCONE retains the original
vocabulary while introducing embeddings for a set of frequent n-grams. These
embeddings provide contextualized representation for each input token and are
learned with a separate model during training. During inference, they are
precomputed and stored in off-accelerator memory with minimal impact on
inference speed. SCONE enables two new scaling strategies: increasing the
number of cached n-gram embeddings and scaling the model used to learn them,
all while maintaining fixed inference-time FLOPS. We show that scaling both
aspects allows SCONE to outperform a 1.9B parameter baseline across diverse
corpora, while using only half the inference-time FLOPS.
Improving Transformer World Models for Data-Efficient RL
Authors: name: Antoine Dedieu, name: Joseph Ortiz, name: Xinghua Lou, name: Carter Wendelken, name: Wolfgang Lehrach, name: J Swaroop Guntupalli, name: Miguel Lazaro-Gredilla, name: Kevin Patrick Murphy
Published: 2025-02-03 18:25:17

We present an approach to model-based RL that achieves a new state of the art
performance on the challenging Craftax-classic benchmark, an open-world 2D
survival game that requires agents to exhibit a wide range of general abilities
-- such as strong generalization, deep exploration, and long-term reasoning.
With a series of careful design choices aimed at improving sample efficiency,
our MBRL algorithm achieves a reward of 67.4% after only 1M environment steps,
significantly outperforming DreamerV3, which achieves 53.2%, and, for the first
time, exceeds human performance of 65.0%. Our method starts by constructing a
SOTA model-free baseline, using a novel policy architecture that combines CNNs
and RNNs. We then add three improvements to the standard MBRL setup: (a) "Dyna
with warmup", which trains the policy on real and imaginary data, (b) "nearest
neighbor tokenizer" on image patches, which improves the scheme to create the
transformer world model (TWM) inputs, and (c) "block teacher forcing", which
allows the TWM to reason jointly about the future tokens of the next timestep.
Process Reinforcement through Implicit Rewards
Authors: name: Ganqu Cui, name: Lifan Yuan, name: Zefan Wang, name: Hanbin Wang, name: Wendi Li, name: Bingxiang He, name: Yuchen Fan, name: Tianyu Yu, name: Qixin Xu, name: Weize Chen, name: Jiarui Yuan, name: Huayu Chen, name: Kaiyan Zhang, name: Xingtai Lv, name: Shuo Wang, name: Yuan Yao, name: Xu Han, name: Hao Peng, name: Yu Cheng, name: Zhiyuan Liu, name: Maosong Sun, name: Bowen Zhou, name: Ning Ding
Published: 2025-02-03 15:43:48

Dense process rewards have proven a more effective alternative to the sparse
outcome-level rewards in the inference-time scaling of large language models
(LLMs), particularly in tasks requiring complex multi-step reasoning. While
dense rewards also offer an appealing choice for the reinforcement learning
(RL) of LLMs since their fine-grained rewards have the potential to address
some inherent issues of outcome rewards, such as training efficiency and credit
assignment, this potential remains largely unrealized. This can be primarily
attributed to the challenges of training process reward models (PRMs) online,
where collecting high-quality process labels is prohibitively expensive, making
them particularly vulnerable to reward hacking. To address these challenges, we
propose PRIME (Process Reinforcement through IMplicit rEwards), which enables
online PRM updates using only policy rollouts and outcome labels through
implict process rewards. PRIME combines well with various advantage functions
and forgoes the dedicated reward model training phrase that existing approaches
require, substantially reducing the development overhead. We demonstrate
PRIME's effectiveness on competitional math and coding. Starting from
Qwen2.5-Math-7B-Base, PRIME achieves a 15.1% average improvement across several
key reasoning benchmarks over the SFT model. Notably, our resulting model,
Eurus-2-7B-PRIME, surpasses Qwen2.5-Math-7B-Instruct on seven reasoning
benchmarks with 10% of its training data.
Improved Training Technique for Latent Consistency Models
Authors: name: Quan Dao, name: Khanh Doan, name: Di Liu, name: Trung Le, name: Dimitris Metaxas
Published: 2025-02-03 15:25:58

Consistency models are a new family of generative models capable of producing
high-quality samples in either a single step or multiple steps. Recently,
consistency models have demonstrated impressive performance, achieving results
on par with diffusion models in the pixel space. However, the success of
scaling consistency training to large-scale datasets, particularly for
text-to-image and video generation tasks, is determined by performance in the
latent space. In this work, we analyze the statistical differences between
pixel and latent spaces, discovering that latent data often contains highly
impulsive outliers, which significantly degrade the performance of iCT in the
latent space. To address this, we replace Pseudo-Huber losses with Cauchy
losses, effectively mitigating the impact of outliers. Additionally, we
introduce a diffusion loss at early timesteps and employ optimal transport (OT)
coupling to further enhance performance. Lastly, we introduce the adaptive
scaling-c scheduler to manage the robust training process and adopt
Non-scaling LayerNorm in the architecture to better capture the statistics of
the features and reduce outlier impact. With these strategies, we successfully
train latent consistency models capable of high-quality sampling with one or
two steps, significantly narrowing the performance gap between latent
consistency and diffusion models. The implementation is released here:
https://github.com/quandao10/sLCT/
AIN: The Arabic INclusive Large Multimodal Model
Authors: name: Ahmed Heakl, name: Sara Ghaboura, name: Omkar Thawkar, name: Fahad Shahbaz Khan, name: Hisham Cholakkal, name: Rao Muhammad Anwer, name: Salman Khan
Published: 2025-01-31 18:58:20

Amid the swift progress of large language models (LLMs) and their evolution
into large multimodal models (LMMs), significant strides have been made in
high-resource languages such as English and Chinese. While Arabic LLMs have
seen notable progress, Arabic LMMs remain largely unexplored, often narrowly
focusing on a few specific aspects of the language and visual understanding. To
bridge this gap, we introduce AIN-the Arabic Inclusive Multimodal
Model-designed to excel across diverse domains. AIN is an English-Arabic
bilingual LMM designed to excel in English and Arabic, leveraging carefully
constructed 3.6 million high-quality Arabic-English multimodal data samples.
AIN demonstrates state-of-the-art Arabic performance, while also possessing
strong English-language visual capabilities. On the recent CAMEL-Bench
benchmark comprising 38 sub-domains including, multi-image understanding,
complex visual perception, handwritten document understanding, video
understanding, medical imaging, plant diseases, and remote sensing-based land
use understanding, our AIN demonstrates strong performance with the 7B model
outperforming GPT-4o by an absolute gain of 3.4% averaged over eight domains
and 38 sub-domains. AIN's superior capabilities position it as a significant
step toward empowering Arabic speakers with advanced multimodal generative AI
tools across diverse applications.
ChunkKV: Semantic-Preserving KV Cache Compression for Efficient
  Long-Context LLM Inference
Authors: name: Xiang Liu, name: Zhenheng Tang, name: Peijie Dong, name: Zeyu Li, name: Bo Li, name: Xuming Hu, name: Xiaowen Chu
Published: 2025-02-01 03:49:47

To reduce memory costs in long-context inference with Large Language Models
(LLMs), many recent works focus on compressing the key-value (KV) cache of
different tokens. However, we identify that the previous KV cache compression
methods measure token importance individually, neglecting the dependency
between different tokens in the real-world language characterics. In light of
this, we introduce ChunkKV, grouping the tokens in a chunk as a basic
compressing unit, and retaining the most informative semantic chunks while
discarding the less important ones. Furthermore, observing that ChunkKV
exhibits higher similarity in the preserved indices across different layers, we
propose layer-wise index reuse to further reduce computational overhead. We
evaluated ChunkKV on cutting-edge long-context benchmarks including LongBench
and Needle-In-A-HayStack, as well as the GSM8K and JailbreakV in-context
learning benchmark. Our experiments with instruction tuning and multi-step
reasoning (O1 and R1) LLMs, achieve up to 10\% performance improvement under
aggressive compression ratios compared to existing methods.
SAeUron: Interpretable Concept Unlearning in Diffusion Models with
  Sparse Autoencoders
Authors: name: Bartosz Cywiński, name: Kamil Deja
Published: 2025-01-29 23:29:47

Diffusion models, while powerful, can inadvertently generate harmful or
undesirable content, raising significant ethical and safety concerns. Recent
machine unlearning approaches offer potential solutions but often lack
transparency, making it difficult to understand the changes they introduce to
the base model. In this work, we introduce SAeUron, a novel method leveraging
features learned by sparse autoencoders (SAEs) to remove unwanted concepts in
text-to-image diffusion models. First, we demonstrate that SAEs, trained in an
unsupervised manner on activations from multiple denoising timesteps of the
diffusion model, capture sparse and interpretable features corresponding to
specific concepts. Building on this, we propose a feature selection method that
enables precise interventions on model activations to block targeted content
while preserving overall performance. Evaluation with the competitive
UnlearnCanvas benchmark on object and style unlearning highlights SAeUron's
state-of-the-art performance. Moreover, we show that with a single SAE, we can
remove multiple concepts simultaneously and that in contrast to other methods,
SAeUron mitigates the possibility of generating unwanted content, even under
adversarial attack. Code and checkpoints are available at:
https://github.com/cywinski/SAeUron.
Zero-Shot Novel View and Depth Synthesis with Multi-View Geometric
  Diffusion
Authors: name: Vitor Guizilini, name: Muhammad Zubair Irshad, name: Dian Chen, name: Greg Shakhnarovich, name: Rares Ambrus
Published: 2025-01-30 23:43:06

Current methods for 3D scene reconstruction from sparse posed images employ
intermediate 3D representations such as neural fields, voxel grids, or 3D
Gaussians, to achieve multi-view consistent scene appearance and geometry. In
this paper we introduce MVGD, a diffusion-based architecture capable of direct
pixel-level generation of images and depth maps from novel viewpoints, given an
arbitrary number of input views. Our method uses raymap conditioning to both
augment visual features with spatial information from different viewpoints, as
well as to guide the generation of images and depth maps from novel views. A
key aspect of our approach is the multi-task generation of images and depth
maps, using learnable task embeddings to guide the diffusion process towards
specific modalities. We train this model on a collection of more than 60
million multi-view samples from publicly available datasets, and propose
techniques to enable efficient and consistent learning in such diverse
conditions. We also propose a novel strategy that enables the efficient
training of larger models by incrementally fine-tuning smaller ones, with
promising scaling behavior. Through extensive experiments, we report
state-of-the-art results in multiple novel view synthesis benchmarks, as well
as multi-view stereo and video depth estimation.
MatAnyone: Stable Video Matting with Consistent Memory Propagation
Authors: name: Peiqing Yang, name: Shangchen Zhou, name: Jixin Zhao, name: Qingyi Tao, name: Chen Change Loy
Published: 2025-01-24 17:56:24

Auxiliary-free human video matting methods, which rely solely on input
frames, often struggle with complex or ambiguous backgrounds. To address this,
we propose MatAnyone, a robust framework tailored for target-assigned video
matting. Specifically, building on a memory-based paradigm, we introduce a
consistent memory propagation module via region-adaptive memory fusion, which
adaptively integrates memory from the previous frame. This ensures semantic
stability in core regions while preserving fine-grained details along object
boundaries. For robust training, we present a larger, high-quality, and diverse
dataset for video matting. Additionally, we incorporate a novel training
strategy that efficiently leverages large-scale segmentation data, boosting
matting stability. With this new network design, dataset, and training
strategy, MatAnyone delivers robust and accurate video matting results in
diverse real-world scenarios, outperforming existing methods.
Scalable-Softmax Is Superior for Attention
Authors: name: Ken M. Nakanishi
Published: 2025-01-31 18:55:35

The maximum element of the vector output by the Softmax function approaches
zero as the input vector size increases. Transformer-based language models rely
on Softmax to compute attention scores, causing the attention distribution to
flatten as the context size grows. This reduces the model's ability to
prioritize key information effectively and potentially limits its length
generalization. To address this problem, we propose Scalable-Softmax (SSMax),
which replaces Softmax in scenarios where the input vector size varies. SSMax
can be seamlessly integrated into existing Transformer-based architectures.
Experimental results in language modeling show that models using SSMax not only
achieve faster loss reduction during pretraining but also significantly improve
performance in long contexts and key information retrieval. Furthermore, an
analysis of attention scores reveals that SSMax enables the model to focus
attention on key information even in long contexts. Additionally, although
models that use SSMax from the beginning of pretraining achieve better length
generalization, those that have already started pretraining can still gain some
of this ability by replacing Softmax in the attention layers with SSMax, either
during or after pretraining.
The Surprising Agreement Between Convex Optimization Theory and
  Learning-Rate Scheduling for Large Model Training
Authors: name: Fabian Schaipp, name: Alexander Hägele, name: Adrien Taylor, name: Umut Simsekli, name: Francis Bach
Published: 2025-01-31 08:55:56

We show that learning-rate schedules for large model training behave
surprisingly similar to a performance bound from non-smooth convex optimization
theory. We provide a bound for the constant schedule with linear cooldown; in
particular, the practical benefit of cooldown is reflected in the bound due to
the absence of logarithmic terms. Further, we show that this surprisingly close
match between optimization theory and practice can be exploited for
learning-rate tuning: we achieve noticeable improvements for training 124M and
210M Llama-type models by (i) extending the schedule for continued training
with optimal learning-rate, and (ii) transferring the optimal learning-rate
across schedules.
PixelWorld: Towards Perceiving Everything as Pixels
Authors: name: Zhiheng Lyu, name: Xueguang Ma, name: Wenhu Chen
Published: 2025-01-31 17:39:21

Existing foundation models typically process visual input as pixels and
textual input as tokens, a paradigm that contrasts with human perception, where
both modalities are processed in a unified manner. With the rise of embodied
and agentic AI, where inputs primarily come from camera pixels, the need for a
unified perception framework becomes increasingly evident. In this paper, we
propose to unify all modalities (text, tables, code, diagrams, images, etc) as
pixel inputs, i.e. "Perceive Everything as Pixels" (PEAP). We introduce
PixelWorld, a novel evaluation suite that unifies all the mentioned modalities
into pixel space to gauge the existing models' performance. Our findings show
that (1) PEAP outperforms baseline with token-based input in multimodal
datasets, benefiting from unified input for better disambiguation, (2)
significant declines in reasoning and coding capabilities across all models
when processing pixel-based input, underscoring the need to enhance foundation
models' perceptual abilities, (3) larger models can maintain strong performance
on non-reasoning tasks under PEAP, while smaller models like Phi-3.5-V suffer
significant performance degradation, (4) the attention pattern of PEAP is
highly aligned with text token input, (5) PEAP can be accelerated significantly
by exploiting the spatial sparsity. We conclude that the existing frontier
models are competent in pixel perception, however, there is still headroom for
improvement. Our code, dataset will be released upon acceptance.
Self-supervised Quantized Representation for Seamlessly Integrating
  Knowledge Graphs with Large Language Models
Authors: name: Qika Lin, name: Tianzhe Zhao, name: Kai He, name: Zhen Peng, name: Fangzhi Xu, name: Ling Huang, name: Jingying Ma, name: Mengling Feng
Published: 2025-01-30 03:40:20

Due to the presence of the natural gap between Knowledge Graph (KG)
structures and the natural language, the effective integration of holistic
structural information of KGs with Large Language Models (LLMs) has emerged as
a significant question. To this end, we propose a two-stage framework to learn
and apply quantized codes for each entity, aiming for the seamless integration
of KGs with LLMs. Firstly, a self-supervised quantized representation (SSQR)
method is proposed to compress both KG structural and semantic knowledge into
discrete codes (\ie, tokens) that align the format of language sentences. We
further design KG instruction-following data by viewing these learned codes as
features to directly input to LLMs, thereby achieving seamless integration. The
experiment results demonstrate that SSQR outperforms existing unsupervised
quantized methods, producing more distinguishable codes. Further, the
fine-tuned LLaMA2 and LLaMA3.1 also have superior performance on KG link
prediction and triple classification tasks, utilizing only 16 tokens per entity
instead of thousands in conventional prompting methods.
INT: Instance-Specific Negative Mining for Task-Generic Promptable
  Segmentation
Authors: name: Jian Hu, name: Zixu Cheng, name: Shaogang Gong
Published: 2025-01-30 21:07:14

Task-generic promptable image segmentation aims to achieve segmentation of
diverse samples under a single task description by utilizing only one
task-generic prompt. Current methods leverage the generalization capabilities
of Vision-Language Models (VLMs) to infer instance-specific prompts from these
task-generic prompts in order to guide the segmentation process. However, when
VLMs struggle to generalise to some image instances, predicting
instance-specific prompts becomes poor. To solve this problem, we introduce
Instance-specific Negative Mining for Task-Generic
Promptable Segmentation (INT). The key idea of INT is to adaptively
reduce the influence of irrelevant (negative) prior knowledge whilst to
increase the use the most plausible prior knowledge, selected by negative
mining with higher contrast, in order to optimise instance-specific prompts
generation. Specifically, INT consists of two components: (1) instance-specific
prompt generation, which progressively fliters out incorrect information in
prompt generation; (2) semantic mask generation, which ensures each image
instance segmentation matches correctly the semantics of the instance-specific
prompts. INT is validated on six datasets, including camouflaged objects and
medical images, demonstrating its effectiveness, robustness and scalability.
Unraveling the Capabilities of Language Models in News Summarization
Authors: name: Abdurrahman Odabaşı, name: Göksel Biricik
Published: 2025-01-30 04:20:16

Given the recent introduction of multiple language models and the ongoing
demand for improved Natural Language Processing tasks, particularly
summarization, this work provides a comprehensive benchmarking of 20 recent
language models, focusing on smaller ones for the news summarization task. In
this work, we systematically test the capabilities and effectiveness of these
models in summarizing news article texts which are written in different styles
and presented in three distinct datasets. Specifically, we focus in this study
on zero-shot and few-shot learning settings and we apply a robust evaluation
methodology that combines different evaluation concepts including automatic
metrics, human evaluation, and LLM-as-a-judge. Interestingly, including
demonstration examples in the few-shot learning setting did not enhance models'
performance and, in some cases, even led to worse quality of the generated
summaries. This issue arises mainly due to the poor quality of the gold
summaries that have been used as reference summaries, which negatively impacts
the models' performance. Furthermore, our study's results highlight the
exceptional performance of GPT-3.5-Turbo and GPT-4, which generally dominate
due to their advanced capabilities. However, among the public models evaluated,
certain models such as Qwen1.5-7B, SOLAR-10.7B-Instruct-v1.0, Meta-Llama-3-8B
and Zephyr-7B-Beta demonstrated promising results. These models showed
significant potential, positioning them as competitive alternatives to large
models for the task of news summarization.