Orionnnn/hub-stats huggingface dataset:
MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for
  Reasoning Quality, Robustness, and Efficiency
Authors: name: Dongzhi Jiang, name: Renrui Zhang, name: Ziyu Guo, name: Yanwei Li, name: Yu Qi, name: Xinyan Chen, name: Liuhui Wang, name: Jianhan Jin, name: Claire Guo, name: Shen Yan, name: Bo Zhang, name: Chaoyou Fu, name: Peng Gao, name: Hongsheng Li
Published: 2025-02-13 18:59:46

Answering questions with Chain-of-Thought (CoT) has significantly enhanced
the reasoning capabilities of Large Language Models (LLMs), yet its impact on
Large Multimodal Models (LMMs) still lacks a systematic assessment and in-depth
investigation. In this paper, we introduce MME-CoT, a specialized benchmark
evaluating the CoT reasoning performance of LMMs, spanning six domains: math,
science, OCR, logic, space-time, and general scenes. As the first comprehensive
study in this area, we propose a thorough evaluation suite incorporating three
novel metrics that assess the reasoning quality, robustness, and efficiency at
a fine-grained level. Leveraging curated high-quality data and a unique
evaluation strategy, we conduct an in-depth analysis of state-of-the-art LMMs,
uncovering several key insights: 1) Models with reflection mechanism
demonstrate a superior CoT quality, with Kimi k1.5 outperforming GPT-4o and
demonstrating the highest quality results; 2) CoT prompting often degrades LMM
performance on perception-heavy tasks, suggesting a potentially harmful
overthinking behavior; and 3) Although the CoT quality is high, LMMs with
reflection exhibit significant inefficiency in both normal response and
self-correction phases. We hope MME-CoT serves as a foundation for advancing
multimodal reasoning in LMMs. Project Page: https://mmecot.github.io/
Typhoon T1: An Open Thai Reasoning Model
Authors: name: Pittawat Taveekitworachai, name: Potsawee Manakul, name: Kasima Tharnpipitchai, name: Kunat Pipatanakul
Published: 2025-02-13 07:55:54

This paper introduces Typhoon T1, an open effort to develop an open Thai
reasoning model. A reasoning model is a relatively new type of generative model
built on top of large language models (LLMs). A reasoning model generates a
long chain of thought before arriving at a final answer, an approach found to
improve performance on complex tasks. However, details on developing such a
model are limited, especially for reasoning models that can generate traces in
a low-resource language. Typhoon T1 presents an open effort that dives into the
details of developing a reasoning model in a more cost-effective way by
leveraging supervised fine-tuning using open datasets, instead of reinforcement
learning. This paper shares the details about synthetic data generation and
training, as well as our dataset and model weights. Additionally, we provide
insights gained from developing a reasoning model that generalizes across
domains and is capable of generating reasoning traces in a low-resource
language, using Thai as an example. We hope this open effort provides a
foundation for further research in this field.
CoT-Valve: Length-Compressible Chain-of-Thought Tuning
Authors: name: Xinyin Ma, name: Guangnian Wan, name: Runpeng Yu, name: Gongfan Fang, name: Xinchao Wang
Published: 2025-02-13 18:52:36

Chain-of-Thought significantly enhances a model's reasoning capability, but
it also comes with a considerable increase in inference costs due to long
chains. With the observation that the reasoning path can be easily compressed
under easy tasks but struggle on hard tasks, we explore the feasibility of
elastically controlling the length of reasoning paths with only one model,
thereby reducing the inference overhead of reasoning models dynamically based
on task difficulty. We introduce a new tuning and inference strategy named
CoT-Valve, designed to allow models to generate reasoning chains of varying
lengths. To achieve this, we propose to identify a direction in the parameter
space that, when manipulated, can effectively control the length of generated
CoT. Moreover, we show that this property is valuable for compressing the
reasoning chain. We construct datasets with chains from long to short for the
same questions and explore two enhanced strategies for CoT-Valve: (1) a precise
length-compressible CoT tuning method, and (2) a progressive chain length
compression approach. Our experiments show that CoT-Valve successfully enables
controllability and compressibility of the chain and shows better performance
than the prompt-based control. We applied this method to QwQ-32B-Preview,
reducing reasoning chains on GSM8K from 741 to 225 tokens with a minor
performance drop (95.07% to 94.92%) and on AIME from 6827 to 4629 tokens, with
only one additional incorrect answer.
mmE5: Improving Multimodal Multilingual Embeddings via High-quality
  Synthetic Data
Authors: name: Haonan Chen, name: Liang Wang, name: Nan Yang, name: Yutao Zhu, name: Ziliang Zhao, name: Furu Wei, name: Zhicheng Dou
Published: 2025-02-12 15:03:33

Multimodal embedding models have gained significant attention for their
ability to map data from different modalities, such as text and images, into a
unified representation space. However, the limited labeled multimodal data
often hinders embedding performance. Recent approaches have leveraged data
synthesis to address this problem, yet the quality of synthetic data remains a
critical bottleneck. In this work, we identify three criteria for high-quality
synthetic multimodal data. First, broad scope ensures that the generated data
covers diverse tasks and modalities, making it applicable to various downstream
scenarios. Second, robust cross-modal alignment makes different modalities
semantically consistent. Third, high fidelity ensures that the synthetic data
maintains realistic details to enhance its reliability. Guided by these
principles, we synthesize datasets that: (1) cover a wide range of tasks,
modality combinations, and languages, (2) are generated via a deep thinking
process within a single pass of a multimodal large language model, and (3)
incorporate real-world images with accurate and relevant texts, ensuring
fidelity through self-evaluation and refinement. Leveraging these high-quality
synthetic and labeled datasets, we train a multimodal multilingual E5 model
mmE5. Extensive experiments demonstrate that mmE5 achieves state-of-the-art
performance on the MMEB Benchmark and superior multilingual performance on the
XTD benchmark. Our codes, datasets and models are released in
https://github.com/haon-chen/mmE5.
EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language
  Models for Vision-Driven Embodied Agents
Authors: name: Rui Yang, name: Hanyang Chen, name: Junyu Zhang, name: Mark Zhao, name: Cheng Qian, name: Kangrui Wang, name: Qineng Wang, name: Teja Venkat Koripella, name: Marziyeh Movahedi, name: Manling Li, name: Heng Ji, name: Huan Zhang, name: Tong Zhang
Published: 2025-02-13 18:11:34

Leveraging Multi-modal Large Language Models (MLLMs) to create embodied
agents offers a promising avenue for tackling real-world tasks. While
language-centric embodied agents have garnered substantial attention,
MLLM-based embodied agents remain underexplored due to the lack of
comprehensive evaluation frameworks. To bridge this gap, we introduce
EmbodiedBench, an extensive benchmark designed to evaluate vision-driven
embodied agents. EmbodiedBench features: (1) a diverse set of 1,128 testing
tasks across four environments, ranging from high-level semantic tasks (e.g.,
household) to low-level tasks involving atomic actions (e.g., navigation and
manipulation); and (2) six meticulously curated subsets evaluating essential
agent capabilities like commonsense reasoning, complex instruction
understanding, spatial awareness, visual perception, and long-term planning.
Through extensive experiments, we evaluated 13 leading proprietary and
open-source MLLMs within EmbodiedBench. Our findings reveal that: MLLMs excel
at high-level tasks but struggle with low-level manipulation, with the best
model, GPT-4o, scoring only 28.9% on average. EmbodiedBench provides a
multifaceted standardized evaluation platform that not only highlights existing
challenges but also offers valuable insights to advance MLLM-based embodied
agents. Our code is available at https://embodiedbench.github.io.
Skrr: Skip and Re-use Text Encoder Layers for Memory Efficient
  Text-to-Image Generation
Authors: name: Hoigi Seo, name: Wongi Jeong, name: Jae-sun Seo, name: Se Young Chun
Published: 2025-02-12 15:03:26

Large-scale text encoders in text-to-image (T2I) diffusion models have
demonstrated exceptional performance in generating high-quality images from
textual prompts. Unlike denoising modules that rely on multiple iterative
steps, text encoders require only a single forward pass to produce text
embeddings. However, despite their minimal contribution to total inference time
and floating-point operations (FLOPs), text encoders demand significantly
higher memory usage, up to eight times more than denoising modules. To address
this inefficiency, we propose Skip and Re-use layers (Skrr), a simple yet
effective pruning strategy specifically designed for text encoders in T2I
diffusion models. Skrr exploits the inherent redundancy in transformer blocks
by selectively skipping or reusing certain layers in a manner tailored for T2I
tasks, thereby reducing memory consumption without compromising performance.
Extensive experiments demonstrate that Skrr maintains image quality comparable
to the original model even under high sparsity levels, outperforming existing
blockwise pruning methods. Furthermore, Skrr achieves state-of-the-art memory
efficiency while preserving performance across multiple evaluation metrics,
including the FID, CLIP, DreamSim, and GenEval scores.
InfiniteHiP: Extending Language Model Context Up to 3 Million Tokens on
  a Single GPU
Authors: name: Heejun Lee, name: Geon Park, name: Jaduk Suh, name: Sung Ju Hwang
Published: 2025-02-13 02:52:01

In modern large language models (LLMs), handling very long context lengths
presents significant challenges as it causes slower inference speeds and
increased memory costs. Additionally, most existing pre-trained LLMs fail to
generalize beyond their original training sequence lengths. To enable efficient
and practical long-context utilization, we introduce InfiniteHiP, a novel, and
practical LLM inference framework that accelerates processing by dynamically
eliminating irrelevant context tokens through a modular hierarchical token
pruning algorithm. Our method also allows generalization to longer sequences by
selectively applying various RoPE adjustment methods according to the internal
attention patterns within LLMs. Furthermore, we offload the key-value cache to
host memory during inference, significantly reducing GPU memory pressure. As a
result, InfiniteHiP enables the processing of up to 3 million tokens on a
single L40s 48GB GPU -- 3x larger -- without any permanent loss of context
information. Our framework achieves an 18.95x speedup in attention decoding for
a 1 million token context without requiring additional training. We implement
our method in the SGLang framework and demonstrate its effectiveness and
practicality through extensive evaluations.
TripoSG: High-Fidelity 3D Shape Synthesis using Large-Scale Rectified
  Flow Models
Authors: name: Yangguang Li, name: Zi-Xin Zou, name: Zexiang Liu, name: Dehu Wang, name: Yuan Liang, name: Zhipeng Yu, name: Xingchao Liu, name: Yuan-Chen Guo, name: Ding Liang, name: Wanli Ouyang, name: Yan-Pei Cao
Published: 2025-02-10 16:07:54

Recent advancements in diffusion techniques have propelled image and video
generation to unprece- dented levels of quality, significantly accelerating the
deployment and application of generative AI. However, 3D shape generation
technology has so far lagged behind, constrained by limitations in 3D data
scale, complexity of 3D data process- ing, and insufficient exploration of
advanced tech- niques in the 3D domain. Current approaches to 3D shape
generation face substantial challenges in terms of output quality,
generalization capa- bility, and alignment with input conditions. We present
TripoSG, a new streamlined shape diffu- sion paradigm capable of generating
high-fidelity 3D meshes with precise correspondence to input images.
Specifically, we propose: 1) A large-scale rectified flow transformer for 3D
shape generation, achieving state-of-the-art fidelity through training on
extensive, high-quality data. 2) A hybrid supervised training strategy
combining SDF, normal, and eikonal losses for 3D VAE, achieving high- quality
3D reconstruction performance. 3) A data processing pipeline to generate 2
million high- quality 3D samples, highlighting the crucial rules for data
quality and quantity in training 3D gen- erative models. Through comprehensive
experi- ments, we have validated the effectiveness of each component in our new
framework. The seamless integration of these parts has enabled TripoSG to
achieve state-of-the-art performance in 3D shape generation. The resulting 3D
shapes exhibit en- hanced detail due to high-resolution capabilities and
demonstrate exceptional fidelity to input im- ages. Moreover, TripoSG
demonstrates improved versatility in generating 3D models from diverse image
styles and contents, showcasing strong gen- eralization capabilities. To foster
progress and innovation in the field of 3D generation, we will make our model
publicly available.
An Open Recipe: Adapting Language-Specific LLMs to a Reasoning Model in
  One Day via Model Merging
Authors: name: Kunat Pipatanakul, name: Pittawat Taveekitworachai, name: Potsawee Manakul, name: Kasima Tharnpipitchai
Published: 2025-02-13 08:10:45

This paper investigates data selection and model merging methodologies aimed
at incorporating advanced reasoning capabilities such as those of DeepSeek R1
into language-specific large language models (LLMs), with a particular focus on
the Thai LLM. Our goal is to enhance the reasoning capabilities of
language-specific LLMs while maintaining their target language abilities.
DeepSeek R1 excels in reasoning but primarily benefits high-resource languages
such as English and Chinese. However, low-resource languages remain underserved
due to the dominance of English-centric training data and model optimizations,
which limit performance in these languages. This limitation results in
unreliable code-switching and diminished effectiveness on tasks in low-resource
languages. Meanwhile, local and regional LLM initiatives have attempted to
bridge this gap by developing language-specific LLMs that focus on improving
local linguistic fidelity. We demonstrate that, with only publicly available
datasets and a computational budget of $120, it is possible to enhance the
reasoning capabilities of language-specific LLMs to match the level of DeepSeek
R1, without compromising their performance on target language tasks.
The Stochastic Parrot on LLM's Shoulder: A Summative Assessment of
  Physical Concept Understanding
Authors: name: Mo Yu, name: Lemao Liu, name: Junjie Wu, name: Tsz Ting Chung, name: Shunchi Zhang, name: Jiangnan Li, name: Dit-Yan Yeung, name: Jie Zhou
Published: 2025-02-13 04:00:03

In a systematic way, we investigate a widely asked question: Do LLMs really
understand what they say?, which relates to the more familiar term Stochastic
Parrot. To this end, we propose a summative assessment over a carefully
designed physical concept understanding task, PhysiCo. Our task alleviates the
memorization issue via the usage of grid-format inputs that abstractly describe
physical phenomena. The grids represents varying levels of understanding, from
the core phenomenon, application examples to analogies to other abstract
patterns in the grid world. A comprehensive study on our task demonstrates: (1)
state-of-the-art LLMs, including GPT-4o, o1 and Gemini 2.0 flash thinking, lag
behind humans by ~40%; (2) the stochastic parrot phenomenon is present in LLMs,
as they fail on our grid task but can describe and recognize the same concepts
well in natural language; (3) our task challenges the LLMs due to intrinsic
difficulties rather than the unfamiliar grid format, as in-context learning and
fine-tuning on same formatted data added little to their performance.
Logical Reasoning in Large Language Models: A Survey
Authors: name: Hanmeng Liu, name: Zhizhang Fu, name: Mengru Ding, name: Ruoxi Ning, name: Chaoli Zhang, name: Xiaozhang Liu, name: Yue Zhang
Published: 2025-02-13 09:19:14

With the emergence of advanced reasoning models like OpenAI o3 and
DeepSeek-R1, large language models (LLMs) have demonstrated remarkable
reasoning capabilities. However, their ability to perform rigorous logical
reasoning remains an open question. This survey synthesizes recent advancements
in logical reasoning within LLMs, a critical area of AI research. It outlines
the scope of logical reasoning in LLMs, its theoretical foundations, and the
benchmarks used to evaluate reasoning proficiency. We analyze existing
capabilities across different reasoning paradigms - deductive, inductive,
abductive, and analogical - and assess strategies to enhance reasoning
performance, including data-centric tuning, reinforcement learning, decoding
strategies, and neuro-symbolic approaches. The review concludes with future
directions, emphasizing the need for further exploration to strengthen logical
reasoning in AI systems.
SelfCite: Self-Supervised Alignment for Context Attribution in Large
  Language Models
Authors: name: Yung-Sung Chuang, name: Benjamin Cohen-Wang, name: Shannon Zejiang Shen, name: Zhaofeng Wu, name: Hu Xu, name: Xi Victoria Lin, name: James Glass, name: Shang-Wen Li, name: Wen-tau Yih
Published: 2025-02-13 18:55:13

We introduce SelfCite, a novel self-supervised approach that aligns LLMs to
generate high-quality, fine-grained, sentence-level citations for the
statements in their generated responses. Instead of only relying on costly and
labor-intensive annotations, SelfCite leverages a reward signal provided by the
LLM itself through context ablation: If a citation is necessary, removing the
cited text from the context should prevent the same response; if sufficient,
retaining the cited text alone should preserve the same response. This reward
can guide the inference-time best-of-N sampling strategy to improve citation
quality significantly, as well as be used in preference optimization to
directly fine-tune the models for generating better citations. The
effectiveness of SelfCite is demonstrated by increasing citation F1 up to 5.3
points on the LongBench-Cite benchmark across five long-form question answering
tasks.
Homeomorphism Prior for False Positive and Negative Problem in Medical
  Image Dense Contrastive Representation Learning
Authors: name: Yuting He, name: Boyu Wang, name: Rongjun Ge, name: Yang Chen, name: Guanyu Yang, name: Shuo Li
Published: 2025-02-07 19:34:22

Dense contrastive representation learning (DCRL) has greatly improved the
learning efficiency for image-dense prediction tasks, showing its great
potential to reduce the large costs of medical image collection and dense
annotation. However, the properties of medical images make unreliable
correspondence discovery, bringing an open problem of large-scale false
positive and negative (FP&N) pairs in DCRL. In this paper, we propose GEoMetric
vIsual deNse sImilarity (GEMINI) learning which embeds the homeomorphism prior
to DCRL and enables a reliable correspondence discovery for effective dense
contrast. We propose a deformable homeomorphism learning (DHL) which models the
homeomorphism of medical images and learns to estimate a deformable mapping to
predict the pixels' correspondence under topological preservation. It
effectively reduces the searching space of pairing and drives an implicit and
soft learning of negative pairs via a gradient. We also propose a geometric
semantic similarity (GSS) which extracts semantic information in features to
measure the alignment degree for the correspondence learning. It will promote
the learning efficiency and performance of deformation, constructing positive
pairs reliably. We implement two practical variants on two typical
representation learning tasks in our experiments. Our promising results on
seven datasets which outperform the existing methods show our great
superiority. We will release our code on a companion link:
https://github.com/YutingHe-list/GEMINI.
PDE-Controller: LLMs for Autoformalization and Reasoning of PDEs
Authors: name: Mauricio Soroco, name: Jialin Song, name: Mengzhou Xia, name: Kye Emond, name: Weiran Sun, name: Wuyang Chen
Published: 2025-02-03 00:03:41

While recent AI-for-math has made strides in pure mathematics, areas of
applied mathematics, particularly PDEs, remain underexplored despite their
significant real-world applications. We present PDE-Controller, a framework
that enables large language models (LLMs) to control systems governed by
partial differential equations (PDEs). Our approach enables LLMs to transform
informal natural language instructions into formal specifications, and then
execute reasoning and planning steps to improve the utility of PDE control. We
build a holistic solution comprising datasets (both human-written cases and 2
million synthetic samples), math-reasoning models, and novel evaluation
metrics, all of which require significant effort. Our PDE-Controller
significantly outperforms prompting the latest open-source and GPT models in
reasoning, autoformalization, and program synthesis, achieving up to a 62%
improvement in utility gain for PDE control. By bridging the gap between
language generation and PDE systems, we demonstrate the potential of LLMs in
addressing complex scientific and engineering challenges. We will release all
data, model checkpoints, and code at https://pde-controller.github.io/.
LLM Modules: Knowledge Transfer from a Large to a Small Model using
  Enhanced Cross-Attention
Authors: name: Konstantin Kolomeitsev
Published: 2025-02-12 08:48:55

In this work, we propose an architecture of LLM Modules that enables the
transfer of knowledge from a large pre-trained model to a smaller model using
an Enhanced Cross-Attention mechanism. In the proposed scheme, the Qwen2-1.5B
model is frozen and its representations are passed through specially designed
attention layers to the GPT-Neo-125M model, which is trained on limited
computational resources. Experimental results on the Bespoke-Stratos-17k
dataset demonstrate that after 15 epochs of training, the combined model
generates responses comparable in quality to those obtained by distillation. We
discuss the advantages of the modular approach, provide examples of input
queries and comparative analysis, and outline prospects for further extension
of the method.
Ignore the KL Penalty! Boosting Exploration on Critical Tokens to
  Enhance RL Fine-Tuning
Authors: name: Jean Vassoyan, name: Nathanaël Beau, name: Roman Plaud
Published: 2025-02-10 14:56:25

The ability to achieve long-term goals is a key challenge in the current
development of large language models (LLMs). To address this, pre-trained LLMs
can be fine-tuned with reinforcement learning (RL) to explore solutions that
optimize a given goal. However, exploration with LLMs is difficult, as a
balance has to be struck between discovering new solutions and staying close
enough to the pre-trained model, so as not to degrade basic capabilities. This
is typically controlled with a Kullback-Leibler (KL) penalty. In this paper, we
investigate the exploration dynamics of a small language model on a simple
arithmetic task. We show how varying degrees of pre-training influence
exploration and demonstrate the importance of "critical tokens" which have a
dramatic impact on the final outcome. Consequently, we introduce a simple
modification to the KL penalty that favors exploration on critical tokens,
increasing the efficiency of the RL fine-tuning stage.