Orionnnn/hub-stats huggingface dataset:
Lost in Cultural Translation: Do LLMs Struggle with Math Across Cultural
  Contexts?
Authors: name: Aabid Karim, name: Abdul Karim, name: Bhoomika Lohana, name: Matt Keon, name: Jaswinder Singh, name: Abdul Sattar
Published: 2025-03-23 10:35:39

Large Language Models (LLMs) have significantly advanced various fields,
particularly coding, mathematical reasoning, and logical problem solving.
However, a critical question remains: Do these mathematical reasoning abilities
persist when LLMs are presented with culturally adapted math problems?
Specifically, how do LLMs perform when faced with math problems embedded in
cultural contexts that have no significant representation in main stream
web-scale AI training data? To explore this, we generated six synthetic
cultural datasets from GSM8K, a widely used benchmark for assessing LLMs'
mathematical reasoning skills. While preserving the mathematical logic and
numerical values of the original GSM8K test set, we modify cultural elements
such as personal names, food items, place names, etc. These culturally adapted
datasets provide a more reliable framework for evaluating LLMs' mathematical
reasoning under shifting cultural contexts. Our findings reveal that LLMs
struggle with math problems when cultural references change, even though the
underlying mathematical structure remains constant. Smaller models exhibit
greater performance drops compared to larger models. Interestingly, our results
also suggest that cultural familiarity can enhance mathematical reasoning. Even
models with no explicit mathematical training but exposure to relevant cultural
contexts sometimes outperform larger, mathematically proficient models on
culturally embedded math problems. This study highlights the impact of cultural
context on the mathematical reasoning abilities of LLMs, underscoring the need
for more diverse and representative training data to improve robustness in
real-world applications. The benchmark data sets and script for reproducing the
results are available at
https://github.com/akarim23131/Lost_in_Cultural_Translation
Diffusion-4K: Ultra-High-Resolution Image Synthesis with Latent
  Diffusion Models
Authors: name: Jinjin Zhang, name: Qiuyu Huang, name: Junjie Liu, name: Xiefan Guo, name: Di Huang
Published: 2025-03-24 05:25:07

In this paper, we present Diffusion-4K, a novel framework for direct
ultra-high-resolution image synthesis using text-to-image diffusion models. The
core advancements include: (1) Aesthetic-4K Benchmark: addressing the absence
of a publicly available 4K image synthesis dataset, we construct Aesthetic-4K,
a comprehensive benchmark for ultra-high-resolution image generation. We
curated a high-quality 4K dataset with carefully selected images and captions
generated by GPT-4o. Additionally, we introduce GLCM Score and Compression
Ratio metrics to evaluate fine details, combined with holistic measures such as
FID, Aesthetics and CLIPScore for a comprehensive assessment of
ultra-high-resolution images. (2) Wavelet-based Fine-tuning: we propose a
wavelet-based fine-tuning approach for direct training with photorealistic 4K
images, applicable to various latent diffusion models, demonstrating its
effectiveness in synthesizing highly detailed 4K images. Consequently,
Diffusion-4K achieves impressive performance in high-quality image synthesis
and text prompt adherence, especially when powered by modern large-scale
diffusion models (e.g., SD3-2B and Flux-12B). Extensive experimental results
from our benchmark demonstrate the superiority of Diffusion-4K in
ultra-high-resolution image synthesis.
OmnimatteZero: Training-free Real-time Omnimatte with Pre-trained Video
  Diffusion Models
Authors: name: Dvir Samuel, name: Matan Levy, name: Nir Darshan, name: Gal Chechik, name: Rami Ben-Ari
Published: 2025-03-23 11:26:48

Omnimatte aims to decompose a given video into semantically meaningful
layers, including the background and individual objects along with their
associated effects, such as shadows and reflections. Existing methods often
require extensive training or costly self-supervised optimization. In this
paper, we present OmnimatteZero, a training-free approach that leverages
off-the-shelf pre-trained video diffusion models for omnimatte. It can remove
objects from videos, extract individual object layers along with their effects,
and composite those objects onto new videos. We accomplish this by adapting
zero-shot image inpainting techniques for video object removal, a task they
fail to handle effectively out-of-the-box. We then show that self-attention
maps capture information about the object and its footprints and use them to
inpaint the object's effects, leaving a clean background. Additionally, through
simple latent arithmetic, object layers can be isolated and recombined
seamlessly with new video layers to produce new videos. Evaluations show that
OmnimatteZero not only achieves superior performance in terms of background
reconstruction but also sets a new record for the fastest Omnimatte approach,
achieving real-time performance with minimal frame runtime.
MetaSpatial: Reinforcing 3D Spatial Reasoning in VLMs for the Metaverse
Authors: name: Zhenyu Pan, name: Han Liu
Published: 2025-03-24 09:18:01

We present MetaSpatial, the first reinforcement learning (RL)-based framework
designed to enhance 3D spatial reasoning in vision-language models (VLMs),
enabling real-time 3D scene generation without the need for hard-coded
optimizations. MetaSpatial addresses two core challenges: (i) the lack of
internalized 3D spatial reasoning in VLMs, which limits their ability to
generate realistic layouts, and (ii) the inefficiency of traditional supervised
fine-tuning (SFT) for layout generation tasks, as perfect ground truth
annotations are unavailable. Our key innovation is a multi-turn RL-based
optimization mechanism that integrates physics-aware constraints and rendered
image evaluations, ensuring generated 3D layouts are coherent, physically
plausible, and aesthetically consistent. Methodologically, MetaSpatial
introduces an adaptive, iterative reasoning process, where the VLM refines
spatial arrangements over multiple turns by analyzing rendered outputs,
improving scene coherence progressively. Empirical evaluations demonstrate that
MetaSpatial significantly enhances the spatial consistency and formatting
stability of various scale models. Post-training, object placements are more
realistic, aligned, and functionally coherent, validating the effectiveness of
RL for 3D spatial reasoning in metaverse, AR/VR, digital twins, and game
development applications. Our code, data, and training pipeline are publicly
available at https://github.com/PzySeere/MetaSpatial.
Verbal Process Supervision Elicits Better Coding Agents
Authors: name: Hao-Yuan Chen, name: Cheng-Pong Huang, name: Jui-Ming Yao
Published: 2025-03-24 09:48:59

The emergence of large language models and their applications as AI agents
have significantly advanced state-of-the-art code generation benchmarks,
transforming modern software engineering tasks. However, even with test-time
computed reasoning models, these systems still struggle with complex software
engineering challenges. This work introduces CURA, a code understanding and
reasoning agent system enhanced with verbal process supervision (VPS),
achieving a 3.65\% improvement over baseline models on challenging benchmarks
like BigCodeBench. Furthermore, CURA, when paired with the o3-mini model and
VPS techniques, attains state-of-the-art performance. This work represents a
step forward in integrating reasoning-driven architectures with LLM-based code
generation, enabling agentic reasoning for language models to solve complex
software engineering tasks.
I Have Covered All the Bases Here: Interpreting Reasoning Features in
  Large Language Models via Sparse Autoencoders
Authors: name: Andrey Galichin, name: Alexey Dontsov, name: Polina Druzhinina, name: Anton Razzhigaev, name: Oleg Y. Rogov, name: Elena Tutubalina, name: Ivan Oseledets
Published: 2025-03-24 16:54:26

Large Language Models (LLMs) have achieved remarkable success in natural
language processing. Recent advances have led to the developing of a new class
of reasoning LLMs; for example, open-source DeepSeek-R1 has achieved
state-of-the-art performance by integrating deep thinking and complex
reasoning. Despite these impressive capabilities, the internal reasoning
mechanisms of such models remain unexplored. In this work, we employ Sparse
Autoencoders (SAEs), a method to learn a sparse decomposition of latent
representations of a neural network into interpretable features, to identify
features that drive reasoning in the DeepSeek-R1 series of models. First, we
propose an approach to extract candidate ''reasoning features'' from SAE
representations. We validate these features through empirical analysis and
interpretability methods, demonstrating their direct correlation with the
model's reasoning abilities. Crucially, we demonstrate that steering these
features systematically enhances reasoning performance, offering the first
mechanistic account of reasoning in LLMs. Code available at
https://github.com/AIRI-Institute/SAE-Reasoning
CFG-Zero*: Improved Classifier-Free Guidance for Flow Matching Models
Authors: name: Weichen Fan, name: Amber Yijia Zheng, name: Raymond A. Yeh, name: Ziwei Liu
Published: 2025-03-24 16:59:57

Classifier-Free Guidance (CFG) is a widely adopted technique in
diffusion/flow models to improve image fidelity and controllability. In this
work, we first analytically study the effect of CFG on flow matching models
trained on Gaussian mixtures where the ground-truth flow can be derived. We
observe that in the early stages of training, when the flow estimation is
inaccurate, CFG directs samples toward incorrect trajectories. Building on this
observation, we propose CFG-Zero*, an improved CFG with two contributions: (a)
optimized scale, where a scalar is optimized to correct for the inaccuracies in
the estimated velocity, hence the * in the name; and (b) zero-init, which
involves zeroing out the first few steps of the ODE solver. Experiments on both
text-to-image (Lumina-Next, Stable Diffusion 3, and Flux) and text-to-video
(Wan-2.1) generation demonstrate that CFG-Zero* consistently outperforms CFG,
highlighting its effectiveness in guiding Flow Matching models. (Code is
available at github.com/WeichenFan/CFG-Zero-star)
Equivariant Image Modeling
Authors: name: Ruixiao Dong, name: Mengde Xu, name: Zigang Geng, name: Li Li, name: Han Hu, name: Shuyang Gu
Published: 2025-03-24 17:59:57

Current generative models, such as autoregressive and diffusion approaches,
decompose high-dimensional data distribution learning into a series of simpler
subtasks. However, inherent conflicts arise during the joint optimization of
these subtasks, and existing solutions fail to resolve such conflicts without
sacrificing efficiency or scalability. We propose a novel equivariant image
modeling framework that inherently aligns optimization targets across subtasks
by leveraging the translation invariance of natural visual signals. Our method
introduces (1) column-wise tokenization which enhances translational symmetry
along the horizontal axis, and (2) windowed causal attention which enforces
consistent contextual relationships across positions. Evaluated on
class-conditioned ImageNet generation at 256x256 resolution, our approach
achieves performance comparable to state-of-the-art AR models while using fewer
computational resources. Systematic analysis demonstrates that enhanced
equivariance reduces inter-task conflicts, significantly improving zero-shot
generalization and enabling ultra-long image synthesis. This work establishes
the first framework for task-aligned decomposition in generative modeling,
offering insights into efficient parameter sharing and conflict-free
optimization. The code and models are publicly available at
https://github.com/drx-code/EquivariantModeling.
AgentRxiv: Towards Collaborative Autonomous Research
Authors: name: Samuel Schmidgall, name: Michael Moor
Published: 2025-03-23 15:16:42

Progress in scientific discovery is rarely the result of a single "Eureka"
moment, but is rather the product of hundreds of scientists incrementally
working together toward a common goal. While existing agent workflows are
capable of producing research autonomously, they do so in isolation, without
the ability to continuously improve upon prior research results. To address
these challenges, we introduce AgentRxiv-a framework that lets LLM agent
laboratories upload and retrieve reports from a shared preprint server in order
to collaborate, share insights, and iteratively build on each other's research.
We task agent laboratories to develop new reasoning and prompting techniques
and find that agents with access to their prior research achieve higher
performance improvements compared to agents operating in isolation (11.4%
relative improvement over baseline on MATH-500). We find that the best
performing strategy generalizes to benchmarks in other domains (improving on
average by 3.3%). Multiple agent laboratories sharing research through
AgentRxiv are able to work together towards a common goal, progressing more
rapidly than isolated laboratories, achieving higher overall accuracy (13.7%
relative improvement over baseline on MATH-500). These findings suggest that
autonomous agents may play a role in designing future AI systems alongside
humans. We hope that AgentRxiv allows agents to collaborate toward research
goals and enables researchers to accelerate discovery.
Defeating Prompt Injections by Design
Authors: name: Edoardo Debenedetti, name: Ilia Shumailov, name: Tianqi Fan, name: Jamie Hayes, name: Nicholas Carlini, name: Daniel Fabian, name: Christoph Kern, name: Chongyang Shi, name: Andreas Terzis, name: Florian Tramèr
Published: 2025-03-24 15:54:10

Large Language Models (LLMs) are increasingly deployed in agentic systems
that interact with an external environment. However, LLM agents are vulnerable
to prompt injection attacks when handling untrusted data. In this paper we
propose CaMeL, a robust defense that creates a protective system layer around
the LLM, securing it even when underlying models may be susceptible to attacks.
To operate, CaMeL explicitly extracts the control and data flows from the
(trusted) query; therefore, the untrusted data retrieved by the LLM can never
impact the program flow. To further improve security, CaMeL relies on a notion
of a capability to prevent the exfiltration of private data over unauthorized
data flows. We demonstrate effectiveness of CaMeL by solving 67% of tasks
with provable security in AgentDojo [NeurIPS 2024], a recent agentic security
benchmark.
Variance Control via Weight Rescaling in LLM Pre-training
Authors: name: Louis Owen, name: Abhay Kumar, name: Nilabhra Roy Chowdhury, name: Fabian Güra
Published: 2025-03-21 19:23:08

The outcome of Large Language Model (LLM) pre-training strongly depends on
weight initialization and variance control strategies. Although the importance
of initial variance control has been well documented in neural networks in
general, the literature on initialization and management of its growth during
LLM pre-training, specifically, is somewhat sparse. In this paper, we introduce
the Layer Index Rescaling (LIR) weight initialization scheme, and the Target
Variance Rescaling (TVR) variance control strategy. Experiments on a 1B
parameter LLaMA model demonstrate that better variance management using these
techniques yields substantial improvements in downstream task performance (up
to 4.6% on common pre-training benchmarks) and reduces extreme activation
values, thus mitigating challenges associated with quantization and
low-precision training. Our code is available at:
https://github.com/bluorion-com/weight_rescaling.
Typed-RAG: Type-aware Multi-Aspect Decomposition for Non-Factoid
  Question Answering
Authors: name: DongGeon Lee, name: Ahjeong Park, name: Hyeri Lee, name: Hyeonseo Nam, name: Yunho Maeng
Published: 2025-03-20 06:04:12

Non-factoid question-answering (NFQA) poses a significant challenge due to
its open-ended nature, diverse intents, and the need for multi-aspect
reasoning, which renders conventional factoid QA approaches, including
retrieval-augmented generation (RAG), inadequate. Unlike factoid questions,
non-factoid questions (NFQs) lack definitive answers and require synthesizing
information from multiple sources across various reasoning dimensions. To
address these limitations, we introduce Typed-RAG, a type-aware multi-aspect
decomposition framework within the RAG paradigm for NFQA. Typed-RAG classifies
NFQs into distinct types -- such as debate, experience, and comparison -- and
applies aspect-based decomposition to refine retrieval and generation
strategies. By decomposing multi-aspect NFQs into single-aspect sub-queries and
aggregating the results, Typed-RAG generates more informative and contextually
relevant responses. To evaluate Typed-RAG, we introduce Wiki-NFQA, a benchmark
dataset covering diverse NFQ types. Experimental results demonstrate that
Typed-RAG outperforms baselines, thereby highlighting the importance of
type-aware decomposition for effective retrieval and generation in NFQA. Our
code and dataset are available at
https://github.com/TeamNLP/Typed-RAG{https://github.com/TeamNLP/Typed-RAG}.
Optimized Minimal 3D Gaussian Splatting
Authors: name: Joo Chan Lee, name: Jong Hwan Ko, name: Eunbyung Park
Published: 2025-03-21 07:41:45

3D Gaussian Splatting (3DGS) has emerged as a powerful representation for
real-time, high-performance rendering, enabling a wide range of applications.
However, representing 3D scenes with numerous explicit Gaussian primitives
imposes significant storage and memory overhead. Recent studies have shown that
high-quality rendering can be achieved with a substantially reduced number of
Gaussians when represented with high-precision attributes. Nevertheless,
existing 3DGS compression methods still rely on a relatively large number of
Gaussians, focusing primarily on attribute compression. This is because a
smaller set of Gaussians becomes increasingly sensitive to lossy attribute
compression, leading to severe quality degradation. Since the number of
Gaussians is directly tied to computational costs, it is essential to reduce
the number of Gaussians effectively rather than only optimizing storage. In
this paper, we propose Optimized Minimal Gaussians representation (OMG), which
significantly reduces storage while using a minimal number of primitives.
First, we determine the distinct Gaussian from the near ones, minimizing
redundancy without sacrificing quality. Second, we propose a compact and
precise attribute representation that efficiently captures both continuity and
irregularity among primitives. Additionally, we propose a sub-vector
quantization technique for improved irregularity representation, maintaining
fast training with a negligible codebook size. Extensive experiments
demonstrate that OMG reduces storage requirements by nearly 50% compared to the
previous state-of-the-art and enables 600+ FPS rendering while maintaining high
rendering quality. Our source code is available at
https://maincold2.github.io/omg/.
FFN Fusion: Rethinking Sequential Computation in Large Language Models
Authors: name: Akhiad Bercovich, name: Mohammad Dabbah, name: Omri Puny, name: Ido Galil, name: Amnon Geifman, name: Yonatan Geifman, name: Izhak Golan, name: Ehud Karpas, name: Itay Levy, name: Zach Moshe, name: Najeeb Nabwani, name: Tomer Ronen, name: Itamar Schen, name: Elad Segal, name: Ido Shahaf, name: Oren Tropp, name: Ran Zilberstein, name: Ran El-Yaniv
Published: 2025-03-24 17:20:35

We introduce FFN Fusion, an architectural optimization technique that reduces
sequential computation in large language models by identifying and exploiting
natural opportunities for parallelization. Our key insight is that sequences of
Feed-Forward Network (FFN) layers, particularly those remaining after the
removal of specific attention layers, can often be parallelized with minimal
accuracy impact. We develop a principled methodology for identifying and fusing
such sequences, transforming them into parallel operations that significantly
reduce inference latency while preserving model behavior. Applying these
techniques to Llama-3.1-405B-Instruct, we create Llama-Nemotron-Ultra-253B-Base
(Ultra-253B-Base), an efficient and soon-to-be publicly available model that
achieves a 1.71X speedup in inference latency and 35X lower per-token cost
while maintaining strong performance across benchmarks. Through extensive
experiments on models from 49B to 253B parameters, we demonstrate that FFN
Fusion becomes increasingly effective at larger scales and can complement
existing optimization techniques like quantization and pruning. Most
intriguingly, we find that even full transformer blocks containing both
attention and FFN layers can sometimes be parallelized, suggesting new
directions for neural architecture design.
Aether: Geometric-Aware Unified World Modeling
Authors: name: Aether Team, name: Haoyi Zhu, name: Yifan Wang, name: Jianjun Zhou, name: Wenzheng Chang, name: Yang Zhou, name: Zizun Li, name: Junyi Chen, name: Chunhua Shen, name: Jiangmiao Pang, name: Tong He
Published: 2025-03-24 17:59:51

The integration of geometric reconstruction and generative modeling remains a
critical challenge in developing AI systems capable of human-like spatial
reasoning. This paper proposes Aether, a unified framework that enables
geometry-aware reasoning in world models by jointly optimizing three core
capabilities: (1) 4D dynamic reconstruction, (2) action-conditioned video
prediction, and (3) goal-conditioned visual planning. Through task-interleaved
feature learning, Aether achieves synergistic knowledge sharing across
reconstruction, prediction, and planning objectives. Building upon video
generation models, our framework demonstrates unprecedented synthetic-to-real
generalization despite never observing real-world data during training.
Furthermore, our approach achieves zero-shot generalization in both action
following and reconstruction tasks, thanks to its intrinsic geometric modeling.
Remarkably, even without real-world data, its reconstruction performance far
exceeds that of domain-specific models. Additionally, Aether leverages a
geometry-informed action space to seamlessly translate predictions into
actions, enabling effective autonomous trajectory planning. We hope our work
inspires the community to explore new frontiers in physically-reasonable world
modeling and its applications.